Data streaming company Confluent, now an IBM company, has announced a series of new capabilities across Confluent Intelligence and Confluent Cloud designed to streamline the development and deployment of real-time AI applications. The updates focus on removing common barriers that prevent AI projects from reaching production, including fragmented data, security risks, and complex toolchains.
The new capabilities include agent-powered workflows that allow AI to manage streaming operations using natural language, automated detection and redaction of personally identifiable information (PII) directly in Flink SQL, private cloud connectivity via Azure Private Link, and a dbt adapter that brings Flink streaming pipelines into the widely-used data engineering framework. These additions aim to unify the AI lifecycle and embed enterprise-grade governance into data streams from the start.
The Challenge of Production AI
According to a recent McKinsey report, eight in ten companies cite data limitations as a roadblock to scaling agentic AI. These limitations often stem from security teams blocking data from entering AI pipelines due to exposure risks, and developers losing hours to tool-switching to inspect and manage the data streams their AI depends on. The resulting slow, manual process turns what should be a fast iteration cycle into a bottleneck.
Sean Falconer, head of AI at Confluent, stated: 'Most AI projects fail before they reach a single customer because the data layer breaks down. Teams have the models and the mandate, but security risks and fragmented data stop them from shipping. We're fixing that by making the streaming layer the foundation for secure, production-ready AI.'
Agent-Powered Workflows with MCP
Confluent has introduced a fully managed Model Context Protocol (MCP) server and Agent Skills that let AI manage streaming operations. The Confluent MCP server acts as a control plane, allowing AI agents to build, manage, and debug streaming operations using natural language commands. Agent Skills add a second layer, encoding best practices and workflows so those operations are executed consistently and in line with organizational standards.
Together, Confluent MCP and Agent Skills enable developers to create and continuously improve real-time applications using AI-powered tools, bringing streaming into modern, agent-driven development workflows. This capability is now generally available for Confluent Cloud users, allowing teams to automate routine tasks, monitor data pipelines, and respond to anomalies without manual intervention.
Automated Data Privacy
A new built-in ML function for PII detection and redaction protects sensitive information directly in Flink SQL, without requiring custom code, external services, or moving data to a warehouse first. This automated approach simplifies compliance with regulations like GDPR and CCPA, and unlocks more AI use cases across highly regulated industries such as financial services, healthcare, and insurance.
The PII redaction function is available in early access for Confluent Intelligence. It uses machine learning models to identify and mask patterns such as email addresses, phone numbers, social security numbers, and credit card details in real time, ensuring that sensitive data never leaves the streaming environment unprotected.
Private Cloud Connectivity
Support for Azure Private Link ensures that AI workloads stay off the public internet by providing secure, private paths to calling external models and querying external tables. Now, Flink jobs can securely connect to Azure-hosted services such as Azure OpenAI, Azure SQL, and Cosmos DB over Microsoft's private backbone, reducing latency and enhancing data security.
This private connectivity is generally available on Confluent Cloud. It allows enterprises to extend their existing cloud infrastructure into AI pipelines without exposing data to the public network, addressing a key concern for organizations with strict security requirements.
Unified Engineering Workflows
The free open-source dbt adapter brings Flink SQL on Confluent Cloud into dbt, the industry-standard framework data engineers use to build and manage data pipelines. Teams can define, test, and deploy streaming pipelines using the same dbt commands and project structure they rely on today, lowering the barrier to Flink adoption and making it easier to extend existing data workflows into real-time use cases.
This adapter is generally available on Confluent Cloud. It supports version control, testing, and documentation generation for streaming pipelines, enabling data engineers to integrate real-time processing into their existing CI/CD workflows without learning a new tool.
Support for Advanced AI Models
Confluent now supports TimesFM models for robust anomaly detection, as well as models from Anthropic and Fireworks AI. Developers can directly use these models in Flink stream processing workflows to build sophisticated real-time AI applications. TimesFM, developed by Google, is particularly well-suited for time-series analysis and anomaly detection in streaming data, while Anthropic's Claude models and Fireworks AI's optimized models offer advanced capabilities for natural language processing and decision-making.
The integration with Flink allows these models to be invoked directly within SQL queries, making it straightforward to enrich data streams with AI-powered insights. For example, a financial institution could detect fraudulent transactions in real time by applying an anomaly detection model to transaction streams, or a healthcare provider could analyze patient vitals and trigger alerts using natural language models.
Confluent's updates reflect a broader trend in the industry toward making AI more accessible and production-ready. By addressing the data layer challenges that have hindered many projects, the company aims to help organizations move from AI experimentation to operationalization. The combination of real-time streaming, security controls, and unified engineering workflows positions Confluent as a key enabler of next-generation AI applications that require continuous, high-quality data.
Source: Computerweekly News