Kafka has become the default backbone when a plant wants tag changes to flow to more than one destination at once and to survive a downstream consumer being slow or offline. It is a durable, replayable log that many systems can read from independently, which is a natural fit for telemetry that a historian, an analytics job, and an alerting service might all want. This page covers how SCADA tag changes get onto Kafka topics, how the topic and partition design decides whether time-series data stays in order, and how Kafka Connect carries the stream onward into warehouses and lakes.
SCADA to Kafka integration in one line: SCADA to Kafka streaming integration is the practice of publishing SCADA tag changes onto Apache Kafka topics so that multiple downstream systems can consume the same telemetry independently and reliably. A bridge or edge producer, often converting from MQTT, writes each tag change as a Kafka record, and the topic and partition design determines how the data is grouped and ordered. Kafka Connect sink connectors then carry the stream into destinations like data warehouses and object stores, while message keys control which readings stay in guaranteed order relative to one another.
The first design decision is how to lay tag changes across topics. Two common patterns sit at opposite ends. Topic-per-site puts all of a site's tag changes onto a single topic, which keeps the topic count manageable and groups data the way operations often think about it, but means consumers interested in only a few tags read past everything else. Topic-per-tag-group splits telemetry into topics by measurement type or logical grouping, which lets consumers subscribe more narrowly but multiplies the number of topics and the administrative overhead. Most plants land somewhere in between, using a modest number of topics organized by site or by broad measurement category, chosen so the common consumers can subscribe efficiently without an explosion of topics to manage.
Kafka only guarantees order within a partition, not across a whole topic, and this is the detail that matters most for time-series data. Each topic is split into partitions for parallelism, and records are placed into a partition by hashing their key. If tag changes for a given tag or a given asset all carry the same key, they all land in the same partition and are therefore delivered in the order they were produced, which is exactly what a consumer rebuilding a value's history needs. Choosing a key at the tag or asset level preserves per-series ordering while still spreading the overall load across partitions.
The mistake to avoid is producing without a meaningful key, or keying too coarsely, so that a single value's readings scatter across partitions and arrive out of order at the consumer. For telemetry that will be turned back into ordered time series, out-of-order delivery forces consumers to sort and dedup after the fact, which is avoidable by keying correctly at the source. The keying decision and the partition count are set together, because too few partitions limits throughput while too many can leave some nearly idle, and the right balance keeps both ordering and parallelism intact.
Once telemetry is on a topic, Kafka Connect is the standard way to move it onward without writing custom consumer code. A sink connector reads from one or more topics and writes into an external system, and there are connectors for common warehouses, object stores, databases, and search systems. This means a SCADA stream on Kafka can feed a data warehouse for analytics, a lake for cheap long-term storage, and a live processing system simultaneously, each through a configured connector rather than a bespoke application, and because each consumer tracks its own position in the log, one falling behind does not hold up the others.
Delivery semantics deserve honest attention because they are easy to get subtly wrong. Kafka can be configured for at-least-once delivery, where a record is guaranteed to be processed but might be seen more than once after a retry, or for exactly-once semantics in the paths that support it, where a record is effectively processed a single time even across failures. For telemetry, at-least-once is often acceptable if consumers can dedup on a tag-and-timestamp key, because a duplicate reading is harmless once discarded, while exactly-once matters more when records drive counting or accumulation where a double-count would corrupt a total. Choosing the right guarantee per pipeline avoids paying for stronger semantics than a given consumer actually needs.
The durability and replay that Kafka provides are the operational payoff. Because the log retains records for a configured window, a sink that was offline can resume from where it left off and catch up rather than losing the telemetry it missed, and a new consumer can replay history from the log to backfill a system. That decoupling, where producers keep writing regardless of consumer health and consumers read at their own pace, is what makes Kafka resilient in a plant environment where network links and downstream systems are not always reliable.
The bridge from the field to Kafka is where the integration actually lives. SCADA edges commonly speak MQTT, so a frequent pattern is an MQTT-to-Kafka bridge that subscribes to the plant's MQTT topics and republishes each message as a Kafka record, mapping the MQTT topic structure onto Kafka topics and keys. Other edges expose OPC UA or a historian interface, in which case a producer polls or subscribes to those sources and writes records. Whatever the source, the producer is responsible for setting a sensible key so ordering is preserved and for shaping the payload into a consistent record format so downstream consumers do not each have to parse a different structure.
Report-by-exception is a natural fit here and worth designing for. Rather than pushing every tag on a fixed interval, many SCADA sources emit a record only when a value changes by more than a deadband, which cuts the volume on the log dramatically without losing meaningful movement. Producing tag changes rather than a steady tick keeps the topics carrying signal rather than a flat stream of unchanged readings, which makes both the storage and the downstream processing more efficient. Consumers that need a regular cadence can resample from the change events, so change-based production does not prevent interval-based analysis downstream.
This is a place where a monitoring layer simplifies the field side considerably. A platform such as Merobix can collect tags across the mixed protocols a real plant runs, apply consistent naming and quality flags, and act as a single, well-behaved producer onto Kafka, rather than every edge device implementing its own bridge with its own conventions. When the log receives clean, consistently keyed records from one collection layer, the whole downstream ecosystem of connectors and consumers becomes simpler, and operators still keep a live view of plant state in the monitoring layer while Kafka handles the durable distribution to analytics.
Topic-per-site keeps the topic count low and groups data the way operations think about it, but consumers interested in a few tags read past everything else. Topic-per-tag-group lets consumers subscribe more narrowly at the cost of many more topics to manage. Most plants use a modest number of topics organized by site or broad measurement category, chosen so common consumers can subscribe efficiently without an explosion of topics.
Kafka only guarantees order within a partition, and a record's partition is chosen by hashing its key. If all readings for a given tag or asset share the same key, they land in the same partition and stay in produced order, which is what a consumer rebuilding a value's history needs. Producing without a meaningful key scatters a value's readings across partitions and delivers them out of order, forcing consumers to sort and dedup afterward.
Not always. At-least-once delivery is often fine for raw telemetry when consumers dedup on a tag-and-timestamp key, since a duplicate reading is harmless once discarded. Exactly-once matters more where records drive counting or accumulation and a double-count would corrupt a total. Choosing the guarantee per pipeline avoids paying for stronger semantics than a given consumer actually needs.
This page references the protocol specifications published by the organizations below. Editions, product capabilities, and documentation change over time - confirm current requirements and specifications directly with the source.
Last reviewed: July 27, 2026. Merobix is not affiliated with, endorsed by, or sponsored by these organizations; their names are used only to identify the standards and products discussed.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.