An MQTT broker is very good at moving messages from many publishers to many subscribers, but it does not store anything for long, so telemetry that arrives on a topic vanishes the moment the last subscriber has read it. To keep that data for trending, reporting, and analytics, something has to subscribe to the broker, turn each message into a row, and write it into a database built for timestamped values. That job is MQTT to time-series database ingestion. This guide walks through the ingestion path - subscribing, decoding payloads, mapping topics or Sparkplug metrics to measurements and tags, batching the writes, and applying backpressure when the database cannot keep up - and explains why the mapping layer in the middle is the part that decides whether the stored data is usable.
MQTT to Time-Series Database Ingestion in one line: MQTT to time-series database ingestion is the pipeline that connects an MQTT broker to a time-series database so that published telemetry is durably stored. A subscriber client receives messages, decodes each payload, maps the topic hierarchy or the Sparkplug metric names to a measurement with tags and fields, and writes the resulting points into the database, usually in batches for throughput. The pipeline also has to handle quality-of-service and durability trade-offs, out-of-order timestamps, and backpressure so that a slow database does not overwhelm the subscriber or lose data.
The pipeline begins with a subscriber that connects to the broker and subscribes to the topics carrying telemetry, often using wildcards so that a single subscription can capture an entire tree of tags. Each message the subscriber receives is a topic string plus a payload, and the payload can be almost anything the publisher chose: a bare number, a small JSON object with a value and a timestamp, or a binary Sparkplug B message carrying many metrics at once. The first job of the ingestion code is to decode that payload into concrete values, which means knowing the encoding for each topic tree and parsing accordingly, rejecting or quarantining anything it cannot understand rather than writing garbage.
Once decoded, each reading has to become a point in the time-series database, and a point needs three things: a timestamp, a set of tags that identify what the reading belongs to, and one or more field values. Where the timestamp comes from matters. If the publisher stamped the reading at the source, the ingestion layer should use that time so the stored history reflects when the measurement was actually taken; only if no source timestamp is present should it fall back to the time of receipt, which can be seconds or minutes later on a congested link. Using receipt time everywhere quietly distorts the history and makes it impossible to line up readings from different devices.
The write itself is rarely done one point at a time, because inserting into a time-series database has per-request overhead that would throttle throughput if every message triggered its own round trip. Instead the subscriber accumulates decoded points in a buffer and flushes them as a batch, either when the buffer reaches a size threshold or when a short timer expires, whichever comes first. Batching dramatically raises the number of readings the pipeline can absorb per second, and the timer guarantees that a slow trickle of messages still gets written promptly rather than sitting in the buffer indefinitely. Getting the batch size and flush interval right is one of the main tuning levers of the whole pipeline.
The mapping layer is where raw messages become well-structured data, and it is the part teams most often underestimate. A time-series database organises data by measurement name, tag set, and field, and the ingestion layer has to decide how an MQTT topic and payload translate into those. A common approach is to split the topic hierarchy into tags: a topic like a site, area, line, and device path becomes tags naming the site, area, line, and device, and the leaf becomes the measurement or field. Doing this deliberately, with a schema-on-write mapping, means the stored data is queryable by any of those dimensions later, which is exactly what a report or dashboard needs.
Sparkplug B changes the shape of the problem in a helpful way. A Sparkplug payload is not a single reading but a set of named metrics, each with its own value, data type, and timestamp, published under a structured topic that already identifies the group, edge node, and device. That structure gives the ingestion layer a much richer source to map from: each metric name becomes a measurement or field, the group and node and device become tags, and the per-metric timestamps carry through directly. Because Sparkplug also defines birth messages that declare the full set of metrics and their types up front, an ingestion layer can register a stable schema and validate incoming values against it rather than guessing at types from whatever arrives.
The reason schema-on-write mapping matters, rather than dumping every message into a single blob column, is that the value of a time-series store is in slicing and aggregating by dimension. If site, asset, and measurement live in tags, a query can pull one asset's temperature over a month, or average a metric across a whole site, cheaply. If that structure was never applied at ingestion, the same questions require reparsing every stored payload after the fact, which is slow and error-prone. Investing in a clean mapping at write time is what turns a firehose of MQTT messages into a queryable historian, and it is precisely the tedious, easy-to-get-wrong work that a purpose-built ingestion service exists to handle.
The durability of the pipeline is shaped first by MQTT quality of service. At QoS 0 a message is fired once with no acknowledgement, so if the subscriber is briefly disconnected the reading is simply gone; at QoS 1 the broker redelivers until acknowledged, which guarantees delivery but can produce duplicates the ingestion layer must tolerate; at QoS 2 delivery is exactly once at the cost of more handshaking. Higher QoS improves the chance every reading reaches the database but does nothing to protect readings published while the subscriber is down unless the broker or the publisher retains or queues them. Choosing QoS is therefore a direct trade between how much telemetry you can afford to lose and how much overhead and duplicate-handling you are willing to build.
Out-of-order arrival is normal and must be planned for. Messages from different devices, or from one device over a lossy link with retries, can reach the subscriber with timestamps that are not monotonically increasing, so the ingestion layer cannot assume each new point is later than the last. Time-series databases generally accept out-of-order writes and place each point at its own timestamp, but any code that assumes strict ordering - for example computing a rate of change on the fly - will produce nonsense unless it sorts by timestamp first. The safe design is to trust the timestamp on each reading and let the database order the history, rather than relying on arrival order.
Backpressure is what keeps the pipeline stable when the database falls behind, and it is essential rather than optional. If ingest is arriving faster than the database can absorb, the subscriber's buffer grows, and without a limit it will eventually exhaust memory and crash, taking the whole ingestion path down. A robust pipeline caps the buffer and, when it fills, applies backpressure: it slows or pauses reading from the broker so unacknowledged messages stay queued at the broker rather than piling up in the subscriber, letting the broker's own retention absorb the surge. This is exactly the kind of plumbing - subscriber, decoder, schema mapping, batching, and backpressure - that a cloud SCADA platform such as Merobix provides as an ingest-plus-store service, so a team can point their broker at it and get a queryable, durable history without building and operating the subscriber-writer themselves.
Direct writes from many field devices tie every device to the database's location, credentials, and availability, so a database outage or a firewall change can stop devices from reporting and even back up their local buffers. Publishing to an MQTT broker decouples the devices from storage: the broker accepts messages regardless of what the database is doing, and a single ingestion service reads from the broker and handles batching, schema mapping, and backpressure in one place. That separation is easier to scale, secure, and maintain than granting hundreds of devices direct database access.
With plain MQTT you usually have to derive structure yourself, splitting the topic path into tags and interpreting whatever the payload happens to contain, because there is no standard for how values and types are expressed. Sparkplug B provides that structure for you: its topics identify the group, edge node, and device, its payloads carry named metrics with explicit data types and timestamps, and its birth messages declare the full metric set up front. That means a Sparkplug-aware ingestion layer can register a known schema, validate types, and map metrics to fields reliably, whereas plain-topic ingestion depends on a mapping the team defines and maintains by convention.
The key is backpressure combined with the broker's own buffering. When the database cannot keep up, the ingestion service caps its in-memory buffer and slows or pauses its reads from the broker instead of accumulating messages until it runs out of memory. With quality of service 1 or 2, unacknowledged messages remain queued at the broker and are redelivered once the ingestion service resumes, so a temporary database slowdown or outage delays writes rather than dropping readings. Data can still be lost at quality of service 0 or if a device publishes while nothing is subscribed and the broker is not retaining, which is why durability requirements should drive both the QoS choice and the broker retention settings.
This page references the protocol specifications published by the organizations below. Editions, product capabilities, and documentation change over time - confirm current requirements and specifications directly with the source.
Last reviewed: July 27, 2026. Merobix is not affiliated with, endorsed by, or sponsored by these organizations; their names are used only to identify the standards and products discussed.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.