A historian collector sits close to the data source and sends values on to the central archive, but that archive is not always reachable. When the link drops or the archive is down, the collector cannot simply throw away the values it is still gathering, so it holds them in a local buffer until it can catch up. This store-and-forward buffer is the collector-side safety net that keeps an outage from becoming a permanent gap in history. This page explains what the buffer is, how to size it for the outages you expect, the trade-off between memory and disk buffering, and what happens when the buffer fills and starts to overwrite the oldest data.
Collector store-and-forward buffer in one line: A collector store-and-forward buffer is the local memory or disk cache inside a historian collector that spools collected values when the central archive is unreachable, then forwards them once the connection returns. It is sized for the longest outage you expect at the tag's collection rate so no data is lost, and it can use memory for speed or disk for durability. When the buffer fills, it typically overwrites the oldest data, so undersizing it causes silent loss.
A collector's normal job is to gather values from a data source and send them to the central archive as it goes. The store-and-forward buffer is what the collector uses when that sending cannot happen, whether because the network link to the archive has failed, the archive server is down or restarting, or the archive is temporarily too busy to accept writes. Instead of discarding the values it continues to collect during that window, the collector writes them into its local buffer, preserving them in order with their original timestamps. The buffer is, in effect, a holding area that decouples the act of collecting data from the act of successfully delivering it.
When the connection to the archive is restored, the collector drains the buffer by forwarding the held values to the archive, so the history that would otherwise have been a gap is filled in after the fact. From the archive's point of view, the data arrives late but complete, carrying the timestamps from when it was actually collected rather than when it was finally forwarded. This is the whole value of store-and-forward at the collector: a communications outage becomes a delay in delivery rather than a permanent loss of data, provided the buffer held everything until the link came back.
It is important to place this buffer precisely, because store-and-forward is a general idea that appears at several points in a telemetry system. Here the concern is specifically the buffer inside the historian collector, between the data source and the central archive. Its capacity, its storage medium, and its behaviour when full are properties of the collector's configuration, and they determine directly whether a given length of archive outage results in complete recovery or in lost data. That collector-side buffer, and the capacity math that governs it, is what this page is about.
Sizing the buffer is fundamentally a capacity calculation: how much data will the collector accumulate over the longest outage it must survive without loss. That depends on how many tags the collector handles, how fast each is collected, and how long a realistic worst-case outage lasts. A collector gathering many fast tags accumulates data quickly, so it needs a larger buffer to cover the same outage duration than a collector handling a few slow tags. The honest way to size the buffer is to decide the outage length you must tolerate, estimate the data rate the collector produces, and provide enough buffer capacity to hold their product with margin, rather than accepting whatever default the collector ships with.
The choice between buffering in memory and buffering on disk is a trade-off between speed and durability. A memory buffer is fast and simple, but it is volatile: if the collector process restarts or the machine loses power during the outage, whatever the memory buffer held is gone, which defeats the purpose at exactly the wrong moment. A disk buffer survives a restart or a power cycle, so the held data is still there when the collector comes back, at the cost of some overhead and dependence on the local storage being healthy. For anything but the shortest, most benign outages, a disk-backed buffer is the safer choice precisely because outages and restarts often coincide.
In practice the sizing and the medium interact, because a disk buffer can usually be made far larger than a memory buffer, which is what allows a collector to ride out a long outage. A remote site that might lose its link to a central archive for hours needs a buffer large enough and durable enough to hold every value collected across that whole window, which almost always means a generously sized disk buffer. Deciding these parameters deliberately, from the expected outage profile of the specific site, is what turns store-and-forward from a nominal feature into a genuine guarantee that an outage will not cost you data.
A buffer has a finite size, and when an outage lasts longer than the buffer can hold, something has to give. The common behaviour is that the buffer overwrites its oldest data to make room for new collected values, so the collector keeps capturing the present at the cost of losing the beginning of the outage window. The practical consequence is a permanent gap at the start of the outage: when the link returns and the buffer drains, the earliest data was already discarded and cannot be recovered. An undersized buffer therefore does not fail loudly; it quietly loses the oldest part of the outage, which is exactly the kind of silent loss that erodes trust in a historian.
Because the failure is silent, monitoring the buffer's fill level is as important as sizing it. A collector that reports how full its buffer is, and warns as it approaches capacity, gives operators a chance to intervene, extend the buffer, or prioritize restoring the archive link before data starts to be overwritten. Treating a nearly full buffer as an actionable warning rather than an internal detail is what keeps a long outage from turning into an invisible hole in history. Without that visibility, the first sign of trouble is discovering, after the fact, that data from the middle of an outage simply is not there.
This matters most for remote SCADA sites, where the collector may sit at a field location whose link to the central archive is exactly the connection most likely to fail, and where no one is on site to notice. A well-provisioned collector buffer is what lets such a site lose communications for an extended period and still deliver its complete history once the link recovers, which is central to trusting data from unmanned locations. A platform such as Merobix pairs generously sized, durable buffering with visibility into buffer health across sites, so that both the guarantee, holding data through an outage, and the warning, flagging when a buffer is close to overwriting, are in place. That combination turns store-and-forward from a hopeful mechanism into a dependable one for a distributed fleet.
Work from the longest outage the collector must survive without loss and the data rate it produces. Multiply the number of tags and their collection rates to estimate how much data accumulates per unit time, then provide enough buffer capacity to hold that rate across the full outage duration, with margin. A collector handling many fast tags needs a larger buffer than one handling a few slow tags to cover the same outage length.
For anything but the shortest, most benign outages, a disk-backed buffer is safer. A memory buffer is fast but volatile, so if the collector process restarts or the machine loses power during the outage, the held data is lost at exactly the wrong moment. A disk buffer survives a restart or power cycle and can usually be made much larger, which is what lets a collector ride out a long outage without losing data.
When an outage lasts longer than the buffer can hold, the buffer typically overwrites its oldest data to make room for new values, creating a permanent gap at the start of the outage window. This loss is silent, so monitoring the buffer's fill level and warning as it approaches capacity is essential, giving operators a chance to extend the buffer or restore the archive link before data is overwritten.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.