All Tags Flatlined at Once: How to Triage It

Back to Blog

First Checks: Map the Scope of the Flat

The fastest and cheapest check is to draw the boundary of what flatlined. Pull up a wide set of tags and note exactly which ones went flat and which are still moving. Are they all from one controller, one poll group, one site, one communication path, or the entire SCADA client? The answer is the diagnosis in outline, because whatever the flat tags share and the live tags do not is the failed element. This costs one screen and thirty seconds and eliminates most of the possibility space.

Confirm the flats are truly simultaneous, not merely close. Zoom the trend to the moment of the stop and check that the last-changing samples line up to the same timestamp. A genuinely shared failure stops everything on the same scan or poll cycle. A cluster of stops spread over seconds or minutes is a cascade, which is a different and usually field-side story. The shape of the edge where the lines go flat carries information: a clean vertical wall across all pens is a communication or poll stop, a staggered set of stops is something propagating.

Read what the flat tags are flat at. Pens pinned at the bottom rail or the top rail are a different symptom than pens flat at whatever value they happened to hold, and that distinction is worth understanding on its own; the guide on reading a flatlined tag walks the pegged-low, pegged-high, and flat-at-last-value patterns in detail. For the all-at-once case, flat at the last plausible value across the whole group is the signature of a stopped update - the values are the last ones delivered before the pipe closed, held on screen and about to go stale.

Stopped Poll Versus Dead Device Versus Frozen Link

If the flat group is exactly one poll group or one poll engine's worth of tags, suspect the poll first. A poll that stalls, overruns, or gets stuck behind a slow device holds every tag it serves at the last value until it recovers, and the scope will match the poll's tag list precisely. Check the poll engine's own health and scan statistics; a poll that is running long or has stopped cycling explains a flat that follows the poll boundary exactly. Understanding how a poll group is organized helps you predict which tags share a fate, and the concept of a scan class in the controller does the same on the device side.

If the flat group is exactly one device's tags and the poll is healthy for everything else, suspect the device or its link. A controller that hung, lost power, or dropped its network connection stops answering, and the SCADA holds its last values and then flags them. The tell is whether other devices on the same poll kept updating: they did, so the poll is fine and this one device is the problem. This is where a communication-failure indication, if the system raises one, confirms that the device specifically stopped answering rather than the whole poll stalling.

If the flat group is an entire screen or the whole client but the historian kept recording, the pipe between the client and the server froze, not the data collection. The values are still being gathered, the display just stopped receiving them. The proof is to compare a live trend on the client against the historian's record for the same tag over the same window: if the historian moved while the screen stayed flat, the collection is healthy and the client session is the fault. That comparison is the cleanest single test for separating a display freeze from a real data stop.

Verifying and Isolating the Shared Element

Turn the scope map into a decisive test by finding one live tag and one dead tag that differ in exactly one shared element. If a tag on poll group A is moving and a tag on poll group B is flat, and they otherwise ride the same site and link, the poll is the difference and the poll is the fault. If two tags on the same device disagree, that is no longer an all-at-once symptom and you have moved to a single-tag problem. Bisecting on the one thing that differs between a healthy and a failed tag converges on the shared element quickly.

Watch for the failure that fakes recovery: a stopped poll or frozen link that resumes on its own after a timeout, leaving a flat segment bracketed by live data. That flat segment is a gap in disguise, and if the process moved during it you have lost that history unless the system backfills. Recognizing the bracketed flat as a transient stall rather than a steady process is important, because it tells you the fault is intermittent and will recur, which changes the fix from a reboot to a root-cause hunt.

In a cloud SCADA such as Merobix, the scope of a simultaneous flat is easy to see because every tag carries its device, poll group, and site, and the historian keeps recording independently of any one display. An engineer can line up the flat tags against the still-moving ones, confirm the shared boundary, and compare the live view to the recorded history in the same tool. The platform makes the scope map, which is the entire first move of this tree, a matter of looking rather than assembling.

When to Escalate

Escalate to the communication or infrastructure owner when the flat spans multiple sites or an entire poll engine, because that is a shared-infrastructure failure rather than a site problem and it needs the head end, the network, or the carrier engaged. Working individual tags during an infrastructure flat wastes effort on symptoms of one upstream cause. Treat a wide flat as a single incident and chase the common element, not the many flat pens.

Escalate to a field dispatch only after the scope map points at a specific device or site and remote recovery has failed. If one controller's tags flatlined, the poll is proven healthy, and no remote reset restores it, that device needs hands on it. Hand the responder the scope map and the timestamp of the flat so they know exactly which device stopped and when, rather than arriving to diagnose a whole site when the tree already named one box.

Frequently Asked Questions