How to Verify Time Sync Across the SCADA Stack

Back to Blog

Establish the One Reference Clock

Everything starts with a single source of truth, so first identify and verify the reference clock the whole stack is supposed to follow. On most sites that is an NTP server, or a GPS-disciplined clock feeding NTP, and on stacks needing tighter alignment it may be PTP. Confirm the reference itself is actually locked to its upstream source and holding good time, because a stack perfectly synchronized to a reference that has drifted is perfectly, uniformly wrong. Verify the master before you check anything that follows it.

Confirm every device is pointed at that same reference and not at a second, disagreeing one. A classic failure is half the stack syncing to one NTP server and half to another, or a device quietly falling back to its own free-running clock when it cannot reach the server. Walk the configuration of each layer and confirm the time source is the intended single reference, because two references are functionally the same as none once they diverge.

Decide your tolerance from what the data is for. A historian trend that people glance at tolerates far looser sync than a sequence-of-events analysis meant to establish which of two nearly simultaneous events came first. Know how tight your alignment has to be before you start measuring, because the answer to whether the stack is synchronized enough is entirely relative to what you are trying to prove with the timestamps, and that ties directly to how you read a SCADA trend and any event order built on it.

Check Each Layer Against the Reference

Work down the stack layer by layer, comparing each device's clock to the reference. Start at the controllers: read the current time from each PLC and RTU and compare it to the reference at the same moment, noting the offset. A device that is seconds off is a device whose event timestamps will not line up with its neighbors, and on a protocol that timestamps events at the source, this offset propagates into every event that device reports. This is the same class of check as verifying DNP3 time sync, applied across the whole heterogeneous stack rather than one protocol.

Then check the servers and clients. The historian stamps or records timestamps, so confirm its clock matches the reference, and confirm whether it trusts device-supplied timestamps or applies its own on receipt, because that single design choice changes what a mismatch means. The HMI clock matters too, since operators read event times off it and a skewed HMI makes an operator's account of an incident disagree with the record. Every place a timestamp is created or displayed is a place a wrong clock does damage.

Distinguish a source timestamp from a receipt timestamp everywhere it matters. An event timestamped when it happened at the RTU carries a different meaning from one stamped when the master received it, and mixing the two silently reorders events. Confirm, per data path, which timestamp the historian is keeping, because a store-and-forward recovery or a comms gap can deliver old events long after they occurred. This interacts directly with store-and-forward buffering, where buffered events must carry their original source time or the recovered data lands in the archive with the wrong timeline entirely.

Prove Event Order End to End

Inject one known event and follow it through. Force a single, unambiguous change - a test bit, a controlled state flip - and read its timestamp everywhere it is recorded: at the device, in the historian, on the HMI. If all three agree within tolerance, the chain for that path is proven; if the historian shows it a second later than the device, you have found either a real clock offset or a receipt-versus-source timestamp confusion, and either one matters. One deliberate event tests the whole path far better than staring at configuration.

Check relative order on a pair of events across two devices, because that is what sequence analysis actually depends on. Cause two events you know the true order of, one on each of two controllers, and confirm the recorded timestamps preserve that order. If clock skew between the two devices exceeds the real time between the events, the record will show them in the wrong order, which is the exact failure that makes a sequence of events blame the wrong cause. Normalizing these onto a common timeline is the job covered in normalizing historian timestamps.

Common Mistakes to Avoid

The first mistake is verifying the devices while trusting an unverified reference clock, so the whole stack is uniformly synchronized to the wrong time. Always confirm the master is locked to good upstream time first. The second is missing that two different time references are in play, with half the stack on each, which looks fine until the two references drift apart and event order across them quietly falls apart.

The third mistake is ignoring the difference between source and receipt timestamps, which mixes when-it-happened with when-it-arrived and silently reorders events, especially after a store-and-forward recovery delivers old data. The fourth is checking sync once at commissioning and never again, when a clock that loses its upstream source drifts steadily afterward. A device that free-runs after losing NTP looks fine for a while and is wrong by more every day, so re-verify periodically rather than assuming a one-time check holds.

Frequently Asked Questions