How to Stage a SCADA Config Change With Rollback

Back to Blog

Back Up and Record the Current State

Before you change anything, capture the system as it is right now, and verify the capture. A backup you assume is good but never checked is not a rollback plan, it is a hope. Confirm the backup completed, covers the components you are about to touch, and is stored somewhere the change itself cannot corrupt. If your backups are scheduled and automated, this is the moment to confirm the latest one is current and restorable, which is exactly the discipline behind a scheduled SCADA backup that has actually been restore-tested.

Record the specific current values you are about to change, not just the whole-system backup. If you are widening a deadband, write down the old number. If you are remapping a register, capture the old mapping. This targeted record is what lets you reverse a small change quickly without a full restore, and it is far faster to reapply three noted values than to roll an entire configuration back to last night. The full backup is your safety net; the noted current state is your fast undo.

Snapshot the observable behavior too, so you have a before picture to compare against. Note what the affected tags read, whether the relevant alarms are clear, and what the trend looks like going into the change. When you verify afterward, you are checking that behavior returned to or improved on this baseline, and without the baseline you cannot tell a pre-existing quirk from something your change caused.

Write the Rollback Before You Change Anything

Write the rollback first, in the same detail as the change itself, and this is the step people skip under time pressure. For every action in your change, write the exact reversing action: set this value back to that, restore this mapping, re-enable this task. A rollback plan that says restore from backup is a blunt last resort; a good plan lists the specific reversing steps so you can back out cleanly and fast, ideally without disturbing anything you did not touch.

Define the trigger for rolling back before you are in the heat of it. Decide in advance what counts as failure - a tag that will not update, an alarm storm, a comms drop, a verification step that will not pass - and commit to rolling back if you hit it rather than pressing on to fix a live system by trial and error. Having the abort criteria written down turns a stressful judgment call into a checklist decision, which is exactly when you want it to be a checklist decision.

Time-box the change and the rollback together. Know how long the change should take, how long the rollback takes, and ensure both fit inside the window with margin. A change you can apply in five minutes but cannot undo in under an hour is a change you should not start late in a window. This planning is the operational core of a formal management-of-change process, and the same rigor belongs in your broader configuration validation for larger changes.

Apply in a Window and Verify Against a Checklist

Apply the change inside a scheduled maintenance window with operations informed, never as a casual live edit. Even a small change can ripple - a remap that shifts a register, a deadband that changes an alarm's behavior - and operations needs to know that if a value goes odd or an alarm behaves differently for a few minutes, it is planned work, not a real event. Announce the window, the affected points, and what operators might see, so a normal side effect is not mistaken for a fault.

Make the change as a set of deliberate, one-at-a-time steps, verifying each before the next where the change has multiple parts. Batching several edits and applying them all at once means that if something breaks you do not know which edit did it, which makes both diagnosis and rollback harder. Sequential application with a check between steps localizes any failure to the last thing you did.

Verify against a written checklist that covers both the intended effect and the collateral. Confirm the thing you changed now does what you wanted, then confirm the things around it still work - the affected tags update, their alarms behave, dependent calculations and trends still read right. Use the ordered verification below so nothing gets skipped under the pressure of an open window:

1. The changed setting now shows the intended value and behavior.
2. The directly affected tags are updating with good quality.
3. Alarms on those tags annunciate and clear correctly.
4. Dependent calculated tags and trends read sensibly.
5. Comms to the affected devices are healthy, no new faults.
6. The change is logged in the audit trail with who, what, and when.

Verifying the Change and the Undo

Prove the change did only what you intended by comparing against the baseline you captured before starting. The intended behavior should be present and everything else should match the before picture. A change that fixed its target but quietly altered a neighboring point's behavior is a partial failure, and only the before-and-after comparison catches the collateral damage that the change's own success can mask.

Confirm the change landed in the audit trail so the record matches reality. Every configuration change should leave a logged entry with who made it, what changed, and when, both for accountability and so the next engineer can see why the system is the way it is. Tie this into your SCADA audit trail so a staged change is self-documenting. If you did have to roll back, verify the rollback with the same checklist you would have used for the change, because a botched rollback is just another failed change.

Common Mistakes to Avoid

The cardinal mistake is changing a running configuration with no verified backup and no written rollback, betting that nothing will go wrong. Something eventually does, and then you are debugging a broken plant live with no way back. Always back up, verify the backup, and write the specific reversing steps before you touch anything. The second mistake is batching many edits at once, which makes it impossible to tell which one broke things.

The third mistake is having no defined abort criteria, so a change that is clearly going wrong turns into an escalating live-repair session instead of a clean rollback. Decide the failure triggers in advance and commit to backing out when you hit them. The fourth is skipping operations notification, so a normal, planned side effect of the change gets reported and reacted to as a real process event, turning routine work into a false alarm.

Frequently Asked Questions