Alarm Integrity:
Making Sure Critical Alerts Arrive

Back to Blog

Part of the SCADA Security Hub - 60+ guides on SCADA security, compliance & certifications.

Why Alarm Delivery Failure Is a Silent Risk

There are two ways an alarm system fails. The first is a detection failure - the condition happens and no alarm is raised. Operators worry about this one constantly, and rightly so. The second is a delivery failure - the alarm is raised correctly, logged correctly, painted red on the HMI correctly, and the SMS or email that was supposed to wake someone up never arrives. The second failure is worse in one specific way: nothing in the system looks broken. The alarm record exists. The historian shows the excursion. The dashboard shows the notification as sent. The only evidence of failure is an operator who slept through the night while a pump ran dry.

Delivery failures are common because the delivery path runs through infrastructure the SCADA vendor does not own. SMS aggregators have outages and, increasingly, silently filter messages under carrier anti-spam rules for application-to-person traffic. Corporate email gateways change their filtering policy and start quarantining alerts that arrived fine for two years. Provider API keys expire. A notification service crashes mid-dispatch and loses everything that was in memory. A contact list still points to the phone of an operator who left six months ago. None of these produce an error on an HMI screen.

Post-incident reviews across industries keep finding the same line: the alarm fired, but the right person never knew. If your operation depends on remote alerting - and every lightly-staffed or unmanned site does - then alarm delivery is a protective layer, and it deserves the same engineering rigor as the alarm logic itself.

The Delivery Pipeline: From Detection to a Human

To reason about alarm integrity, break the path into stages. A critical alert typically traverses:

• Detection - the alarm engine evaluates the condition and creates an alarm event.
• Enqueue - a notification record is created for each recipient and channel.
• Dispatch - a worker calls the provider API: SMTP relay, SMS gateway, push service, webhook endpoint, or SIEM connector.
• Provider handoff - the provider accepts the message and forwards it into the carrier or mail network.
• Network delivery - the carrier or mail infrastructure moves the message to the device or inbox.
• Human receipt and acknowledgement - someone sees it, understands it, and confirms they own it.

Each hop has a distinct failure mode. Enqueue fails when the process crashes between raising the alarm and persisting the notification. Dispatch fails on expired credentials, rate limits, or a provider outage. Provider handoff fails silently when a message is accepted with a 200 response and then filtered downstream - the platform believes it delivered, and it did not. Human receipt fails when the phone is on silent, the recipient is on vacation, or the message drowned in a flood of forty other alerts. A trustworthy alerting design puts a control at every one of these hops, which is exactly what the rest of this guide walks through.

The Durable Outbox: Never Lose an Alert Between Systems

The foundational control is the transactional outbox pattern. Instead of firing a notification directly from the alarm-processing code - where a crash, restart, or network blip loses the message forever - the system writes a notification record to durable storage in the same database transaction that records the alarm event. A separate dispatcher process then works through the outbox, attempts delivery on each channel, and records the outcome of every attempt against the record.

This buys you three properties that fire-and-forget alerting cannot offer:

• Crash safety. If the server dies after the alarm is raised but before the SMS goes out, the pending record is still in the outbox when the dispatcher restarts. Nothing evaporates in memory.
• At-least-once semantics. The record is not marked complete until a delivery attempt succeeds or the retry budget is exhausted - so the failure mode shifts from "silently lost" to "visibly stuck," which is a failure you can alarm on.
• An audit trail. Every critical alert has a row showing when it was generated, each attempt, each provider response, and its final state. When a regulator or an incident review asks whether the 2 AM alarm was delivered, you answer from records, not memory.

This is the architecture Merobix ships today: security and operational notifications flow through a durable notification outbox with email, SMS, webhook, and SIEM delivery, and every attempt is tracked with retry and dead-letter handling. Whatever platform you evaluate, ask the vendor to show you the notification record for a failed delivery - a vendor with a real outbox can pull it up in seconds.

Retries, Backoff, and Dead-Letter Queues

Most delivery failures are transient - a provider blip, a timeout, a rate limit. The correct response is to retry, but naive retry loops make outages worse. Mature implementations use exponential backoff with jitter: wait a little, then longer, then longer still, with randomness so a thousand queued alerts do not hammer a recovering provider in the same second. Just as important is error classification: a timeout deserves a retry; an invalid phone number or a hard bounce does not, and retrying it forty times only delays the moment a human learns the contact record is wrong.

When the retry budget is exhausted, the notification must not be deleted. It moves to a dead-letter queue - a holding area for messages the system could not deliver. Two rules make dead-letter queues useful rather than decorative:

• The dead-letter queue must itself be monitored. A DLQ nobody watches is just a slower way of dropping messages. Any critical alert landing there should raise its own operational alarm through an independent path.
• There must be a reconciliation process. Someone - or something - periodically compares what was supposed to be sent against what providers confirm was delivered, replays anything recoverable, and fixes the root cause of anything that is not. Retries also mean the same alert can occasionally arrive twice, so reconciliation should recognize duplicates rather than letting them erode operator trust.

Ask vendors what happens to an alert after the final failed retry. "It's logged" is a red flag; you want a dead-letter store, an alarm on that store, and a documented replay procedure. This is the same delivery discipline that applies to security event export - our guide to SCADA threat detection and SIEM integration covers why configurable SMS and email alerts matters just as much for security alerts as for process alarms.

Provider Canaries: Proving the Path Actually Works

An outbox with retries proves your platform handed the message to the provider. It cannot prove the provider delivered it. SMS filtering in particular is invisible from the sender's side: the API returns success, the carrier discards or delays the message, and no error ever comes back. The only way to close this gap is to test the path with real traffic - which is what a provider canary does.

A canary is a synthetic alert sent through the production pipeline on a schedule - the same code path, the same provider account, the same message class as a genuine critical alarm - addressed to a monitored inbox or phone number the platform controls. The receiving side confirms arrival and measures end-to-end latency. If a canary fails to arrive within its window, the platform knows the provider path is degraded before a real alarm needs it, and can alert operations staff through an independent channel or fail over to a backup provider.

Canaries pair with a second concept: sticky degraded status. When a provider fails a canary or a burst of real deliveries, the platform should mark that provider degraded and keep it marked until it proves healthy again - not flap back to green the first time a single message squeaks through. Sticky status prevents the worst pattern in delivery health: a provider that is 60 percent reliable being treated as fine because the last check happened to pass.

Merobix is building both into its alert-delivery assurance roadmap - provider canaries, sticky degraded status, and duplicate reconciliation, layered on the shipped outbox and the platform's existing health surfaces for API dependencies and telemetry freshness. When evaluating any vendor, the question to ask is simple: how would your platform know if your SMS provider stopped delivering, and how long would it take to find out? Silence, or "we'd hear from customers," tells you everything.

Escalation Chains, On-Call, and the Acknowledgement Loop

Delivering the message to a phone is still not the goal - the goal is a human taking ownership of the problem. That requires an acknowledgement loop: the recipient confirms receipt in the platform, the acknowledgement is timestamped and audited, and the escalation clock stops. No acknowledgement means the system assumes the message failed at the human hop and moves down the chain.

A workable escalation design looks like this:

• Tier 1: the on-duty operator, notified on at least two channels (for example push and SMS). Ack window of 5–15 minutes for critical alarms, tuned to how fast the process can hurt someone or something.
• Tier 2: a second qualified responder - backup operator or lead - notified automatically when the window expires, ideally adding a more intrusive channel such as a voice call.
• Tier 3: a supervisor or on-call manager, at which point the event is treated as both a process problem and a staffing-response problem.

The on-call rotation behind the chain matters as much as the chain itself. Rotations should transfer automatically at shift handoff so a critical alarm never routes to whoever was on duty last Tuesday, every tier should have more than one reachable person, and contact details should be reviewed on a schedule - stale phone numbers are one of the most common delivery failures and the cheapest to fix. Acknowledging from a phone in the field also has security implications: see our guide to mobile SCADA access security for doing that without widening your attack surface, and our OT incident response plan guide for what happens after the acknowledgement, when the alert turns out to be an incident.

Choosing Channels: No Single Path Is Reliable Enough

Every delivery channel fails differently, which is the argument for using more than one on anything critical:

Channel	Typical Latency	Common Silent Failure	Delivery Confirmation	Best Role
SMS	Seconds to minutes	Carrier anti-spam filtering; aggregator outages	Partial (delivery receipts vary by carrier)	Primary wake-up channel for critical alarms
Email	Seconds to hours	Spam/quarantine filtering; mailbox rules	Weak (bounce only)	Detail-rich follow-up; non-urgent notifications
Push notification	Seconds	App uninstalled; OS notification settings; token expiry	Good (app can confirm receipt)	Fast primary channel with in-app acknowledgement
Voice call	Seconds	Voicemail; call screening	Good (answer detection)	Escalation tier - hard to sleep through
Webhook / SIEM	Seconds	Endpoint down; certificate or auth drift	Strong (HTTP response, retry on failure)	Machine-to-machine handoff to SOC and ITSM tooling

The pattern that falls out of the table: critical alarms should ride at least two channels with different failure domains - SMS plus push, or push plus voice on escalation - while webhooks feed the tooling your security and operations teams already watch. A platform built on an outbound-only gateway architecture (as covered in our secure SCADA for OT networks guide) handles all of this from the cloud side, so alerting keeps working even when the site itself is unreachable and telemetry is riding store-and-forward buffering.

Alarm Fatigue Is an Integrity Problem Too

Delivery engineering fails at the last hop if the human has learned to ignore the channel. An operator who receives two hundred notifications a shift will miss the one that matters - not because the SMS failed, but because attention is a finite resource and nuisance alarms spent it. Alarm-management standards such as ISA-18.2 - adopted internationally as IEC 62682, published by the IEC - exist precisely because unrationalized alarm loads have contributed to major accidents.

Integrity and fatigue are two sides of one design problem:

• Rationalize before you notify. Only alarms that require a human action deserve a phone interruption. Status changes and informational events belong on dashboards or in digests.
• Prioritize the pipeline. Critical alarms take the guaranteed multi-channel path with escalation; low-priority events must never queue ahead of them or share their retry budget.
• Suppress storms without dropping criticals. Flood suppression and deduplication are essential during an upset, but the suppression logic must be priority-aware - collapsing fifty repeats of one alarm is good; swallowing a new high-priority alarm inside a storm window is a delivery failure by another name.

A useful audit: pull one week of notifications and count what fraction required action. If it is under half, your alarm philosophy - not your SMS provider - is the biggest threat to delivery integrity.

Testing Alarm Paths and What to Ask Vendors

Alarm delivery is a protective layer, and protective layers get proof-tested. A reasonable regime: a full end-to-end drill at least quarterly and after any change to providers, contacts, or escalation rules - trigger a designated test alarm, confirm arrival on real operator devices on every configured channel, let the first tier deliberately ignore it and verify the escalation timers fire, then acknowledge and verify the chain stops. Between drills, automated canaries exercise the provider path continuously. Log the results; auditors and insurers increasingly ask for them.

When evaluating a cloud SCADA platform, these seven questions separate real delivery engineering from a checkbox that says "SMS alerts":

1. Is there a durable notification outbox? Show me the stored record for a delivered alert and for a failed one.
2. What is the retry policy? Backoff, jitter, and how transient errors are distinguished from permanent ones.
3. Where do permanently failed alerts go? Dead-letter store, monitoring on that store, and the replay procedure.
4. How do you detect a silently failing provider? Canaries, delivery receipts, degraded-status handling - or nothing.
5. What escalation logic is built in? Ack windows, multi-tier chains, multi-channel per tier, automatic rotation handoff.
6. Is every notification and acknowledgement audited? You will need this after an incident.
7. Can I run a live delivery test right now, in the demo? A confident vendor will send you the SMS while you watch.

Frequently Asked Questions

Sources & Further Reading

• ISA18 Committee - Alarm Systems Standards (ISA-18.2, Management of Alarm Systems for the Process Industries) (International Society of Automation)
• IEC 62682:2022 - Management of Alarm Systems for the Process Industries (available from the IEC Webstore)