Every SCADA platform can generate an alarm. Far fewer can prove the alarm actually reached a human. Between the moment a tank level crosses its high-high limit and the moment a phone buzzes sits a chain of queues, provider APIs, carrier networks, and contact lists - and every link can fail silently while the dashboard stays green. This guide covers the engineering that makes alarm delivery trustworthy: durable notification outboxes, retry and dead-letter handling, provider canaries, sticky degraded status, escalation chains, and acknowledgement loops - plus how to test your alarm paths and what to demand from any cloud SCADA vendor.
Part of the SCADA Security Hub - 60+ guides on SCADA security, compliance & certifications.
There are two ways an alarm system fails. The first is a detection failure - the condition happens and no alarm is raised. Operators worry about this one constantly, and rightly so. The second is a delivery failure - the alarm is raised correctly, logged correctly, painted red on the HMI correctly, and the SMS or email that was supposed to wake someone up never arrives. The second failure is worse in one specific way: nothing in the system looks broken. The alarm record exists. The historian shows the excursion. The dashboard shows the notification as sent. The only evidence of failure is an operator who slept through the night while a pump ran dry.
Delivery failures are common because the delivery path runs through infrastructure the SCADA vendor does not own. SMS aggregators have outages and, increasingly, silently filter messages under carrier anti-spam rules for application-to-person traffic. Corporate email gateways change their filtering policy and start quarantining alerts that arrived fine for two years. Provider API keys expire. A notification service crashes mid-dispatch and loses everything that was in memory. A contact list still points to the phone of an operator who left six months ago. None of these produce an error on an HMI screen.
Post-incident reviews across industries keep finding the same line: the alarm fired, but the right person never knew. If your operation depends on remote alerting - and every lightly-staffed or unmanned site does - then alarm delivery is a protective layer, and it deserves the same engineering rigor as the alarm logic itself.
To reason about alarm integrity, break the path into stages. A critical alert typically traverses:
Each hop has a distinct failure mode. Enqueue fails when the process crashes between raising the alarm and persisting the notification. Dispatch fails on expired credentials, rate limits, or a provider outage. Provider handoff fails silently when a message is accepted with a 200 response and then filtered downstream - the platform believes it delivered, and it did not. Human receipt fails when the phone is on silent, the recipient is on vacation, or the message drowned in a flood of forty other alerts. A trustworthy alerting design puts a control at every one of these hops, which is exactly what the rest of this guide walks through.
The foundational control is the transactional outbox pattern. Instead of firing a notification directly from the alarm-processing code - where a crash, restart, or network blip loses the message forever - the system writes a notification record to durable storage in the same database transaction that records the alarm event. A separate dispatcher process then works through the outbox, attempts delivery on each channel, and records the outcome of every attempt against the record.
This buys you three properties that fire-and-forget alerting cannot offer:
This is the architecture Merobix ships today: security and operational notifications flow through a durable notification outbox with email, SMS, webhook, and SIEM delivery, and every attempt is tracked with retry and dead-letter handling. Whatever platform you evaluate, ask the vendor to show you the notification record for a failed delivery - a vendor with a real outbox can pull it up in seconds.
Most delivery failures are transient - a provider blip, a timeout, a rate limit. The correct response is to retry, but naive retry loops make outages worse. Mature implementations use exponential backoff with jitter: wait a little, then longer, then longer still, with randomness so a thousand queued alerts do not hammer a recovering provider in the same second. Just as important is error classification: a timeout deserves a retry; an invalid phone number or a hard bounce does not, and retrying it forty times only delays the moment a human learns the contact record is wrong.
When the retry budget is exhausted, the notification must not be deleted. It moves to a dead-letter queue - a holding area for messages the system could not deliver. Two rules make dead-letter queues useful rather than decorative:
Ask vendors what happens to an alert after the final failed retry. "It's logged" is a red flag; you want a dead-letter store, an alarm on that store, and a documented replay procedure. This is the same delivery discipline that applies to security event export - our guide to SCADA threat detection and SIEM integration covers why guaranteed delivery matters just as much for security alerts as for process alarms.
An outbox with retries proves your platform handed the message to the provider. It cannot prove the provider delivered it. SMS filtering in particular is invisible from the sender's side: the API returns success, the carrier discards or delays the message, and no error ever comes back. The only way to close this gap is to test the path with real traffic - which is what a provider canary does.
A canary is a synthetic alert sent through the production pipeline on a schedule - the same code path, the same provider account, the same message class as a genuine critical alarm - addressed to a monitored inbox or phone number the platform controls. The receiving side confirms arrival and measures end-to-end latency. If a canary fails to arrive within its window, the platform knows the provider path is degraded before a real alarm needs it, and can alert operations staff through an independent channel or fail over to a backup provider.
Canaries pair with a second concept: sticky degraded status. When a provider fails a canary or a burst of real deliveries, the platform should mark that provider degraded and keep it marked until it proves healthy again - not flap back to green the first time a single message squeaks through. Sticky status prevents the worst pattern in delivery health: a provider that is 60 percent reliable being treated as fine because the last check happened to pass.
Merobix is building both into its alert-delivery assurance roadmap - provider canaries, sticky degraded status, and duplicate reconciliation, layered on the shipped outbox and the platform's existing health surfaces for API dependencies and telemetry freshness. When evaluating any vendor, the question to ask is simple: how would your platform know if your SMS provider stopped delivering, and how long would it take to find out? Silence, or "we'd hear from customers," tells you everything.
Delivering the message to a phone is still not the goal - the goal is a human taking ownership of the problem. That requires an acknowledgement loop: the recipient confirms receipt in the platform, the acknowledgement is timestamped and audited, and the escalation clock stops. No acknowledgement means the system assumes the message failed at the human hop and moves down the chain.
A workable escalation design looks like this:
The on-call rotation behind the chain matters as much as the chain itself. Rotations should transfer automatically at shift handoff so a critical alarm never routes to whoever was on duty last Tuesday, every tier should have more than one reachable person, and contact details should be reviewed on a schedule - stale phone numbers are one of the most common delivery failures and the cheapest to fix. Acknowledging from a phone in the field also has security implications: see our guide to mobile SCADA access security for doing that without widening your attack surface, and our OT incident response plan guide for what happens after the acknowledgement, when the alert turns out to be an incident.
Every delivery channel fails differently, which is the argument for using more than one on anything critical:
| Channel | Typical Latency | Common Silent Failure | Delivery Confirmation | Best Role |
|---|---|---|---|---|
| SMS | Seconds to minutes | Carrier anti-spam filtering; aggregator outages | Partial (delivery receipts vary by carrier) | Primary wake-up channel for critical alarms |
| Seconds to hours | Spam/quarantine filtering; mailbox rules | Weak (bounce only) | Detail-rich follow-up; non-urgent notifications | |
| Push notification | Seconds | App uninstalled; OS notification settings; token expiry | Good (app can confirm receipt) | Fast primary channel with in-app acknowledgement |
| Voice call | Seconds | Voicemail; call screening | Good (answer detection) | Escalation tier - hard to sleep through |
| Webhook / SIEM | Seconds | Endpoint down; certificate or auth drift | Strong (HTTP response, retry on failure) | Machine-to-machine handoff to SOC and ITSM tooling |
The pattern that falls out of the table: critical alarms should ride at least two channels with different failure domains - SMS plus push, or push plus voice on escalation - while webhooks feed the tooling your security and operations teams already watch. A platform built on an outbound-only gateway architecture (as covered in our secure SCADA for OT networks guide) handles all of this from the cloud side, so alerting keeps working even when the site itself is unreachable and telemetry is riding store-and-forward buffering.
Delivery engineering fails at the last hop if the human has learned to ignore the channel. An operator who receives two hundred notifications a shift will miss the one that matters - not because the SMS failed, but because attention is a finite resource and nuisance alarms spent it. Alarm-management standards such as ISA-18.2 - adopted internationally as IEC 62682, published by the IEC - exist precisely because unrationalized alarm loads have contributed to major accidents.
Integrity and fatigue are two sides of one design problem:
A useful audit: pull one week of notifications and count what fraction required action. If it is under half, your alarm philosophy - not your SMS provider - is the biggest threat to delivery integrity.
Alarm delivery is a protective layer, and protective layers get proof-tested. A reasonable regime: a full end-to-end drill at least quarterly and after any change to providers, contacts, or escalation rules - trigger a designated test alarm, confirm arrival on real operator devices on every configured channel, let the first tier deliberately ignore it and verify the escalation timers fire, then acknowledge and verify the chain stops. Between drills, automated canaries exercise the provider path continuously. Log the results; auditors and insurers increasingly ask for them.
When evaluating a cloud SCADA platform, these seven questions separate real delivery engineering from a checkbox that says "SMS alerts":
Key takeaway: An alarm that fires but never reaches a human is operationally identical to an alarm that never fired - and it fails without any visible symptom. Demand delivery infrastructure you can inspect: a durable outbox, monitored dead-letter handling, provider health checks, and audited escalation. Merobix documents its notification and detection architecture on the security page, and you can watch an alert travel from alarm condition to phone in a live demo.
SCADA alarm notifications fail for reasons that rarely show up in the SCADA system itself: SMS providers suffer outages or silently filter messages under carrier anti-spam rules, email lands in junk folders or is rejected by a changed corporate gateway, API credentials for the notification provider expire, notification queues back up or crash mid-dispatch, and escalation contacts go stale after staff changes. Because the alarm was correctly generated, dashboards show green while the message never reaches a phone. That is why delivery needs its own instrumentation - durable outbox records, provider status checks, and synthetic canary alerts that prove the full path end to end.
A notification outbox is a durable record of every alert the system intends to send, written to the database in the same transaction as the alarm event itself. A separate dispatcher then works through the outbox, calling the email, SMS, or webhook provider and recording the outcome of each attempt. If the server crashes, restarts, or loses connectivity mid-send, the outbox still holds the pending notification and the dispatcher picks it up again. This gives at-least-once delivery semantics instead of fire-and-forget, and it produces an auditable trail showing exactly when each critical alert was generated, attempted, delivered, or exhausted into a dead-letter queue.
A provider canary is a synthetic test alert sent through the real notification pipeline at a regular interval - a scheduled message that travels the same code path, provider API, and carrier network as a genuine critical alarm, arriving at a monitored inbox or phone number. If the canary fails to arrive within its expected window, the platform knows the provider path is broken before a real alarm needs it, and can raise a delivery-health alert or fail over to a backup channel. Canaries convert silent delivery failure into a detectable event. Merobix is building provider canaries into its alert-delivery assurance roadmap on top of its shipped notification outbox.
Structure escalation in tiers with explicit acknowledgement windows. Tier one notifies the on-duty operator on at least two channels, such as push and SMS. If the alarm is not acknowledged within a defined window - commonly 5 to 15 minutes for critical alarms - the system automatically escalates to a second responder, then to a supervisor or on-call manager. Every tier should have more than one named person reachable through more than one channel, rotations should transfer automatically at shift handoff, and acknowledgements should be recorded in the audit trail so post-incident reviews can verify who was notified, when, and who responded.
Test critical alarm delivery end to end at least quarterly, and after any change to notification providers, contact lists, or escalation rules. A real test triggers a designated test alarm and verifies the message arrives on actual operator devices across every configured channel, that escalation timers fire when the first tier ignores the alert, and that acknowledgement clears the escalation. Between full drills, automated canary alerts can exercise the provider path continuously. Treat a failed test exactly like a failed safety-system proof test: the alarm path is a protective layer, and an untested path should be assumed broken.
Durable notification outbox, retry and dead-letter handling, multi-channel alerts, and an audit trail for every message - watch a critical alarm travel from condition to phone, live.