When a broker or a region goes down and comes back, every gateway that was connected to it wants back in at the same moment, and if they all retry immediately and in lockstep, they can knock the recovering service straight back over. Reconnect backoff is the strategy that stops this: each client waits a growing, randomized interval before retrying, so a fleet reconnects as a spread-out trickle rather than a synchronized flood. This guide explains what reconnect backoff is, how exponential growth and jitter tame the retry storm, and why it matters when hundreds of telemetry clients recover together.
Reconnect backoff in one line: Reconnect backoff is a reconnection strategy where a telemetry client, after losing its connection, waits before retrying and lengthens that wait with each failed attempt, usually growing it exponentially up to a cap and adding random jitter. Spreading retries out over time and randomizing them stops hundreds of gateways from hammering a broker simultaneously after an outage, which would otherwise create a thundering herd that prevents the service from recovering.
Consider what happens when a broker that serves a large fleet goes offline briefly. Every connected gateway notices the loss at roughly the same time and, wanting to restore its telemetry, tries to reconnect. If each one retries immediately, and keeps retrying immediately, then the instant the broker comes back it is hit by the entire fleet at once - a wall of simultaneous connection attempts far larger than its normal steady-state load. This synchronized surge is the thundering herd, and it can overwhelm the very service that was just recovering, pushing it back down.
The problem is made worse by synchronization. Because all the clients lost the connection together, their retries are naturally aligned, and naive fixed-interval retries keep them aligned - they all wait the same amount and all try again at the same instant, over and over, in lockstep. Even if the broker survives the first wave, it faces repeated synchronized waves at each retry interval. A retry storm like this can turn a short outage into a long one, because the service cannot get enough breathing room to stabilize before the next surge arrives.
The fix has two parts, and both are needed. First, clients should back off - wait progressively longer between attempts - so the pressure eases rather than staying constant. Second, clients should stagger themselves so their retries do not all land at the same moment. Backing off without staggering still leaves synchronized waves, just spaced further apart; staggering without backing off still leaves constant pressure. Combining a growing wait with randomization gives both a falling load and a spread-out one.
Exponential backoff is the standard way to make the wait grow. After the first failed attempt the client waits a short base interval; after the next it waits roughly double; after the next, double again, and so on. The wait rises quickly, so a client that keeps failing backs off fast and stops adding much load, while a client whose first or second retry succeeds recovers promptly. Because it can grow without bound, the interval is usually held to a maximum cap so a client keeps retrying at some sensible ceiling rather than eventually waiting hours between attempts.
Backoff alone does not break the synchronization, though. If every client doubles its wait on the same schedule, they remain aligned and still surge together, just at the doubled intervals. Jitter is the second ingredient: each client adds a random amount to its wait so that no two clients retry at exactly the same instant. The randomization scatters the reconnection attempts across a window of time, converting what would be a spike into a spread. It is the jitter, more than the backoff, that actually disperses the herd.
Together the two produce well-behaved recovery. Early retries are quick, so a client that can reconnect does so without much delay and the fleet largely restores itself soon after the service returns. Clients that keep failing back off exponentially so they stop piling on, and the jitter ensures that even the clients retrying at the same backoff stage do not land simultaneously. The fleet reconnects as a smooth, decaying trickle rather than a series of synchronized hammer blows, which lets the recovering service absorb the load and stabilize.
This is fundamentally a fleet-scale concern, which distinguishes it from the retries a single gateway does on an individual poll. A device retrying one failed read talks to one field device and affects only itself; reconnect backoff is about how a whole population of clients behaves toward a shared service when they all lose it at once. The larger the fleet and the more it shares a single broker or region, the more important the strategy becomes, because the synchronized surge grows with the number of clients. At small scale a crude retry may cause no harm; at large scale it can be the difference between quick recovery and a prolonged outage.
It matters most precisely when things are already going wrong. A regional network problem, a broker restart, or a cloud disruption can knock a large fraction of a fleet offline together, and that is the exact moment when a thundering herd would do the most damage - hitting a fragile, recovering service with a synchronized flood. Good reconnect backoff is therefore part of designing for graceful degradation: the fleet should ease itself back rather than stampede, so a partial outage recovers cleanly instead of cascading into something worse.
For a cloud SCADA platform such as Merobix, where many field gateways connect to shared cloud infrastructure, sensible reconnect behaviour in the gateway clients protects the whole system's ability to recover. Each gateway backing off exponentially with jitter after a disconnection means that when connectivity returns, the platform sees a manageable, spread-out wave of reconnections rather than every site slamming in at once. That keeps recovery fast and stable at fleet scale, and it is the kind of well-behaved client strategy that lets a monitoring platform ride out an outage and come back cleanly, no matter how many sites were affected.
It is the surge that happens when many clients lose a shared service at the same time and all try to reconnect simultaneously the moment it returns, hitting it with far more connection attempts than its normal load. That synchronized flood can overwhelm the recovering service and push it back down, turning a short outage into a long one. Reconnect backoff with jitter exists to break up this synchronized surge into a spread-out trickle.
Because backoff alone keeps clients synchronized: if every client doubles its wait on the same schedule they still retry at the same instants, just less often. Jitter adds a random amount to each client's wait so no two retry at exactly the same moment, scattering the attempts across a window of time. It is the jitter that actually disperses the herd, converting a synchronized spike into a smooth spread; the exponential growth reduces the overall load.
Per-poll retry is about a single gateway retrying an individual failed read to one field device, affecting only that device and that gateway. Reconnect backoff is a fleet-scale strategy about how a whole population of clients behaves toward a shared broker or cloud service when they all lose it together. The concern is the synchronized surge across many clients, which grows with fleet size, rather than the serial retries of one poll.
Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.