How to Configure Historian Collector Compression

Back to Blog

Understand the Two Stages Before You Touch a Number

Collector compression is usually two filters in series, and configuring it means knowing which one you are adjusting. The first is the exception deadband at the collector: it decides whether a new sample is different enough from the last sent value to bother sending it upstream at all. The second is the compression, often a swinging-door algorithm, at the archive: it decides whether a sample that arrived actually needs to be stored or whether it lies on a straight line between its neighbors and can be dropped. Both discard data; they do it at different points and for different reasons.

The swinging-door idea is worth internalizing because it is not a simple deadband. It keeps a point only when the value deviates from the straight line drawn through the last stored point by more than the compression deviation, which means it preserves slopes and trends far more faithfully than a fixed band would for the same storage. The mechanism and the trade-offs behind it are exactly what the concept of deadband compression in historians describes, and understanding it stops you from setting the compression deviation as if it were a plain amplitude deadband.

Know which stage you are solving before you turn a knob. If the trend is missing fast transients, the exception setting may be dropping them before they ever reach the archive, and no amount of archive tuning brings them back. If the trend looks blocky and loses gentle slopes, the compression deviation is too wide. Diagnosing which stage is at fault is the difference between fixing the problem and chasing it.

Decide What Never Gets Compressed

The first configuration decision is not a number, it is a list: the points that must be stored raw. Regulatory measurements, custody-transfer values, alarm and event records, and anything you might need to defend in an audit or incident investigation should have compression turned off or set so tight it never drops a sample. Lossy compression is fine for a temperature you trend for operations and catastrophic for a flow total you invoice on, so separate the two populations before you tune anything.

Event and state changes deserve special care because compression algorithms designed for smooth analog signals handle discrete transitions badly. A digital state that flips, or an alarm that comes and goes, is exactly the kind of sharp edge a slope-preserving compressor can smear or lose, so store those as exceptions with effectively no compression. The rule is simple: if the value's meaning is in its exact transitions rather than its smooth shape, do not let a smoothing algorithm near it.

Set compression policy by tag class, not one global default. Group the fleet into raw-mandatory, lightly compressed, and freely compressed populations, and apply settings to the group. This mirrors how you should already be thinking about a historian collector: it is a policy engine as much as a data mover, and the policy is which class each point belongs to. A single default across all classes is how audit-critical points end up quietly compressed.

Set the Deadbands From Measured Noise

For the points you do compress, size both the exception deadband and the compression deviation from the signal's real noise, not from a round percentage. Trend the raw value at steady state, read the noise band, and set the exception deadband just above it so noise never propagates and any real move does. Then set the compression deviation - the swinging-door tolerance - a little tighter than the smallest process feature you care to preserve, because that is what governs how faithfully slopes survive.

Keep the compression deviation and the exception deadband in a sensible relationship: the compression deviation should generally be no smaller than the exception deadband, because there is no point trying to preserve detail finer than what the collector is willing to send. When the two are mismatched you either waste storage or lose data at the wrong stage. Sizing a single point's compression deviation from noise and trend fidelity is the focused procedure covered in tuning a compression deviation limit; this page is the system-level policy that decides which points get that treatment at all.

Remember that compression trades storage against fidelity, and the right point on that trade depends on why you keep the data. Data you will run analytics or anomaly detection over needs more fidelity than data you keep only for occasional operator glances, because a smoothed trend can hide exactly the small features an algorithm is looking for. Where you feed anomaly detection, lean toward lighter compression so the model sees the texture of the signal, not a piecewise-linear cartoon of it.

Verifying You Are Not Deleting Evidence

Validate compression by reconstruction, which is the only honest test. Log a representative point raw for a period alongside its compressed archive, then overlay the reconstructed compressed trend on the raw one and measure the reconstruction error - how far the stored line ever strays from the real one. If the error stays inside the noise band you accepted, the compression is doing its job; if it clips real peaks or flattens real slopes, it is too aggressive and you dial it back before it runs on the whole fleet.

Spot-check the hardest cases, not the easy ones. Pick the fastest transient, the sharpest state change, and the point nearest a regulatory boundary, and confirm each survives compression intact. Compression that looks fine on a slow, smooth tank level can still be quietly destroying the detail on a jumpy flow or a fast pressure event. The commissioning question is not whether the average point looks good, it is whether the point you would least want to lose is safe, and that check belongs in your pre-go-live configuration validation.

Common Mistakes to Avoid

The most dangerous mistake is a single aggressive compression default applied across every tag, which inevitably compresses the audit-critical points along with the disposable ones. Separate the populations first and turn compression off for anything you might have to defend, then tune the rest. The second mistake is treating the compression deviation as a plain amplitude deadband; on a swinging-door compressor it governs slope fidelity, and misreading it makes trends blocky.

The third mistake is letting a smoothing compressor near discrete states and events, where it smears the exact transitions that carry the meaning. Store those as exceptions with no meaningful compression. The fourth is trusting compression settings you never validated by reconstruction - the storage savings are visible immediately, but the fidelity you lost is invisible until the day you go looking for a detail that is no longer there.

Frequently Asked Questions