How to Set Historian Retention and Rollup Tiers

Back to Blog

Map Retention Tiers to How the Data Is Used

Start by listing why anyone reaches into history, because each reason wants a different resolution and age. Operators and engineers investigating a recent event need full-resolution raw data, but only for a recent window - days to weeks. Long-term trend and performance analysis needs coverage over months or years but tolerates aggregated resolution. Regulatory reporting needs specific data kept for a specific period regardless of anything else. Those three uses map to three tiers, and naming them is the whole design.

Set the raw tier from the investigation window. Keep every sample at full resolution for as long as people realistically go back to look at sample-level detail, then let older raw data roll up. The raw window is usually shorter than people first guess, because sample-level scrutiny of six-month-old data is rare, and paying to store every raw sample for years to serve queries nobody makes is wasted storage. Size the raw tier from actual query behavior, not from a fear of throwing something away.

Feed the tier design back into how the collector is configured. Retention and compression are two halves of the same storage story - compression decides how much detail each sample carries, retention decides how long you keep it. Design them together so you are not paying to retain heavily compressed data past its usefulness, and reference your collector compression settings when you size each tier.

Roll Up Old Data Into Aggregates

Rollup replaces many raw samples with periodic summaries, and choosing the summary is the key decision. A time-based rollup stores, for each interval, the aggregates that matter for later use - typically minimum, maximum, average, and often the count or total. Those four preserve the shape and the extremes of a signal at a fraction of the storage, so a year of one-minute averages plus min and max still shows the peaks and the trend even though the raw samples are gone.

Choose aggregates that survive the questions you will ask later. If you will ever need to know the worst excursion in a period, you must roll up the maximum, because an average alone hides it. If you invoice on totals, you must preserve the total. Decide the questions the deep-past tier has to answer before you pick the aggregates, because once the raw data is gone, an aggregate you did not compute is a question you can no longer answer.

Keep the aggregate tier honest about what it is. An averaged, rolled-up value is not the same as a raw sample, and anyone querying the deep past needs to know they are looking at a summary, not the original signal. This matters most where a downstream tool or an anomaly detection model expects sample-level texture, because feeding it aggregates silently changes what it can see. Label the tiers so a query knows which resolution it is getting.

Protect Regulatory and Audit Data Separately

Carve the regulated data out of the ordinary retention rules entirely. Anything under a regulatory, contractual, or custody-transfer requirement has a retention period set by that obligation, not by storage convenience, and it must be exempt from both early purge and lossy rollup. Treat these points as a distinct population with their own policy so a routine storage cleanup can never quietly delete data you were required to keep, which is precisely the kind of point you also store raw in your compression policy.

Know exactly which points carry an obligation and for how long. This is not something to guess at during a storage crunch; it is a list to build deliberately with whoever owns the compliance requirement, so the retention policy encodes the real obligation rather than an engineer's assumption. Getting a retention period wrong on regulated data is a compliance failure that no amount of storage saved can justify.

Make sure the retention rules interact correctly with data recovery. If a store-and-forward buffer or a CSV backfill delivers old data after the fact, the retention engine must file it into the correct tier by its real timestamp, not treat it as new. Data that arrives late but belongs to a regulated period still has to land in the protected tier, and a naive retention rule that keys off arrival time instead of event time will misfile it.

Verifying the Policy Behaves

Test that each tier actually holds what you expect by querying at three ages: recent, mid-range, and deep. The recent query should return full-resolution raw samples, the mid-range should return whatever you defined, and the deep query should return aggregates. If a deep query still returns raw samples, rollup is not running and your storage is quietly filling; if a recent query only returns aggregates, you are rolling up too aggressively and losing investigation detail.

Prove the regulated data is safe by confirming it survives a rollup and purge cycle untouched. Check that a point under a retention obligation still has its required history at full fidelity after the normal cleanup has run, and that its retention is keyed to event time so late-arriving data files correctly. This is the check that matters most, because the failure it catches - regulated data quietly rolled up or purged - is the one with real consequences, and it belongs in your retention review.

Common Mistakes to Avoid

The most consequential mistake is applying one retention rule to regulated and unregulated data alike, so a routine cleanup deletes something you were required to keep. Carve out the regulated population with its own protected policy. The second mistake is sizing the raw tier from fear rather than from actual query behavior, paying to store years of raw samples to serve queries nobody makes.

The third mistake is rolling up the wrong aggregates, so the deep-past tier cannot answer a question you will eventually ask - typically the worst excursion, which an average alone hides. Decide the questions first, then pick the aggregates. The fourth is keying retention off arrival time instead of event time, so late-arriving data from a store-and-forward recovery or backfill lands in the wrong tier and can be purged before its real age would allow.

Frequently Asked Questions