Automation Glossary • Resolution vs Accuracy

What Is The Difference Between Sensor Resolution And Accuracy?

Merobix Engineering • • 6 min read

A display reading twelve point three four five looks authoritative. Five digits of apparent precision suggest a measurement pinned down to a thousandth. But those digits describe only how finely the number is written, not how close it is to the truth, and the two can be worlds apart. Confusing fine resolution with real accuracy is one of the most common and costly misreadings in instrumentation, and untangling them protects you from trusting decimal places that the sensor never earned.

Back to Blog

Resolution vs Accuracy in one line: Resolution is the smallest change a sensor or display can detect and show, essentially the fineness of its increments. Accuracy is how close the reported value is to the true value. A device can have very fine resolution, showing many decimal places, while being badly inaccurate, because how precisely a wrong number is written has nothing to do with how wrong it is.

Fine Increments Versus Closeness To Truth

Resolution answers a question about discrimination: what is the smallest step the instrument can distinguish and report? A display with more digits, or a converter with more bits, has finer resolution, meaning it slices the range into smaller increments. Resolution is a property of the reporting mechanism, and improving it lets you see smaller changes, but only if those changes are real and the underlying reading is sound.

Accuracy answers a different question about truth: how far is the reported value from the actual value? Accuracy is degraded by bias, drift, nonlinearity, and environmental effects, none of which are cured by adding digits. A pressure reading that is consistently three units off is three units off whether it is displayed as a whole number or to four decimal places. The extra decimals do not make it closer to the truth; they simply describe the error more precisely.

The classic illustration is a display showing twelve point three four five when the true value is eleven. The resolution is excellent, thousandths, but the accuracy is terrible, off by more than a full unit. All the resolution has done is describe an inaccurate value in fine detail. This is why the two properties must be quoted and judged separately, and why a specification that boasts of resolution while staying silent about accuracy should raise suspicion rather than confidence.

The False Precision Trap

False precision happens when the number of digits shown exceeds the instrument's real uncertainty. If a measurement is only trustworthy to within a couple of units, then displaying it to three decimal places invents information that is not there. Those trailing digits jitter with noise and drift, and reading them as meaningful leads to decisions and adjustments chasing figures that are pure fiction. The honest number of significant figures is set by the uncertainty, not by how many the display can render.

This trap is easy to fall into because digital displays and data systems will happily render any number of digits regardless of whether the sensor supports them. A converter can produce sixteen bits of resolution from a sensor whose true accuracy is far coarser, and the extra bits will faithfully digitize noise, drift, and error along with the signal. The resolution is real in the sense that the steps are small; it is false in the sense that most of those small steps carry no trustworthy information about the true value.

The practical rule is to let the uncertainty govern how many digits you trust and act on. Round reported values, alarm thresholds, and control setpoints to a precision the accuracy actually supports, and treat any digit finer than the uncertainty band as decoration. This is not about hiding data; the raw fine-resolution signal is still useful for detecting small relative changes and trends. It is about not mistaking the fineness of the number for the certainty of it.

Trusting SCADA Tag Decimals Correctly

In a SCADA system, every point carries a display format, and it is easy to configure a tag to show more decimals than the underlying sensor's accuracy justifies. Operators then read those decimals as if they mean something, comparing readings to a precision the instrument never delivered and raising phantom concerns over changes that are entirely inside the error band. Setting tag decimals in line with the point's real uncertainty keeps the display honest and prevents the historian from documenting false precision as if it were fact.

There is a subtlety worth preserving, though. Fine resolution, even beyond the accuracy limit, is genuinely useful for detecting small relative changes over time. A sensor with a modest absolute accuracy can still reveal a slow trend or a small step change, because those show up as changes in the reading even if the absolute value is biased. So the historian benefits from logging the fine-resolution signal, while reports and alarms should present values rounded to the trustworthy precision.

For an operator running many points through Merobix, the discipline is to separate two uses of the same tag: the raw high-resolution trend for spotting relative movement and drift, and the reported, rounded value for anything that touches a contract, an audit, or a control decision. Documenting the accuracy of each point alongside its display format lets everyone see, at a glance, how many of a tag's decimals are truth and how many are just the display filling space. That distinction is what keeps confident-looking numbers from being trusted beyond what the sensor earned.

Frequently Asked Questions

Can a sensor have high resolution but low accuracy?

Yes, and it is common. Resolution is only the fineness of the increments the sensor can display, while accuracy is closeness to the true value. A device can slice its range into tiny steps and show many decimals while being biased or drifted far from the truth. The extra digits describe an inaccurate value precisely rather than making it correct.

What is false precision?

False precision is showing more digits than the measurement's uncertainty supports, which invents information that is not really there. If a reading is trustworthy only to within a couple of units, displaying it to three decimals implies a certainty the instrument never provided. Those trailing digits jitter with noise and should be treated as decoration, not data.

How many decimal places should a SCADA tag show?

As many as the point's real uncertainty supports and no more. Any digit finer than the accuracy band carries no trustworthy information for absolute readings, so alarms, reports, and control values should be rounded accordingly. Fine resolution is still worth logging for spotting small relative changes and trends, but it should not be presented as precision the sensor cannot back up.

From Definitions to a Live Dashboard

Merobix reads your field devices into a cloud SCADA - the real thing behind these terms, live in days from any browser.

Request a Free Demo +1 (903) 307-7300
More in Automation Glossary
Quantization Error  •  Hysteresis vs Dead Band  •  Zero Shift vs Span Shift  •  Drift Over Time  •  Temperature Coefficient  •  Reference Conditions  •  All Automation Glossary →
Free SCADA operator training
Merobix University - 70 video lessons & 261 quiz questions, from first login to compliance reporting. No demo call required.
Start free →