Your historian is throwing readings away on purpose

Somewhere in the settings of the system that records your plant's data is a small number. Probably 1.0. Possibly 0.5.

It decides which readings get written down and which get silently dropped for not being different enough from the last one.

Somebody typed that number in during installation. In most plants nobody has looked at it since. It is the one place where data is deliberately destroyed as a matter of policy, and it is usually the least examined setting in the building.

So I measured what it actually costs.

What the setting does#

A reading is kept only if it differs from the last reading that was kept by more than the threshold. Not from the previous reading — from the last one written down. What you get back afterwards is a series of steps.

That is a real signal from a running system. A hundred and forty readings arrive. Nineteen get stored. If somebody looks at that period next year, the steps are what they see. There is no note on the chart saying the real line was smooth.

For most purposes the steps are perfectly fine. That is the whole justification for doing this, and it is a good one. The question is whether anybody ever checked.

What it costs#

I ran eighteen signals through five different settings, and measured two things: how many readings were thrown away, and how wrong the stored version ever got.

At the common setting, it discards 41% of everything and the stored data is never wrong by more than about 0.7%. At a more aggressive setting you keep only 15% of your data and are never wrong by more than 5%.

Put like that it is an excellent bargain. If somebody had deliberately chosen it, there would be nothing to write about.

The number that should bother you#

The average hides the finding completely. Here is what that single setting did to each of the eighteen signals separately.

0% saved66% saved

6 of 18

saved nothing at all

41.1%

the average everyone quotes

0 – 66.6%

the range it is hiding

Six of the eighteen threw away nothing at all. Not a little — zero. Meanwhile others threw away two thirds.

Same plant, same setting, same day. The difference is entirely in how each signal behaves: how far it moves between readings compared with its full range. A slow, gentle signal gets squeezed hard. A signal that moves briskly never triggers the threshold, so everything is kept and the setting does precisely nothing.

That is the real state of most of these systems. One number, applied everywhere, doing three unrelated jobs at once:

• On some signals it does nothing — you are paying for full storage while believing you have a saving.
• On some it is about right.
• On some it is too harsh, quietly removing detail somebody will come looking for one day.

The averages look sensible because those three cancel each other out. Nobody is reading the averages.

The part you cannot undo#

Storage is recoverable. Disks are cheap, and you can always change the setting from tomorrow onwards.

What you cannot do is go back and un-throw-away a reading.

This is the one setting in the plant whose mistakes are permanent. If it was too harsh on a pressure reading for three years, those three years are steps now, and no amount of care afterwards brings the detail back. When somebody eventually needs to explain a four-second event, the honest answer may be that the plant decided in advance, without discussing it, not to keep it.

What is worth doing#

Find out what yours is set to. Most people reading this do not know. It takes two minutes and puts you ahead of nearly everybody.

Check whether it is doing anything at all. Compare how many readings you have stored against how many should have arrived, for a handful of signals. If they match, the setting is doing nothing on that signal and your saving exists only on paper.

Set it per signal, not for the whole plant. The right threshold depends on how each signal behaves. One number for everything is a single decision pretending to be hundreds.

Decide it against what the data is for, not against disk space. The question is not how much room you save. It is: what is the smallest change anybody will ever need to see? Answer that and the number follows. Answer the storage question instead, and you find out what you destroyed years later, from somebody who needed it.

The full measurement — all eighteen signals, five settings, and a mistake in my own method that reversed the first result — is in what a deadband actually costs.

There is a second setting that looks like the same kind of knob and is not: how often the value is read in the first place. Traded against this one, it turns out to be far more expensive — two settings decide what your plant remembers.