Nothing was wrong with the configuration

If the thing that records your plant stopped writing tonight, how would you find out?

Not in principle. Actually. Which screen would change, which message would arrive, and how long would it take.

For most estates the honest answer is that somebody opens a trend some weeks later, finds it flat, and asks a question. That is not a monitoring gap anybody chose. It is a side effect of what monitoring watches.

Everything you watch answers the wrong question#

Look at what a plant actually has. The gateway status page says RUNNING. The container reports healthy. The connection says connected. The dashboard tile is green, and the service has an uptime measured in weeks.

Every one of those answers is it up. Not one of them answers when did a row last land.

Those are different questions, and only the second one is about the plant. The first is about software.

The gap between them is where the expensive failures live, and none of the causes are exotic. A licensing state changes and the historian quietly drops into a mode that stops recording. A subscription drops and nothing queues behind it, so the outage becomes a permanent hole rather than a delay. A sink loses the topic it was reading. A redundant pair fails over and moves the screens but not the recording.

In every one of those the process stays up. It has to. The process is fine. It is the data that stopped, and nothing was looking at the data.

Seventeen days#

Here is the case that made me write this down.

A multi-site system I run reported every component up and healthy for seventeen days while its historian took no rows at all. Every container healthy. Every gateway answering. Traffic on the wire the whole time.

The last row landed at 08:39:42. The log line explaining why was written at 08:39:43 — one second later, a licensing state change that stopped the recording path while leaving everything else running exactly as before.

So the information existed. It was in a log, one second after the last good row, and it sat there unread for two and a half weeks because nothing in the estate was watching the one number that would have shown it.

Green is a property of the process. It says nothing about the data.

Then I looked in the wrong place for an hour#

This is the part I would rather not write, which is usually the sign it is worth writing.

When I came to diagnose it, I did what everybody does: I opened the configuration. And I reached three confident wrong conclusions in a row, told somebody each of them, and had to retract all three.

The first was a query of my own that dropped a column. Asking for the distinct tag paths returned six, so I announced the system was recording six signals and was essentially a toy. The table had a site column I had left out. The real answer was eighteen, which is exactly what it was designed to carry.

The second was a search for a connector by the technology it used. It matched nothing, so I reported the connector missing. It was there, stored under the name somebody had given the connection years earlier, as connectors generally are.

The third was reading the wrong tree. Registered data sources and the content of those sources live in different places, and a source that fetches its content from elsewhere correctly has none stored locally. I found the empty one and announced the sources were gone.

Every one of those readings was careful. Every one was wrong. And one query against the data — when did the newest row arrive — would have settled the whole thing before any of them.

What configuration can and cannot tell you#

The reason is not that I was careless with the files. It is that configuration answers a different question, and it is easy to forget which.

Configuration tells you what a system was asked to do. Only the data tells you what it did. Those diverge constantly and quietly — a licence state, a dropped subscription, a permission, a full disk, a certificate. In every one of those cases the configuration is still correct. It is still exactly what you wrote. Reading it again more carefully will never reveal the problem, because the problem is not in there.

Nothing was wrong with the configuration. Everything was wrong with what it was producing.

The change worth making#

There is one, and it is small.

Alert on the age of the newest row, not on whether the service is up. One query, one threshold, one alert. If the newest reading is older than it should be, something is wrong, and you do not need to know which of the causes above it was — they all produce the same symptom, and the symptom is the thing worth watching.

It is also the only check on the list that asks a question about the plant rather than about the software. Uptime, heartbeat, health endpoint and connection status are all statements about processes. The age of the newest row is a statement about whether you still know what your plant is doing.

Being fair about it: this is not free. You need somewhere to run the query, a sensible threshold per signal — a value that genuinely updates weekly should not alarm after an hour — and the discipline not to let it become noise. That is an afternoon, once.

Go and run it#

Take the newest timestamp from your historian right now, for a handful of signals you would actually care about. Most of the time it will be seconds old and you will feel slightly silly.

Then ask the harder question, which is not about the number you got but about the one you did not: if that had come back three weeks old, what would have told you?

If the answer is "somebody opening a trend", then that is your genuine detection time for a silent loss of history, and it is measured in weeks. Everything else on your dashboard would have stayed green throughout, exactly as designed.