There is more on the plant network than anyone thinks

Since the end of July, water utilities in at least seven American states have reported break-ins to the FBI. More than thirty in Minnesota alone were hit in a single coordinated weekend. Operators went back to running things by hand.

The way in was equipment that could be reached from the internet.

Most of the coverage has been about how weak the protocols are, and I have already written that piece — the protocol is not the problem. What stayed with me this time was a smaller detail that turns up in nearly every account: the operators did not know that equipment could be reached.

That is not a technology failure. It is not knowing what you own. And it is far more embarrassing, because everybody believes they know what they own.

So I counted my own network three different ways, to see whether I did.

The first count: ask the control system#

The obvious place to start is the control system's own settings. It is the most official record there is. What does it think it is connected to?

One database. That is the entire list of outside things it knows about.

It also holds two other machines it trusts completely — approved, ready to exchange data. Both of those machines were scrapped some time ago. The settings still list them. Still approved. Still ready to welcome anything that turns up answering to their name.

So the most authoritative record in the building is wrong in both directions at the same time. It is missing nearly everything that is really out there, and it lists two things that no longer exist.

The second count: knock and see who answers#

Next I asked the network directly. I checked every machine for the connections you would expect on an industrial network — the control ports, the messaging ports, the database ports, the usual remote-access ones. Twenty-two of them.

Eight machines answered, offering sixteen ways in between them.

That is already five times what the control system knew about. One of them was a second messaging system — a whole parallel setup with its own database, its own graphs and its own data feed, which appears in the control system's records nowhere at all. Nothing sinister. Somebody built it, it works, and the control system has no idea it is there.

Then I did the thing that actually mattered.

I looked for what I expected to find#

Twenty-two connections is a sensible list. It is the list you would write if somebody asked you to check an industrial network. It had everything I could think of on it.

So I threw the list away and checked all sixty-five thousand.

Sixteen ways in became forty-two, across only five machines. One machine I had written down as having two had sixteen. Another had fifteen.

Nothing changed on the network between those two checks. The only thing that changed was how hard I looked.

This is the part I would put in front of anyone who believes they have a list of their equipment. Checking your network does not tell you what is there. It tells you what you thought to ask about. The utilities that were broken into were almost certainly not careless. They had almost certainly checked. Checking for what you expect finds what you expect.

The third count: what is actually allowed through#

The last question is the simplest one. Of everything that is listening, how much does the firewall let through?

I looked. The firewall was switched off. The answer is everything.

In fairness, this is a test network in a spare room, not a water utility facing the internet, and switched off is a defensible choice there. The uncomfortable part is that I did not know until I looked — on a network I built myself, where I chose every single component, on the same afternoon I was writing about other people not knowing what they had.

That is the honest version of this story. The problem is not that anybody is bad at their job. It is that these three questions get asked by different people at different times, and nobody ever lays the three answers side by side.

What is worth doing#

Put the three lists next to each other. Not one of them. What the control system thinks it is connected to, what actually answers, and what the firewall allows. The disagreements between them are the whole point, and no single list can show you one.

Once in a while, look without a list. Your expectations are the weakest part of any check. A full sweep is slow and dull and you only need it occasionally. It is also the only version capable of surprising you, and being surprised is the entire purpose.

Treat anything unexpected as a finding. A second messaging system nobody wrote down is not a curiosity. It is another way in, with its own passwords and its own records, sitting outside every process you have.

Delete trust in things that no longer exist. Two scrapped machines still marked as trusted are two names anything on the network could claim. Since a backup carries that trust with it, it spreads rather than fades.

The oldest question in security is not whether you are up to date. It is what you have got. Three counts of the same small network produced three different networks, and the one the control system believed in was the least accurate of the three.

The method, the exact results and the machine-by-machine numbers are in counting the plant three ways.