Back online is not the same as recording

Ransomware cases climbed 22% in a single month this summer — 894 of them in July. How fast you can rebuild a plant stopped being a theoretical question a while ago, and most places answer it with a number they have never actually measured.

The number they quote, if they have one, is how long it takes to get the system running again. It is the number the backup software prints and the number that goes in the disaster-recovery folder.

It is the wrong number. I killed a control system to find out by how much.

The drill#

Not a polite shutdown — killed outright, the way failures actually happen. Then a replacement built from the backup file, and a clock left running until the plant was genuinely recording again.

drill 1running 24.34s · first row 33.02s · +8.7s unrecorded

drill 2running 23.67s · first row 32.95s · +9.3s unrecorded

system reports it is running — the number in the report up, green, and recording nothing

Twenty-four seconds to a running system. That is the number that goes in the report, and it is genuinely good.

Thirty-three seconds until the first real reading was written down.

The nine seconds in between are the ones nobody counts. The system is up, the screens are drawing, the status light is green — and nothing is being recorded yet, because connecting to the database and getting the first reading through is work that happens after "the system is back".

That is 36% longer than the number in the folder. I ran the whole thing twice to be sure. The two runs agreed to within a second.

The number that actually matters#

Here is what neither figure describes.

12:07:12 — last reading the old system wrote26.8 s missing12:07:39 — first reading its replacement wrote

~84

readings lost

0

recoverable from the backup

never

when the gap closes

Twenty-seven seconds of the plant's history is simply gone. Not delayed — gone. There is a hole in the record between the last reading the old system wrote and the first one its replacement wrote, and nothing in the recovery fills it.

At the rate this plant was recording, that is about eighty-four readings that were real, were measured, and now exist nowhere.

The reason is worth being clear about. These systems hold on to readings they cannot yet write down, and that works well. But the holding area is inside the system itself. Kill the system and the held readings die with it. The replacement is built from a backup file, and a backup file contains no readings at all — it holds the settings and the address of the database, not its contents.

So the faster you recover, the smaller your hole. But no recovery is fast enough to make it zero, and none of any speed recovers what was lost while nothing was running.

Why the good number is the dangerous one#

Twenty-four seconds is impressive, and that is exactly the problem. A number that good ends the conversation.

What it hides is that this was the easiest possible version of the disaster:

• The backup file was already on the machine. Nobody had to find it, fetch it, or get permission to use it.
• The database was untouched — I stopped it, I did not lose it. Everything recorded before the incident was still there.
• The replacement started on a trial licence. A real one needs activating, by a person, during an emergency, possibly at three in the morning.
• Nothing needed a certificate, a network change or a new firewall rule.

Take those away and none of the remaining work is measured in seconds. So the honest reading of "twenty-four seconds" is this: the part a computer does is fast, and it is the only part anybody has ever timed.

What is worth measuring#

Time to the first real reading, not time to restart. Pick one signal, watch the records, and start the clock when the trouble starts rather than when the recovery does. It is harder to produce and it is the only number that means anything.

Measure your hole, and decide whether you can live with it. Every recovery leaves one. Twenty-seven seconds is fine. Discovering during an audit that yours is four hours, because nobody noticed the system had stopped writing, is not.

Keep the backup somewhere realistic when you rehearse. If the file is already on the machine you are recovering, you have not tested a disaster. You have tested a restart.

Time the human steps separately, and honestly. Licences, approvals, finding passwords, waking somebody up. Those dominate the real number and none of them appear in any tool's report.

The plant came back in half a minute. Twenty-seven seconds of what it measured never did — and the recovery report would have shown neither of those facts.

Both runs, the full method and the exact size of the hole are in timing a recovery to the first good row.