My resilience test passed perfectly. It had not tested anything.

I was testing whether a remote site could survive losing its network. The site buffers readings locally and forwards them when the link returns, so the test is simple: cut the link, wait, restore it, then count what arrived.

The result was flawless. Every reading present, original timestamps, no gap anywhere in the window. I was one paragraph into writing it up when something small bothered me.

The site had never noticed anything was wrong.

The thing that did not add up#

The buffering process logs a line whenever it loses its connection, and it checks often enough that a two-minute outage should produce several. There were none.

There was a second signal too. The local queue is only supposed to grow when the link is down and messages cannot be delivered. It had stayed empty the whole time.

Both facts have nothing to do with the data and everything to do with whether the test happened. The data said the system had recovered perfectly from an outage. These two said there had been no outage.

The second reading was right. The rule I had used to cut the link did not apply to the process I was cutting it from — the traffic took a different path through the network stack than the one my rule was filtering, so it flowed the entire time. I had measured a healthy system behaving healthily and called it resilience.

Why this is the dangerous shape of wrong#

Most mistakes announce themselves. A wrong password fails, a wrong address fails, a broken query returns an error. You are stopped, you investigate, you fix it.

This one produced the exact result I was hoping for. Every incentive was pointing at writing it down and moving on: the number was good, the story was clean, and the work was finished. There was no error anywhere to find, because nothing had gone wrong — the system genuinely had not lost data during a period in which nothing happened to it.

That is the property worth naming. A test that reports no failures and a test that never ran are indistinguishable in their output. If your evidence is only the output, you cannot tell them apart, and the one that never ran is the one that feels best to read.

What I do now#

Two extra checks, both taking seconds, both looking at something other than the result.

Did the system notice? Whatever you switched off, something should have complained. A client logs a disconnect, a process logs a refusal, a health check flips. If you break a thing and nothing anywhere reacts, you did not break it.

Did the state change? Fault injection should leave a mark somewhere other than the final measurement. A queue grows. A backlog appears. A counter increments. That mark is independent evidence, and it is what caught this.

When I re-ran the cut with the correct rule, the process logged a disconnect within fifteen seconds and the queue climbed to two hundred and sixty messages before draining to nothing on reconnect. The data looked exactly as it had before, which is the point: only the two side-channels could tell the runs apart.

The general version#

This is not really about networks. It generalises to anything where you verify resilience by removing something.

If you test a failover by stopping a service, check the service actually stopped rather than restarting under a supervisor you forgot about. If you test a backup restore, check you restored into an empty target instead of reading the rows that were already there. If you test that an alarm fires, make the condition true and watch it fire, rather than confirming that it is configured.

In each case the failure mode is the same: the system was never in the state you believe you tested, and the evidence you collected cannot tell you that.

So the habit is one question, asked before you believe your own good news. Not "did it survive?" but "did anything happen to it?" — and the answer has to come from somewhere other than the measurement you set out to take.

The test that eventually did work, along with the queue depths that proved it, is in the lab notes. The estate it was measured against and the six other failures it survived are in the companion lab.