Concluded·3 Aug 2026 · 5 min read
A hundred remote sites into one broker, and the measurement that reversed itself
Redpanda v24.2.18 (single node, 2 cores, 2 GB) · kafkajs · Docker Compose · Proxmox LXC
Two questions come up whenever a fleet grows past a handful of sites. Will one cluster carry it, and what happens to the other sites when one goes wrong.
I built a hundred sites to find out, and got the second answer wrong before I got it right.
The rig
| Broker | Redpanda v24.2.18, single node, 2 cores, 2 GB |
| Estate | 100 sites × 50 tags, published every second |
| Sites | international place names — rotterdam-01, houston-01, seville-02 … |
| Strategies | one topic with 100 partitions, and one topic per site |
Two different things are measured, and conflating them is how the wrong answer happens:
- publish latency — how long the producing client waited for an ack
- end-to-end lag — from a reading being produced at a site to a central consumer actually seeing it
Only the second can tell you anything about blast radius, which is why the rig has a separate consumer recording per-site percentiles.
Experiment 1 — capacity
| Messages | 295,000 in 61.5 s |
| Rate | 4,797 msg/sec |
| Errors | 0 |
| Publish latency | p50 58 ms · p95 65 ms · p99 119 ms |
| End-to-end lag | p50 82 ms · p95 98 ms · p99 164 ms |
| Sites seen | 100 / 100 |
THE ESTATE
Dull, in the good way. Two cores carried the estate and every reading reached the centre in about a tenth of a second.
Experiment 2 — one site misbehaves
rotterdam-01 was set to publish at 200× normal rate. This ran twice and the
two runs disagree, which is the most useful thing in this write-up.
First run: every site sharing one producer client
| Baseline | Flooding | |
|---|---|---|
| Total rate | 4,797/s | 14,342/s |
| Errors | 0 | 0 |
| Publish p50 | 58 ms | 257 ms |
| End-to-end, the 99 innocent sites p95 | 98 ms | 330 ms |
Read on its own that says a shared cluster does not isolate its tenants, and one bad site costs everyone 3.4×.
It is wrong, and the fault was mine. The simulator published all hundred sites through a single Kafka client, so the other ninety-nine sites' messages were queued behind the flood inside my own producer, before the broker was involved at all. I had measured my gateway and written the broker's name on it.
Second run: the flooding site on its own client
| Baseline | Flooding, separate clients | |
|---|---|---|
| Total rate | 4,797/s | 14,447/s (4,797 + 9,650) |
| Errors | 0 | 0 |
| End-to-end, the 99 innocent sites p95 | 98 ms | 98 ms |
End-to-end, houston-01 p95 | 97 ms | 94 ms |
| End-to-end, the flooding site p95 | — | 230 ms |
END-TO-END LAG OF THE INNOCENT SITES, p95
Solid bar is the ninety-nine well-behaved sites together; the faded bar beside it is one named site, houston-01. Milliseconds.
Identical. The ninety-nine well-behaved sites did not notice a neighbour publishing two hundred times its normal volume. Only the offending site slowed down, which is precisely the behaviour you would design for.
WHY THE FIRST ANSWER WAS WRONG
innocent sites queue behind the flood, before the broker is involved
the flood fills only its own path
Isolation follows the connection, not the cluster. This is the finding worth carrying into a real design. A hundred sites each holding their own connection are isolated from one another by the broker. A hundred sites funnelled through one shared gateway process are not, and no broker setting can fix it, because the queueing happens upstream of the broker.
Worth checking your own architecture drawing for a single box that every site publishes through — one Node-RED, one integration server, one edge concentrator. If it is there, your sites share a fate regardless of what the cluster is doing.
WHAT ONE BROKER DID
Experiment 3 — one topic per site, or one partitioned topic
| One partitioned topic | One topic per site | |
|---|---|---|
| Rate | 4,797/s | 4,797/s |
| Publish p50 | 58 ms | 55 ms |
| Publish p99 | 119 ms | 337 ms |
| Topics on the cluster | 1 | 100 |
| A new site appears to a central consumer | immediately | not at all |
ONE TOPIC PARTITIONED, OR ONE TOPIC PER SITE
Throughput identical; tail latency 2.8× worse with a topic per site, because tracking a hundred topics is metadata work carried alongside the data.
But the latency is not the problem.
THE FAILURE THAT COST ME AN AFTERNOON
The central consumer was subscribed by pattern — ^plant.site..* — and received
zero messages across the entire run. The per-site topics were created after
it subscribed, and it never went back to look.
On a real rollout that is the nastiest kind of failure. A site is commissioned, publishes correctly, its data genuinely lands in the cluster, and the central historian does not know it exists. No error, no alarm, no gap in any existing trend. Just a site that never appears, until somebody restarts the consumer.
Per-site topics still earn their place where per-site retention, access control or deletion is a contractual requirement. Pay the cost knowingly, and put "does the centre know about the new site" on the commissioning checklist.
Harness bugs, recorded because both lied
1. The shared producer. Turned "the broker isolates sites perfectly" into "one site degrades everyone 3.4×". The measurement was real and the instrument was the thing being measured. The only reason I checked was that 3.4× looked too tidy for a system that had otherwise refused to break.
2. push(...array) blows the call stack. The consumer merged per-site
latency arrays with spread. Under a flood, one site's array reached about 1.8
million elements, and spreading that as function arguments crashed the process
with Maximum call stack size exceeded. It failed only on flood runs, so it
looked like a load limit rather than a bug in my own code. Fixed with a loop and
.slice().
Verdict
At a hundred sites, capacity is not the question. One modest broker carried the estate with enough headroom to absorb a site behaving two hundred times worse than it should, and never lost a message in any run.
Give every site its own connection. Partition one topic by site rather than creating a topic per site, unless governance requires otherwise.
And distrust a result that makes shared infrastructure look bad until you have checked what else in the path is shared.
The argument, without the measurements, is in the article.
Newsletter
New essays, by email.
SCADA, cloud, AI, and the plant floor — a short email when something new is published. No noise, unsubscribe anytime.