3 Aug 2026 · 4 min read
One hundred remote sites, one small broker, and the wrong answer I nearly published
redpanda · kafka · scada · architecture · edge · mqtt · ot
You have twenty sites now and somebody is talking about a hundred. The architecture drawing has one box in the middle that everything reports into, and the two questions in the room are always the same. Will one cluster carry it, and what happens to everybody else when one site loses its mind.
I built the hundred sites and measured both.
THE ESTATE
A hundred simulated sites, fifty tags each, publishing every second into a single Redpanda broker given two cores and two gigabytes of memory — deliberately modest, because the interesting question is not whether a large cluster works.
The capacity answer is boring, which is the useful kind of boring. Just under five thousand messages a second, no errors at all, and every reading arriving at the centre within about a tenth of a second. Nothing was close to a limit. At this size the broker is not the thing to worry about.
The question that actually matters
Shared infrastructure makes people nervous for a good reason. If one site's gateway starts spraying data — a misconfigured scan class, a stuck loop, a commissioning engineer testing something at four in the morning — does everybody else suffer?
So I made one site publish two hundred times its normal volume and watched the other ninety-nine.
END-TO-END LAG OF THE INNOCENT SITES, p95
Solid bar is the ninety-nine well-behaved sites together; the faded bar beside it is one named site, houston-01. Milliseconds.
The middle group is what I measured first, and it looks damning. Every innocent site's data went from arriving in about 98 milliseconds to about 330. No losses, but a three and a half times penalty for being on the same cluster as one bad neighbour. That is the kind of number that ends up in a slide arguing for separate clusters per region.
It is also wrong.
What I had actually measured
My simulator published all hundred sites through one client connection. So when one site flooded, the other ninety-nine sites' readings were sitting in a queue inside my own program, waiting their turn, before they ever reached the broker. I had measured my gateway and labelled it the cluster.
WHY THE FIRST ANSWER WAS WRONG
innocent sites queue behind the flood, before the broker is involved
the flood fills only its own path
Running it again with the flooding site on its own connection gives the opposite result. The ninety-nine well-behaved sites measured 98 milliseconds — identical to the baseline, to the millisecond. Only the offending site slowed down, to about 230 milliseconds, which is exactly the behaviour you want: the site causing the problem is the site that feels it.
Meanwhile the broker carried both loads at once, over fourteen thousand messages a second, still without dropping anything.
WHAT ONE BROKER DID
The lesson is not really about Redpanda. It is that isolation follows the connection, not the cluster. A hundred sites with their own connections are isolated from one another. A hundred sites funnelled through one shared gateway process are not, and no amount of broker tuning will help, because the queue forms before the broker is ever involved.
Which is worth checking in your own drawing. If there is a single box that all your sites publish through — one Node-RED, one edge gateway, one integration server — then your sites are sharing a fate whatever the cluster underneath looks like.
One topic per site, or one topic split by site
The other decision that comes up early, and the one where the OT instinct and the Kafka instinct disagree.
The OT instinct is a topic per site, because then retention, access and deletion are all per-site and you can point at a thing and say that is Rotterdam. The Kafka instinct is one topic with a partition per site, keyed on the site name.
I ran both.
ONE TOPIC PARTITIONED, OR ONE TOPIC PER SITE
Throughput was identical. The tail latency was not: the ninety-ninth percentile went from 119 milliseconds to 337, because keeping track of a hundred topics is work the broker does alongside carrying your data.
But the thing that actually bit me was not performance.
THE FAILURE THAT COST ME AN AFTERNOON
My central consumer was subscribed by pattern, meant to pick up every per-site topic. It received nothing. Zero messages, across the whole run, because the per-site topics were created after it subscribed and it never went back to look.
Think about what that means on a real rollout. You commission a new site on a Tuesday. It publishes happily, the data is genuinely in the cluster, and your central historian simply does not know it exists — until somebody restarts the consumer. There is no error, no alarm, and no gap in any existing trend. There is just a site that never shows up.
With one partitioned topic, a new site starts appearing the moment it publishes.
Per-site topics still earn their place if a contract or a regulator requires per-site retention or access control. That is a real requirement and worth the cost. Just pay it knowingly, and put "does the central consumer know about new sites" on the commissioning checklist.
What I would take away
At a hundred sites, capacity is not your problem. One modest broker handled the estate with enough room left to absorb a site behaving two hundred times worse than it should.
Give every site its own connection to the cluster. It is the difference between one bad site being its own problem and one bad site being everybody's.
And be suspicious of a measurement that makes shared infrastructure look bad before you have checked what else is shared. I nearly published the opposite of what is true, and the only reason I did not is that the number seemed too neat.
The full rig, both runs, the topic comparison and the two harness bugs are in the lab write-up.
Keep reading
Newsletter
New essays, by email.
SCADA, cloud, AI, and the plant floor — a short email when something new is published. No noise, unsubscribe anytime.