One hundred remote sites, one small server, and the wrong answer I nearly published

You have twenty sites today and somebody is talking about a hundred. The drawing on the wall has one box in the middle that everything reports into, and the two questions in the room are always the same.

Will one box carry it all. And what happens to everybody else when one site loses its mind.

I built the hundred sites and measured both.

A hundred simulated sites, fifty tags each, reporting every second into a single server given two processor cores and two gigabytes of memory. Deliberately modest, because the interesting question is not whether an expensive cluster works.

The capacity answer is boring, which is the useful kind of boring. Just under five thousand readings a second, nothing lost at all, and every reading arriving at head office within about a tenth of a second. Nothing came close to a limit. At this size, the box in the middle is not the thing to worry about.

The question that actually matters#

Sharing infrastructure makes people nervous, and for good reason. If one site starts spraying data at everything, because a scan rate got typed in wrong or a commissioning engineer is testing something at four in the morning, does everybody else suffer?

So I made one site report two hundred times faster than it should, and watched the other ninety-nine.

What I measured first looked damning. Every innocent site's data went from arriving in about a tenth of a second to about a third of a second. Nothing was lost, but that is a three and a half times penalty for the crime of sharing a server with one badly behaved neighbour.

That is the kind of number that ends up on a slide arguing for separate servers per region.

It is also wrong.

THE SAME QUESTION, ANSWERED TWICE

FIRST ATTEMPT

3.4× slower

everyone shared one pipe, including my test rig

DONE PROPERLY

no change

the other ninety-nine sites never noticed

What I had actually measured#

My simulator sent all hundred sites' data through one connection. So when one site flooded, the other ninety-nine sites' readings were queued up inside my own program, waiting their turn, before they ever reached the server in the middle.

I had measured my own equipment and put the server's name on the result.

WHY ONE BAD SITE DID, THEN DIDN’T, HURT THE REST

ALL SITES SHARE ONE PIPE

everyone slows down

EACH SITE HAS ITS OWN

only the bad site slows

Running it again with the badly behaved site on its own connection gives the opposite answer. The ninety-nine well-behaved sites were completely unaffected, to the millisecond. Only the offending site slowed down, which is exactly what you want: the site causing the problem is the site that feels it.

Meanwhile the server in the middle carried both loads at once, three times its earlier traffic, and still lost nothing.

The lesson here is not really about any particular product. It is that isolation follows the connection, not the server. A hundred sites with their own connections are protected from each other. A hundred sites funnelled through one shared box on the way out are not, and no amount of tuning at the far end will fix it, because the queue forms before the far end is ever involved.

Which is worth checking on your own drawing. If there is a single box that all your sites report through, one gateway or one integration server, then your sites share a fate no matter what sits behind it.

One folder per site, or one big folder#

The other decision that comes up early, and the one where instinct pulls the wrong way.

Think of the server in the middle as holding folders of readings. You can give every site its own folder, which feels tidy and means you can set different retention and different permissions per site. Or you can have one folder for everything, with each reading labelled by which site it came from.

The tidy-looking option is the one most people ask for. I ran both.

They carried the same traffic at the same speed. But keeping track of a hundred separate folders is work the server has to do alongside its real job, and the slowest responses got nearly three times worse.

That was not what actually bit me, though.

THE FAILURE WITH NO ERROR MESSAGE

Head office was set up to watch every site folder automatically. It received nothing. Not one reading, for the entire run, because the folders were created after it started watching and it never went back to look.

Think about what that means on a real rollout. You commission a new site on a Tuesday. It reports perfectly, its data genuinely arrives at head office, and head office does not know it exists. There is no error and no alarm, and no gap appears in any existing trend. There is just a site that never shows up, until somebody restarts something.

With one folder for everything, a new site starts appearing the moment it reports.

Separate folders per site still earn their place if a contract or a regulator requires you to keep or delete one site's data differently from another's. That is a real requirement. Just pay the cost knowingly, and put "does head office know about the new site" on the commissioning checklist.

What I would take away#

At a hundred sites, capacity is not your problem. One modest server handled the lot with enough room left over to absorb a site behaving two hundred times worse than it should.

Give every site its own connection out. That is the difference between one bad site being its own problem and one bad site being everybody's.

And be suspicious of any measurement that makes shared infrastructure look bad until you have checked what else in the path is being shared. I nearly published the opposite of what is true, and the only reason I did not is that the number looked too neat.

The full setup, both runs, the folder comparison and the two mistakes in my own test rig are in the lab write-up.