Networks and servers · monitoring and Linux
How a chance warning on a network map led to a network storm investigation and the discovery of an unknown administrator
I had only just started building a network monitoring portal. One of the first checks watched the CPU load of the central network equipment: when it crossed a threshold, a red exclamation mark appeared next to the device on the map.
That was all. Monitoring did not yet record such warnings as incidents and did not notify anyone. The load could spike for a few minutes and settle before the next poll, and the mark would vanish without a trace. One day, while working on the map, I happened to notice it. That is where the investigation began.
During the overload
A couple of minutes later
Event logempty
Simplified diagram with a made-up load value. In the early version of monitoring the warning lived only on screen: as soon as the load returned to normal, nothing was left of it.
The check showed that the unusual traffic was coming from one client segment. The storm rolled in waves: a surge, a lull, another surge. Between the waves everything looked normal, which made the mark easy to miss.
A network storm is like a building's group chat where someone starts forwarding the same message over and over, and everyone who receives it forwards it to everyone too. A minute later the chat is unreadable and the messages that matter are buried. In a network, the messages are packets addressed to everyone at once. Every device has to accept and process them, and the central equipment burns its CPU on junk instead of useful work.
One test in September shows the scale. The port facing the segment was briefly switched on with interim limits, and in about three seconds 3,478,763 packets came through it. Only 64 of them were addressed to a specific recipient; everything else was broadcast and multicast.
I searched from the centre of the network outwards. On every device along the way I checked which port received the largest incoming flow and which port it left through, and used neighbour data to find out what was connected to that port. The chain ended at a port of an aggregation switch, behind which the client's network began: a property management company with its own equipment.
warning threshold
The shape of the waves is illustrative and shows only the nature of the storm. The real figures are in the cards below.
From our side there was no way to see how the client's network was built inside. So the first step was to shut that port down, and the storm stopped reaching the rest of the network. The segment stayed disconnected until there was a chance to investigate on the client's equipment itself.
A trial switch-on in September showed that the storm was still there. The interim limits did not hold it back: within about three seconds even the switch's management session began to fail, and an emergency connection opened in advance shut the port down at once. After that, simply switching the port on again was out of the question.
From then on the port was enabled only in quarantine: a single service network, about 512 kbit/s, no more than roughly a hundred packets per second for each kind of broadcast, a filter for three known devices, a ready emergency session and a timer that would shut the port down by itself. First windows of 6 to 40 seconds, then 15 and 30 minutes. In one 40-second window 85,975 packets reached the port, and nothing got further into the network.
The client's MikroTik that the storm came from was looked after by another system administrator on the management company's side. From there we worked together: I kept the quarantine on our port and opened access windows, and they were responsible for the device itself and its network.
In quarantine the router finally answered: ping went through without loss, SSH asked for a password. But logging in failed, because the known passwords did not work. I did not keep guessing: the device belongs to someone else, and its owner is the one to decide what to do with it. I closed the next window, with a new password, early: the safety check did not finish in the allotted time, and at the slightest doubt the port has to stay off.
In the end we agreed on a reset. While the work went on, the client's network was connected through a temporary router on another port. Before the reset, the MikroTik's user list turned out to contain an account named System with full rights, which none of the people responsible for the device had created. Its last login was recorded on 1 May 2026, the day after the storm was first logged.
After the reset the client's administrator set the MikroTik up again, and our port was moved to a separate network for new connections, where the device gets its address automatically and carries no old settings with it. While we were watching, the storm did not come back. The next day the internet worked through the new path, and the temporary router was switched off.
Stylised router management window: a user list that includes an unknown System account in the full group with full rights.
A stylised fragment with made-up data. Only the name System and the full group match the real device; names, times and comments are invented.
We never found out exactly how it all started. It remains unclear whether the storm and the System account were connected. There is no information that System reappeared after the reset.
A router looks like a box with blinking lights, but inside it has an operating system, programs and user accounts, much like a computer. And wherever there are programs, there are bugs that can let a stranger in or give someone more rights than they should have. That is why such devices also need updates, and why their management should not be open to the whole internet.
In autumn 2026 MikroTik released fixes for several RouterOS vulnerabilities at once. Among them were SSH bugs that, combined, allowed full control of a device. On 5 September CERT Polska reported real attacks since at least 2 September: on hijacked routers the attackers created an account of their own. At the end of September the US agency CISA published an advisory about another critical vulnerability, in the web management interface.
Our events happened in spring, months before these publications, so attributing them to any specific vulnerability would be wrong. How System got onto the device is unknown. But the story shows well why a foreign account on network equipment calls for a reset rather than deleting one line: you cannot know in advance what else has been changed.
Sources: MikroTik security advisories · CERT Polska, 5 September 2026 · CISA ICSA-26-272-06, 29 September 2026
The first monitoring showed only the current state: what is happening this minute. For a storm that comes in waves, that is not enough. The warning vanished together with the load, and if nobody was looking at the map at that moment, it was as if the problem had never happened. I saw the mark by chance, which means I could just as easily have missed it.
It made clear that monitoring needs a memory. An event should be kept even after everything has calmed down: when it started, how long it lasted, whether it repeated. A person should be told about it, and after recovery the system should check that everything really is back to normal.
The portal grew step by step. Right after the investigation the equipment began saving snapshots of CPU load and traffic on its own whenever they crossed a threshold, so the next wave would not have to be caught by hand. Then came incidents with history: a recurring problem reopens the existing incident instead of creating new ones. After that, notifications to an engineer in MAX, a separate message on recovery, automatic closing only after stable normal operation, and an external check that reports when the portal itself is down. A simple map with icons became a system that lets you reconstruct what happened and when.
Then: only the present
Now: an event with a history
High CPU load · network core
three surges in a day, kept in the history
Time, duration and number of repeats are made up; the card shows what is now recorded about an event.
The second part of the work is Linux servers that run the portal, billing and background jobs. They have a similar trap: the server answers ping, SSH lets you in without trouble, and meanwhile a particular service or scheduled job is not doing what it should.
On the portal card of one device, question marks sometimes appeared for a few minutes instead of the port states. Yet the device itself was online, and the job that collects port states finished successfully every time.
The job was scheduled every four minutes starting from the fourth minute of the hour: 04, 08 and so on up to 56. After minute 56 the next run came only at 04 of the next hour: almost eight minutes, not four. Port data counts as fresh for five minutes, so once an hour it went stale, even though not a single run ended in an error.
The run history gave it away: 479.7 seconds passed between two neighbouring starts. The fix was one line of the schedule: the runs moved to 01, 05 … 57, and the step became exactly four minutes, across the hour boundary too. I did not extend the freshness period. I closed the task only after a real 11:57 → 12:01 transition passed without stale data.
Timeline in minutes, scale is illustrative. The eight minutes and the five-minute freshness are the real values of this case.
Upgrading the virtualisation servers to a new version went by the book: the nodes rebooted, the virtual machines started, SSH answered. But billing did not work. The new version started its virtual machine with a different type of virtual hardware, the billing licence is bound to the hardware, and the system decided the machine was a stranger. The licence was rejected, and the services that handle authorisation and traffic accounting kept exiting with the same licence error. The service manager started them again, and they failed again.
Restarting them once more made no sense: the cause was still there, and every new start ended the same way. What helped was fixing the cause itself. I pinned the previous virtual hardware type for this machine, restored the licence record from a local database copy made before the work, and restarted the main billing service once. The licence was accepted, and subscriber sessions were accounted again. That day authorisation and traffic accounting were down for about an hour.
The story had a tail as well: after the recovery some subscribers could not connect because of stuck sessions in billing. There is a separate write-up about that →
What I check after an upgrade
It all started with a small red mark that would have vanished without a trace a couple of minutes later. Had I not been looking at the map at that moment, the storm would have kept rolling in for a while, and the strange account on the client's router might have gone unnoticed.
Since then I look at monitoring differently. Showing the current state is only the start. What matters is keeping the event, telling a person about it and checking that after recovery everything really works. I still think of that chance mark every time I add a new check to the portal.