Engineering note · IPoE and LANBilling
How we tracked down stuck IPoE sessions and automated getting subscribers back online
A subscriber calls: no internet. The router has been rebooted, the cable is fine, the optical signal is normal. Yet the authorization log shows the same line over and over — AlreadyConnected. The billing system is sure the subscriber is already online.
Normal session
Accounting-Update every few minutes
Accounting-Stop lost
A crash, a lost link or a billing restart
The states diverge
AlreadyConnected reject
The device asks to connect again, and the billing system replies that this subscriber is already online
On an IPoE network there's no login and password in the usual sense. The access server sees a device, asks the billing system over RADIUS and, after an Access-Accept, opens a session. Every few minutes it sends an Accounting-Update, and on disconnect an Accounting-Stop, at which point LANBilling closes the session on its side. As long as the messages get through, the records on the access server and in the billing system match.
The billing system was restarting, the link between servers dropped, the access server lost its sessions after a crash — and the Accounting-Stop never arrived. The access server has already forgotten the session, while LANBilling keeps it open. The subscriber's device asks to connect again, and the billing system rejects it with AlreadyConnected: as far as it knows, the subscriber is already online.
Rebooting the router or the ONU produces a new request, but the open session in the billing system stays and the rejection repeats. The stale-session timeout in our setup is one day, and even after it the session doesn't close on its own. A standard Disconnect from the billing side doesn't help either: the access server doesn't know that session and answers with a Disconnect-NAK, so the billing system keeps the record open.
The portal already had automation for recovering after a billing restart: it removed outdated client entries on the access server so the device would go through authorization again. After maintenance it found fifteen subscribers without access, did its job — and on the next cycle found the same fifteen. This went on for over an hour, with no errors and no alert to the engineer. All fifteen sessions were still open in the billing system, their accounting had stopped a few minutes before the outage, and the automation did nothing about that record.
The standard Disconnect returned a Disconnect-NAK. I didn't want to edit the database tables directly: that bypasses the billing logic. The LANBilling API has a method that ends a RADIUS session by its exact attributes. The first call answered “success”, but the session stayed: while the request was being built, the access server address turned into an empty value. Since then success means only a fresh read in which the session is gone. I ran the fixed call for one subscriber first, waited a full accounting interval, and only then did the other fourteen. Everyone got their access back.
The first automated version took its candidates from the comparison with the access server. Soon there was a subscriber it couldn't see: their entry on the access server lived for a few seconds, while the comparison, run every two minutes, only counts entries older than a minute. The person was without access for more than a day. That's how the second source appeared — the authorization reject log in the billing system. Along the way the cycle learned to start not only after the billing server reboots, but also when its services restart and when the comparison finds the same thing several times in a row.
Every two minutes the automation gathers data from three places and decides whether there is a session it can safely close.
1
Open RADIUS sessions: account, MAC, access server, time of the last Accounting-Update. Through the API — the account's state: tariff and blocks.
2
AlreadyConnected rejects: when the last one happened, how many attempts, from which MAC and through which access server. Exported once a minute, read-only.
3
Whether this MAC has an active session right now. A state snapshot no older than five minutes.
Checks
If any condition fails, the session is left alone and the reason is logged
Session ended through the LANBilling API
Recovery check, up to 30 minutes
New session and accounting → “recovered” message
Not confirmed → detection only, message to the engineer
If everything adds up, the automation calls the LANBilling API to end the RADIUS session with the attributes it has just read, then re-reads the list several times: the API sometimes shows a deleted session for a couple more seconds. After that it waits up to half an hour for confirmation: a new session with the same MAC, accounting for it, the subscriber active on the access server, tariff and blocks unchanged. Until then the case stays open.
Closing every session without fresh accounting is the simplest option and the most dangerous one: accounting can be late for a live session too. When I tested the filter on live data, several dozen accounts had AlreadyConnected rejects that day, almost all of them harmless — the device asked for access again while its session was working. Only one session was really stuck. So a session with fresh accounting is protected, and if data is missing or out of date, the automation touches nothing and logs the reason.
Times are counted from the reset, taken from the controlled recovery.
Before switching on automatic mode, I ran one controlled recovery — on the very subscriber the first version couldn't see. A dry run showed every check passing, and the reset was done for that subscriber only:
That run caught the last bug. For a second after the deletion the API still listed the session, and the automation recorded a successful reset as a failure. The check was reworked to read several times.
After that, automatic mode was switched on for good. The limits for the first period are deliberately strict: no more than one reset per cycle and no more than two attempts per account. If the old session comes back after a reset, the deletion isn't confirmed, or the subscriber's access settings change, the automation switches itself to detection only: it keeps finding stuck sessions but resets nothing else. A message goes to MAX straight away. Automatic mode can only be switched back on by hand, once the cause is understood.
I won't promise that any subscriber now recovers within two minutes. Two minutes is how often the check runs. A session first has to go quiet for 15 minutes, the subscriber has to try to connect a few times, and with one reset per cycle a large outage is worked through one session at a time. What has changed is that a stuck session no longer waits for the subscriber to get through to support and for an engineer to dig into the billing system by hand.
The automation messages the engineer in MAX about every reset and every failure. The messages contain no MAC addresses, names or contract numbers — only counts, the mode and a reason code. The details are in the portal.
Demo example. Times, counts and statuses are made up; the format follows the real alerts. The bot writes in Russian — the messages are translated for this page.
Today
IPoE: stuck session closed automatically
Clients without an active session on the access server: 1
Stuck sessions in billing: 1; recovered: 0; not recovered: 0
Automation: automatic closing on
Reason: none
IPoE: session recovery complete
Clients without an active session on the access server: 0
Stuck sessions in billing: 1; recovered: 1; not recovered: 0
Automation: automatic closing on
Reason: none
Open background jobsToday
IPoE: attempt to close a stuck session failed
Clients without an active session on the access server: 1
Stuck sessions in billing: 1; recovered: 0; not recovered: 1
Automation: automatic closing on
Reason: delete_without_effect
Check the “IPoE recovery after boot” job. Do not reset all clients.
IPoE: automatic closing stopped, detection only
Clients without an active session on the access server: 1
Stuck sessions in billing: 1; recovered: 0; not recovered: 1
Automation: detection only
Reason: delete_without_effect
Open background jobsThis is the first note in a series about how the portal works. The next ones will appear in the “Engineering notes” section of the main case.