Part IV — The Storm
Servicing interrupts faster is not the same as receiving fewer of them.
Isolate the interrupts. Separate the tactical from the structural. Lower instead of merely raising . Let the failures train the people. Protect the planned work.
Priority queues were invented for a reason. Interrupt-driven work and planned work do not belong on the same run queue. Together, the interrupts starve the planned work — every time, not occasionally, and not because anyone lacked discipline.
It’s what happens when two priority classes share a queue and one can preempt. So: two daily scrums. A delivery scrum, plus a separate fifteen-minute scrum for on-calls. Separate queue, separate scheduler, separate pool. The segregation is the precondition for everything that follows.
Different Org, Different Scale, Different Problems.
I joined an e-Commerce organisation after they were acquired by one of the largest business houses in the world. Post-acquisition, operational scaling broke the system. Enterprise integrations — IAM migrations, SAP ERP bridges, POS retail syncs, and B2B pipelines — hit simultaneously. Order volume exploded, taking customer and internal on-call PagerDuty calls with it. Churn spiked, and engineering capacity withered. Management’s default response was “work harder™” or assign raw recruits.
The problem was fairly deep, but how deep was realised when we measured it: roughly six thousand on-calls a year. The backlog started exploding due to competing priorities. Commitments due to the acquisition couldn’t be pushed but neither could the Customers’ grievances1.
I took a third of the incoming intern cohort and put them directly to the interrupt queue. So eight people, dedicated to the interrupt/on-call queue, all fresh and had not a clue2 about how to solve the issues.
This Bounded pool = Bounded blast radius: the storm still existed, but hit eight people instead of seventy five.
With persistence & patience the platoon started the tactical fixes first, but very soon every on-call GENERATOR got decomposed into two fixes, both mandatory: - Tactical: restore service now. Minimal, fast, interrupt context. Raises . - Structural: make this CLASS of on-call stop arriving. Deferred to scheduled context. Lowers .
That is precisely how an OS handles interrupts. Staying in interrupt context is what livelocks the system3.
Teams invariably ship the top half, because it stops the pager — and the moment this pager stops, the incentives move to the next page. Mandating the bottom half4 is the entire difference between servicing interrupts faster and receiving fewer interrupts.
And this is what the on-call scrum was structurally FOR. It’s the workqueue — the scheduled context in which bottom halves actually run.
Phase 1 — stabilise. Top halves. Triage, ownership, service restored fast. up, drops below 1, the queue stops diverging. Everyone attempts this phase. Phase 2 — drain and hold. Bottom halves compound. Each structural fix permanently removes a generator, so falls and KEEPS falling. Almost nobody reaches this phase, and it is the only reason the number stayed down.
Phase 1 took about a quarter. Not a quarter to drain the queue – a quarter for the slope to reverse, which is the earlier event and by far the more informative one. A backlog that is still large but has started falling is a solved problem waiting on time. A backlog that is flat and enormous is not.
That moment is also where most engagements end. The chart has turned, the pager is quieter, and everyone would now like to talk about the roadmap. Phase 2 is the dull part – removing one class of generator at a time, in scheduled context, long after it stopped being interesting5. The ten-fold came from the dull part. Phase 1 only buys the right to attempt it.
One caveat on what is being counted, because it is the usual place a claim like this quietly inflates: these are valid on-call tickets reaching engineering, not raw traffic into the group mailbox. The mailbox stayed busy throughout. Conflating the two is how an organisation argues itself out of a result it has actually achieved.
Nobody wanted this. It is worth being precise about who objected and on what grounds, because the three objections were not one objection, and only one of them was wrong in an interesting way.
The senior engineers split. One camp wanted the interns pointed at their grunt work – the tickets they were tired of. That is a different queue with a different sampling function, and work selected by whoever is most annoyed teaches nothing in particular. That version of the idea looks identical from the outside, is worthless, and is the version most likely to be built by somebody copying this. On-call is worth something as a curriculum6 precisely because it is sampled by failure frequency rather than by irritation.
The second camp wanted the interns on new features. This is the generous instinct and the conventional one, and it is right about the individual while being wrong about the system7. Features fail slowly, ambiguously, and months downstream of the decision that caused the failure. On-call fails in minutes, unambiguously, with a stack trace attached. If the object is to build(or how not to build, depends on perspective) an engineer, the second signal is worth more, and it is available in week one.
The platoon objected as well, and theirs was the correct objection8. Fixing other people’s bugs is not a dream first job and no amount of queueing theory makes it one. That cost was real, it was paid by eight people who did not choose it, and the case for it was a promise about the future: that a quarter of being paged by the system would leave them understanding it better than most of the people who built it. The promise happened to come true. It was not verifiable at the time, and anyone proposing this arrangement is asking to be trusted on precisely that point.
A quarter of fixing on-calls trained eight interns in a vast OMS-to-fulfillment ecosystem. Looks like a happy accident. It isn’t — it’s structural: On-call is the most complete curriculum an organisation possesses, because it is importance-sampled by real failure frequency.
No syllabus designer could produce it. It teaches exactly the parts that break, weighted by how often they break, in the order they break. They didn’t learn the ecosystem by reading it. They learned it by being paged by it. In the model’s terms: they started cold, every page a cache miss. A quarter later the working set was resident.
The platoon absorbing interrupts is what made the delivery pipeline uninterruptible. You cannot run a preemptive scheduler on a pipeline that something else is already preempting. Ticks, small batches and a bounded horizon only work on protected capacity. Reliability was never competing with delivery for the same people. Fencing it off is what made delivery possible at all.
And why the parts are ordered as they are: the right actors on the right queues (I), instrumented well enough to see (II), scheduled often enough to run the bottom halves (III and IV).
None of this had to be taught to a room of seventy five application engineers. They already knew what an unstable queue costs. They had simply never been told that the thing they were standing in was one.
Again almost like choosing rent or food.↩︎
The system had documentation. It was as accurate as of the last Ice Age.↩︎
Any baremetal / OS splits handlers into a top half in interrupt context doing the bare minimum, and a bottom half — softirq, tasklet, workqueue — deferring heavy work to a schedulable context. Trouble stems when there is always an interrupt waiting while the CPU is serving one!↩︎
Every organisation ships the tactical fix. The structural fix is scheduled for “next quarter” — a unit that shares its name with a calendar period but behaves rather more like a horizon in general relativity: permanently visible, never actually reached.↩︎
Every organisation ships the tactical fix. The structural fix is scheduled for “next quarter” — a unit that shares its name with a calendar period but behaves rather more like a horizon in general relativity: permanently visible, never actually reached.↩︎
No training budget was spent on any of this. I mention it because I have since sat through several presentations on learning-and-development strategy, and not one of them proposed “allow the system to page them at three in the morning for twelve weeks” — which is a pity, because it is the only method I have seen work at scale.↩︎
And we had many. It wasn’t just an OMS or a fulfillment Systems as in an e-Commerce, it was the complete Supply Chain of an e-Pharmacy with the nuances of regulations also built in. On top of it, mostly built by interns whose zeal to write code is unmatchable. If only they could read code, I wouldn’t have inherited a 10,000 lines;Python.↩︎
There is a moment, somewhere around the fourth page of the night, when an engineer stops fixing the system and starts updating their CV. Nobody instruments this. It is the most predictive metric you have.↩︎