The Long Weekend a Hospital IT Group Learned ServerlessOps

Health Technology

Imagine a mid-sized hospital’s digital operations group, four people who look after the administrative software that sits between departments: room-status boards, supply requests, appointment reminders and billing statements. Nothing in this story is clinical, and the team and the events are invented for illustration. It follows one long weekend when a backlog formed in an event-driven system, and how the group applied ServerlessOps habits to recover without losing or duplicating anything. If you work on healthcare administration software, you may recognise the shape of it.

Friday evening: the quiet beginning

The group’s reminder pipeline ran on small cloud functions. When a department scheduled something, an event went onto a queue; a function picked it up, formatted a message and handed it to a messaging provider. For months it had worked without comment. On Friday at six, the provider began responding slowly. Each function run now waited much longer than usual, the queue’s messages stayed in flight, and a few were handed to second workers because their hidden-message window had expired.

The team lead noticed nothing at first. The error chart stayed flat, because slow is not the same as failing. What changed was a number the group had added to their dashboard only a month earlier: the age of the oldest message still waiting. By nine it read forty minutes, and by midnight it was past three hours.

Saturday morning: reading the evidence

The on-call engineer began with the Ops Atlas, whose entry on ServerlessOps lists signals for exactly this situation and sets out limits for the main platforms. She cross-checked three things. First, how long unprocessed events would be kept before being discarded; second, whether the group’s duplicate protection would still recognise a repeat a day later; and third, how much of the shared parallel allowance the slowed functions were using.

The answers were mixed. Retention on the source queue was generous, but the duplicate-protection records had been configured to expire after one hour, a library default the team had never changed. A backlog lasting longer than an hour would therefore cause replays to be treated as new messages. The engineer extended the memory of the guard to cover several days and tied it to the appointment reference, not the message ID. The Atlas is careful on this point: when work is redriven, messages get new identifiers while the business reference stays the same.

Saturday afternoon: limiting the damage

The functions were now holding on to parallel capacity while they waited, and the shared pool was filling. The team lowered the cap on the reminder function so that supply requests and room-status updates, which used the same account, would not be starved. They also shortened the timeout on the call to the messaging provider. A call that took ten seconds to fail was worse than one that failed in three and retried later, because the waiting itself was billed.

One colleague asked whether to simply raise retries. The group agreed not to. Several layers were already retrying, and the Atlas repeats a warning that attempts multiply across layers, with five layers of three producing 243 times the load. The provider was struggling, and more traffic would not help.

Sunday: draining the pile

When the provider recovered, the backlog was large but intact. The failure queue held a few hundred messages that had used up their attempts. Because the team had given that queue an owner and kept its retention longer than the source’s, nothing had expired. They redrove the contents in small batches, watching the duplicate-effect count, and confirmed that messages already sent were skipped. Partial batch responses meant that when two malformed items turned up, only those two went back rather than the whole group.

By Sunday night the oldest-message age was back under a minute. The group wrote down the timeline while it was fresh.

Monday: what they changed

  • The guard’s memory was lengthened and keyed to the appointment reference.
  • Each function received an explicit parallel cap, with a warning when headroom fell below roughly a third.
  • Alerts were added for backlog age and for the age of the oldest failed item, not only for error counts.
  • A quarterly rehearsal was scheduled in which duplicate events are fired on purpose, to confirm that nothing happens twice.

The Atlas describes such a drill as the point where a team reaches the level it calls “safe” in its own maturity ladder. The ladder is a proposal from the Atlas rather than an industry standard, but the group found the vocabulary useful for explaining priorities to their manager.

The Tuesday debrief

At the review the following week, the manager asked a fair question: why had nobody been paged earlier? The honest answer was that the old alarms measured the wrong thing. They fired when code threw an error, and a sluggish partner throws nothing. The group walked through each alert and asked which real-world harm it would catch, and which it would miss. Two alerts were retired as noise, three were added, and each new one was given a named reader and a written first step.

They also agreed to record the weekend’s costs. The slow provider had stretched every function run, so the invoice for the period was visibly higher, and the finance contact wanted to know why. A short note explaining the link between waiting time and billing saved a longer conversation later.

Lessons worth borrowing

The Atlas cites a real regional cloud outage from October 2025, during which a queue-polling subsystem failed and the provider throttled some asynchronous work to protect synchronous traffic. Its conclusion, which the imaginary team reached by itself, was that retention and duplicate-protection expiry must both be longer than the worst backlog you can picture. Short defaults do not announce themselves. They wait for a bad weekend.

The full reference can be read at the Ops Atlas page devoted to ServerlessOps, including its caveat that the label is an older one and no body certifies it. Readers interested in the cost side of cloud operations may like our piece on learning practical cloud cost control, and the administrative side of remote consultations is covered in a roadmap for people using virtual clinics.