Digital workers
An AI worker needs an incident runbook before a better model
- An AI worker that answers customers or changes records is a production service. When it fails, run an incident, not a model review.
- Put heartbeats on outcomes as well as volume. Fewer replies means an outage. Resolutions, escalations or refunds moving while volume holds means wrong answers.
- In Google's postmortems, most outages followed a change. Keep the last good version one switch away and roll back before you debug.
- Name the people before anything breaks: who leads, who rolls back, who tells customers, who writes the new tests and who covers the weekend.
We think an AI worker needs an incident runbook before it needs a better model. A better model lowers the error rate, but it will not notice that the worker started giving wrong refund answers on a Saturday night, and it will not undo the change that caused it.
By AI worker we mean an agent with a defined job that runs on its own against live systems, answering customers or updating records. Once customers depend on it, it is a service.
A vendor's pager does not cover your setup
On 14 July, Fin's engineering team described how it handles incidents on the Intercom blog. It covers heartbeat metrics, pages, named roles, a rollback rule and written reviews, which is how a company that builds an AI agent runs it as a service.
If you run a worker on a vendor's platform, the vendor's pager covers the vendor's product. It does not cover your instructions, your help articles, your procedures or the permissions you gave the worker. Failures in those are yours to catch.
A tribunal in British Columbia took the same view in 2024. Air Canada's website chatbot had told a customer that a bereavement fare could be claimed after travel, within 90 days of buying the ticket. The airline's own policy page said it could not. The tribunal found the airline answerable for everything on its website, chatbot replies included. This is not legal advice.
Heartbeats belong on outcomes
Fin's heartbeat metrics go past server uptime and measure whether users can complete core tasks. One counts the responses Fin generates in conversations. A sudden drop opens an incident and pages the owning team, so nobody has to notice by hand.
That catches an outage. It does not catch a worker that keeps replying with the wrong answer, because volume holds while the damage grows. Air Canada acknowledged its chatbot's misleading answer about three months after the chat, in reply to one customer's emails.
An AI worker needs a second set of heartbeats on outcomes: resolution rate, escalations to people, reopened cases, refunds or credits issued, and complaints that mention the bot. Each needs a normal range for each hour of the week and a person who is paged when it leaves that range.
Three kinds of AI worker incident
| Outage | Wrong answers | Misuse or breach | |
|---|---|---|---|
| What you notice first | Replies slow down or stop | Resolutions, escalations or refunds move while reply volume holds | The worker's accounts make unusual calls or read unusual data |
| Heartbeat that catches it | Replies per hour, per channel | Outcome rates against their normal range for that hour | Alerts on the worker's own credentials and access logs |
| First move | Send the queue to people | Roll back the last change to instructions, content or procedures | Revoke the worker's credentials, then investigate |
Outage
- What you notice first
- Replies slow down or stop
- Heartbeat that catches it
- Replies per hour, per channel
- First move
- Send the queue to people
Wrong answers
- What you notice first
- Resolutions, escalations or refunds move while reply volume holds
- Heartbeat that catches it
- Outcome rates against their normal range for that hour
- First move
- Roll back the last change to instructions, content or procedures
Misuse or breach
- What you notice first
- The worker's accounts make unusual calls or read unusual data
- Heartbeat that catches it
- Alerts on the worker's own credentials and access logs
- First move
- Revoke the worker's credentials, then investigate
Keep the last good version one switch away
Google's site reliability engineers sorted thousands of postmortems from 2010 to 2017 by what triggered each outage. In Google's terms, a binary push releases new code and a configuration push changes settings. Changes the teams pushed themselves top the list.
For an AI worker, a prompt edit, a new help article, a changed procedure or a new model version is a configuration push. Fin's engineers are trained to roll back first and fix forward later, and a rollback currently takes a little under two minutes to land.
A rollback that fast needs two things in place before the incident. The previous configuration must be a saved version you can restore in one action, not someone's memory of what the prompt used to say. And there must be a switch that sends the worker's queue to people while you find the cause.
NIST's voluntary AI Risk Management Framework asks for the same in general terms: assigned responsibility and a way to disengage or deactivate an AI system whose outcomes stray from its intended use.
Name the roles, including the weekend
Fin names four incident roles: an engineering lead who acknowledges the page and assesses impact, an incident commander who coordinates when the response grows, a business lead who keeps Support and Sales informed, and a resolution owner who sees the follow-ups through.
An AI worker team needs the same four names, even if one person holds two of them. The worker's owner in operations commands. Someone who can restore the last good configuration leads the technical work. Someone tells support staff and affected customers what is happening. The resolution owner turns each incident into new test cases.
Fin's out-of-hours responder group, STRIKE, covers every engineering pager in evenings, weekends and holidays. On 16 July, Hugging Face disclosed an intrusion driven end to end by an autonomous AI agent system that moved into several of its internal clusters over a weekend. One of its fixes was alerting that pages a responder within minutes, any day of the week.
Plan for the model you depend on to be unavailable
Hugging Face's disclosure has a second lesson. When its team analyzed the attack log with frontier models behind commercial APIs, the providers' safety filters blocked the requests, unable to tell a responder from an attacker. The team switched to an open-weight model on its own infrastructure. It does not argue against safety filters. It advises defenders to vet a model they can run themselves before an incident.
A runbook names its tools in advance and tests them. If your worker runs on one provider's model, decide now what handles the queue when that model is down, refuses or misbehaves: a second model that has passed your tests, or a human team sized for the volume.
The runbook to write first
Six steps to write before the first incident
DetectSet the heartbeats and their normal ranges
Write down each volume and outcome heartbeat with its normal range by hour, and the named person it pages.
DeclareLet anyone declare an incident
Anyone who sees credible harm to customers can declare one. Standing down later costs less than waiting for certainty.
ContainContain before you diagnose
Take the first move for the incident type from the comparison above, even before anyone knows the cause.
RecoverRoll back to the last good version
Restore the previous instructions, content, procedures and model version as one saved bundle. Fix forward once customers are safe.
CommunicateTell the people affected
Give support staff a short script, contact customers who got a wrong answer and correct any record the worker changed.
LearnTurn the incident into tests
Write a review with specific, bounded follow-ups. Add every failed case to the worker's test set, so the same failure blocks the next change.
Where we land
A better model is worth testing once the worker can be watched, rolled back and owned. Before that, a model upgrade is one more configuration change that nobody is watching.
We argued that pilots die for lack of an operating system, not a smarter model. The runbook is the part of that system you need at the worst moment. Running the worker like staff, with its own accounts and audit log, lets you revoke one worker's access without stopping everything else.
Begin with a drill. Push a harmless change to the worker's configuration, roll it back, and write down how many minutes the rollback took. That number is the first line of the runbook. Our digital workers approach begins with the same pieces: a scoped role, the worker's own accounts and audit log, a performance dashboard and a named human manager.
Sources
- Mark Gorman, Fin, Doing the right thing when things go wrong, Intercom blog, 14 July 2026
- Hugging Face, Security incident disclosure, July 2026, 16 July 2026
- Google, Results of Postmortem Analysis, Site Reliability Workbook appendix C, 2018
- NIST, Artificial Intelligence Risk Management Framework, AI RMF 1.0, January 2023
- Civil Resolution Tribunal of British Columbia, Moffatt v. Air Canada, 2024 BCCRT 149, 14 February 2024