Evals and accuracy
Evals are half the job. Releases are the other half.
- An eval shows that a change passed your test conversations. It cannot show how the change behaves with live customers, so treat every change to a support agent as a release.
- In a support agent, an edited help article, a new procedure or a policy tweak is a production change. Version them together, with an owner and a list of what changed.
- Send each release to a small share of conversations first, compare its outcomes with the current version, and keep a one-step rollback.
- Measure the releases the way engineering teams do: how many you rolled back, and how long each rollback took.
We think a support team that only tests its AI agent has done half the job. The other half is a release process that decides how each change reaches customers and how fast it can be undone, because no test set holds every question customers will ask.
On 13 August, Fin announced Evals and Releases for its customer service agent. Evals score themed suites of simulated customer conversations pass or fail. Releases bundle changes to content, procedures and guidance, test them, then publish them to everyone, ramp traffic gradually or run an A/B test that splits conversations between the new and current setup, with a one-step rollback. Fin pairs both with Monitors, which check live conversations, and calls the loop eval-driven delivery.
A help article edit is a production change
Part of Fin's stated reason for building both is drift. Support teams change the agent's context whenever they launch a product, adjust a policy, rewrite a help article or add a procedure, and one change can break another without anyone noticing. Fin says that at the scale some of its customers run, a 1% regression could affect thousands of conversations a day. At a smaller desk, the same 1% is still one conversation in a hundred.
Software teams now have the same problem. On 12 August, CodeRabbit raised $143 million at a $1.5 billion valuation and introduced Agentic Change Management, which it calls a control layer for software changes made by people and agents. Its launch post notes that anyone on a team can now start a code change, product managers and marketers included.
Support teams reached that point earlier. Whoever edits a help article that the agent reads changes what the agent tells customers. We think the help center, the procedures and the guidance are part of the agent's configuration, and a change to any of them is a release.
Why a passing eval is not a release
An eval is a sample. It holds the questions someone wrote down or copied from past conversations, and customers will ask others. Even one test conversation is a sample, because Fin, like any agent, can answer the same question in different ways. Anthropic's guide to agent evals shows the cost: an agent that succeeds 75% of the time passes three runs in a row about 42% of the time.
The same guide splits the work. Automated evals are the first line of defense before launch and on every change. Production monitoring catches drift and failures nobody anticipated, and A/B tests confirm significant changes once there is enough traffic. Two of those three checks happen after launch, and a release process is how a team runs them.
Tested once against released in stages
| Tested once, then published | Tested, then released in stages | |
|---|---|---|
| Who meets a bad change first | Every customer the agent talks to | The small share of conversations in the first stage |
| How the problem shows up | Complaints, escalations and reopened cases, days later | A gap in outcomes between the release and the current version |
| How fast it can be undone | As fast as someone can rewrite the old content | One step back to the previous release |
| What the team learns | That something broke | Which change broke it, plus a new eval case that blocks a repeat |
Tested once, then published
- Who meets a bad change first
- Every customer the agent talks to
- How the problem shows up
- Complaints, escalations and reopened cases, days later
- How fast it can be undone
- As fast as someone can rewrite the old content
- What the team learns
- That something broke
Tested, then released in stages
- Who meets a bad change first
- The small share of conversations in the first stage
- How the problem shows up
- A gap in outcomes between the release and the current version
- How fast it can be undone
- One step back to the previous release
- What the team learns
- Which change broke it, plus a new eval case that blocks a repeat
Stage the rollout
Software teams answer this with canary releases. Google's Site Reliability Workbook describes canarying as sending a change to a small part of a service for a limited time and comparing it with the rest, which acts as the control.
The workbook's example shows why. A defect that fails 20% of requests, sent to 5% of traffic, fails 1% overall. In a single overall error rate, that 1% can look like ordinary noise. Split by group, the canary's 20% is obvious.
For a support agent, the canary is a share of conversations on the new release, and Fin's A/B reporting compares resolution rate, escalation rate and CSAT, its customer satisfaction score, across the two groups.
The workbook also warns that a canary needs enough traffic, at representative times, to mean anything. For a support desk, that means covering your busiest days. If volume is too low for a fair split, ramp slowly and read a sample of the new release's conversations by hand.
A release process for a support agent
Five steps from change to full traffic
VersionBundle the change into a named release
Content, procedures, guidance and model settings that must ship together go in one release, with an owner and a list of every edit.
TestPass the eval suite before any customer sees it
Rerun every themed suite, including the ones you did not touch, and run each simulation more than once.
StageSend a small share of conversations to the release
Keep the rest on the current version as the control, and widen in steps.
WatchCompare outcomes against limits written in advance
Resolution, escalation and satisfaction for each group, plus a sample of new-release conversations read by a person.
DecideWiden, or roll back in one step
Roll back the whole release when a limit is crossed, and turn each failed conversation into a new eval case before the next attempt.
Measure the releases as well as the agent
Engineering teams already have numbers for this. The DORA guide to software delivery performance metrics lists five, and two matter most here. Change fail rate is the share of deployments that need immediate intervention, often a rollback or a hotfix. Failed deployment recovery time is how long it takes to recover from one. DORA says the metrics fit any system, large language models included.
For a support agent, we would track the share of releases rolled back and the minutes each rollback took, next to how many releases shipped. DORA's advice on batch size applies too, since smaller changes are easier to recover from. Bundle what must ship together, such as a product launch, and keep everything else in its own release, so a failed release points at one cause.
Where we land
Evals answer whether a change is ready to try. A release process answers who sees it first, how far it goes and how fast it comes back. A support agent needs both, because small errors compound across a multi-step procedure, and a slight drop at each step becomes a large drop end to end.
Staging a release puts a change on probation, as we recommend for AI employees before they get full duties. When a release does go wrong in production, the incident runbook takes over.
Check the last three changes your team made to the agent. For each, ask whether you could have rolled it back in one step, and how long that would have taken. Any answer of no is where the release process begins. Our digital workers approach applies the same idea to the whole worker: a supervised probation before full duties, then a performance dashboard and a human manager loop.
Sources
- Brian Donohue, Fin, Announcing Evals and Releases, 13 August 2026
- CodeRabbit, Series C announcement and Introducing Agentic Change Management, 12 August 2026
- Anthropic, Demystifying evals for AI agents, January 2026
- Google, Canarying Releases, Site Reliability Workbook chapter 16, 2018
- DORA, DORA's software delivery performance metrics, updated January 2026