Evals and accuracy
AI code ships faster than it is tested
- Spotify's merged changes more than doubled in a year. In the incidents it reviewed, AI-written code was not a material direct cause. Checks that could not keep pace with the volume were the risk.
- We think verification has to grow with change volume: tests on every change, repeat runs for anything an agent produces, and rollback that takes minutes.
- Average pass rates hide the tasks an agent gets right only sometimes. Count a task as passed only when every repeat run passes.
- Budget for verification capacity the way you budget for compute, and compare it with change volume every month.
We think the bigger risk in AI-assisted development is the number of changes, not the quality of each one. When AI doubles what a team ships, the tests, repeat runs and rollback behind it have to double too, and that capacity is the part of the budget that keeps the extra output safe.
What Spotify found
On 16 September, Tyson Singer, who heads technology and platforms at Spotify, published what the company learned about quality as AI sped up its engineering.
Spotify in August 2026, against August 2025
Volume
About 17,000
merged changes in the month, up from about 8,100 a year earlier
Mix
31%
of merged work was quality and optimization, up from 27%
Every month Spotify reviews all its major incidents. During its AI ramp-up it added two questions: did AI-authored code contribute directly, and did the extra volume strain review, testing, rollout or observability? So far it has not found AI-authored code to be a material direct contributor. It did find the second risk. In Spotify's words, "the volume of change increased faster than some of our verification controls could adapt."
One case shows the shape of the problem. An automated dependency upgrade passed Spotify's checks and still failed in production, affecting users. Spotify's response is stronger safeguards, more rollback capacity, and scheduling automated changes inside the owning teams' working hours.
That fits the central theory of the 2025 DORA report, Google Cloud's survey of nearly 5,000 technology professionals. DORA found AI adoption linked to higher delivery throughput but still to lower delivery stability. It argued that without strong automated testing and fast feedback, more change leads to instability.
Passing once is weak evidence
Automated changes fail in a particular way. Spotify's background coding agent team wrote in December 2025 that its most serious failure is a pull request that passes continuous integration, the automated build and test run, yet is functionally wrong. Little or no test coverage in the target code was one cause they named.
The team added a second model that judges whether each change stays within its request. It vetoes about a quarter of agent sessions, and some agents had tried to disable flaky tests, the ones that pass and fail on the same code.
Agents also vary from one run to the next. IBM Research ran an agent five times on each task at temperature zero, a setting meant to remove randomness, and tasks still flipped between pass and fail.
Average pass rate against passing every run
| Baseline agent | With IBM's guidelines | |
|---|---|---|
| Average pass rate across five runs | 77.4% | 81.0% |
| Tasks passed in all five runs | 53.0% | 69.0% |
IBM traces the flips to near-tied decisions, and a run chains dozens of them, the same compounding we described in why small error rates explode. We would count an agent-made change as passing only when every repeat run passes.
The suite has to grow too
Doubling the changes doubles the test runs, and the noise in them. Google's testing team reported in 2016 that about 1.5% of its test runs were flaky, and that about 84% of the pass-to-fail transitions it saw involved a flaky test. Google also wrote that engineers often ignore real failures in flaky tests because of the false alarms.
Tests written by AI need checking as well. In VibeCheck, a small study posted on 5 September, three coding agents wrote unit tests for 15 student repositories. The tests often ran, but weak assertions and missing edge cases were more common than outright failures.
A test with weak assertions passes on broken code, so its green result proves little. One check is to plant a small bug on purpose and confirm that a test fails, which is called mutation testing.
Forty pull requests a day without a QA team
On 24 August, solo founder Ryan Carson told How I AI, a Lenny's Newsletter podcast, that he runs 15 of Cognition's Devin agents at once. A skill he built handles 40 pull requests a day, with review loops, a video walkthrough and auto-merge, and no QA team reviewing them. He and host Claire Vo agreed that more AI output is the wrong goal.
We give him credit for building review into the loop instead of skipping it. At that volume the automated gates are the QA team, and they deserve the budget and the owner a QA team would get. In August we argued for buying the suite and its evidence instead of tester hours.
Approval is moving to models too. Since 1 September, GitHub lets admins allow a Copilot approval to count toward a repository's required approvals, in public preview. Each approval a model can give puts more weight on checks that execute, and we set out which changes a model may approve in a written risk policy for AI approvals.
What to put in place
We would set these up before the change rate doubles, not after the first incident.
Verification that grows with change volume
Chart change volume next to verification capacity
Each month, plot merged changes beside pipeline minutes, reviewer hours, rollbacks and incidents. When changes grow faster than the rest, add capacity before adding agents.
Run the full suite on every change, whoever wrote it
No change skips the pipeline because a bot or an agent opened it.
Repeat-run what agents produce
Run agent tasks and new tests several times, and count a pass only when every run passes. Track the flaky rate, and give each quarantined test an owner and a date to fix it.
Test the tests
Plant small bugs on purpose and check that a test fails. Reject AI-written tests whose assertions would pass on broken code.
Make rollback one action, and rehearse it
Every automated change should be revertible by the team that owns the service, without a new build. Run a rollback drill before an incident needs one.
Ship automated changes when owners are online
A person who knows the service should be at work when an automated change lands.
Add the volume question to incident reviews
Ask whether a change came from AI and whether the change rate strained review, testing or rollout. Count the answers over time.
Where we land
Verification capacity belongs in the budget beside compute, as a line that grows with change volume and gets reviewed monthly. Our human-in-the-loop approach follows the same order: automated checks on every change, repeat runs where results vary, and people on the cases the checks cannot settle, with each correction turned into a new test.
Start with two numbers from last month: merged changes, and the hours your pipeline and reviewers spent checking them. If the first grew faster than the second, that gap is the first thing to fund.
Sources
- Spotify Engineering, Tyson Singer, AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity, 16 September 2026
- Spotify Engineering, Max Charas and Marc Bruggmann, Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3), 9 December 2025
- Google Cloud, Nathen Harvey and Derek DeBellis, Announcing the 2025 DORA Report: State of AI-assisted Software Development, September 2025
- IBM Research on Hugging Face, Evelyn Duesterwald and colleagues, Your Agent Aced the Task. Will It Do It Again?, 15 September 2026
- Google Testing Blog, John Micco, Flaky Tests at Google and How We Mitigate Them, 27 May 2016
- Tabassum and colleagues, VibeCheck: Assessing the Quality of LLM-Generated Unit Tests, arXiv, 5 September 2026
- Lenny's Newsletter, Ryan Carson and Claire Vo, I spent $20,000 on Devin in a month. Here's what I learned, How I AI, 24 August 2026
- GitHub Changelog, Copilot code review can now approve pull requests, 1 September 2026