Buying guides
Your AI vendor can be sold. Keep your test set.
- An AI support agent priced per outcome is sold on its resolution rate. That rate is a test result: someone chose the conversations, the definition of resolved and the scoring.
- Fin's own guide says differing definitions of resolved can move a reported rate by 20 to 30 points. When you pay per outcome, the definition sets your invoice.
- Keep a test set built from your own past conversations, and a copy of the agent's setup outside the vendor's tool. With both, you can check the claim every month and move if you need to.
- Salesforce's agreement to buy Fin, announced on 15 June, is a reminder that the company behind your agent can change.
We think anyone buying an outcome-priced AI agent should keep two things outside the vendor's control: a test set built from their own conversations, and a copy of the agent's setup. Without them, a buyer cannot check the resolution rate on the invoice, and cannot leave without starting again.
On 15 June, Salesforce agreed to buy Fin, formerly Intercom, for about $3.6 billion, with closing expected in the fourth quarter of Salesforce's fiscal 2027. Fin's chief executive, Eoghan McCabe, wrote to customers that "little will practically change." We take that at face value.
The release lists the integration of Fin and possible disruption to business relationships among the deal's risks, which is standard language in a deal announcement. Our argument is about any vendor, not this one. Owners, prices and priorities can change during a contract, and a buyer who holds the test set can check what changed instead of relying on promises.
A resolution rate is a test result
Every test result rests on three choices: which cases are in the test, what counts as a pass, and who does the scoring. A resolution rate rests on the same three.
The Salesforce release cites Fin's agents resolving on average 76 percent of support volume end to end. Fin's guide to resolution rates, published on 17 June, defines the rate against the conversations the agent handled, and puts Fin's average at 76 percent across more than 12,000 customers. Those bases differ whenever the agent does not see every conversation. Both descriptions may be fair. The base is a choice someone made.
The guide is candid. It says vendors define resolved differently, enough to move a reported rate by 20 to 30 percentage points. It calls resolution rate the metric vendor marketing cites most, and "also the most commonly manipulated." Its seven questions for buyers include whether you can audit the conversations, and whether the system scores itself or has an independent check. We agree, and think the independent check should belong to the buyer.
When you pay per outcome, the definition sets your invoice
The same guide gives Fin's price as $0.99 per outcome. When a bill counts outcomes, the definition of an outcome sets the bill, and definitions move. Fin changed its billing unit from resolutions to outcomes on 12 March 2026, a broader measure it says reflects multi-step queries, transfers and completed tasks. It said customers' setups and performance were unaffected.
That is a normal product decision, and it shows what a buyer should write down. The contract should name the base, the period in which a repeat contact about the same problem cancels a resolution, and how abandoned conversations and negative replies count. Then check a monthly sample of billed outcomes against those words. This is not legal advice.
Build the test set from your own conversations
A test set, in plain words, is a fixed list of past conversations, each with the outcome a good agent should reach. Fin's evaluation guide for ecommerce teams tells buyers to build one from 90 days of real support history, and to give every vendor the same conversations, knowledge base and scoring rubric. We agree, and add one step: keep the set after you sign.
Real conversations make better tests than written prompts. For an OpenAI paper published on 16 June, researchers took about 1.3 million de-identified ChatGPT conversations, with personal details removed, from four production deployments. They kept each conversation's opening turns and had a model awaiting release write the next reply.
In separate calls, a model asked whether each conversation was a test flagged 98.9 to 100 percent of conversations from standard evaluations as tests, against 5.38 percent of real ones. The replayed conversations were flagged at close to the real rate.
Replays still miss things. The paper forecast one GPT-5.4 behavior at 1.7 cases per 100,000 turns, and after launch it ran at 35.2, about 21 times higher. The authors trace their largest errors mainly to replay environments that did not match production, including tools that depend on live data. In a support agent, those tools are order lookups and refunds. So test before you sign, and keep sampling live conversations after go-live.
The set should also grow. On How I AI, Ankur Goyal, chief executive of the eval software company Braintrust, calls evals "the modern version of a PRD", a product requirements document. An eval is a test case with a known good answer. The episode roundup gives his advice to turn real-world data into evals automatically. For a support team, each reopened ticket, complaint or poor handoff becomes a new test case.
Three ways to check a resolution rate
| The vendor's figure | A replay of your conversations | Samples of live conversations | |
|---|---|---|---|
| Who picks the conversations | The vendor | You, from your own history | Your customers, as they arrive |
| Whose definition of resolved | The vendor's | Yours, in writing | Yours, in writing |
| When you can run it | When the vendor reports it | Before you sign, and after every change | Only after go-live |
| What it can miss | Cases the vendor did not pick | Behavior that depends on live systems | Anything nobody samples |
| Moves with you to a new vendor | No (No) | Yes (Yes) | Only the samples you kept (Partly) |
The vendor's figure
- Who picks the conversations
- The vendor
- Whose definition of resolved
- The vendor's
- When you can run it
- When the vendor reports it
- What it can miss
- Cases the vendor did not pick
- Moves with you to a new vendor
- No (No)
A replay of your conversations
- Who picks the conversations
- You, from your own history
- Whose definition of resolved
- Yours, in writing
- When you can run it
- Before you sign, and after every change
- What it can miss
- Behavior that depends on live systems
- Moves with you to a new vendor
- Yes (Yes)
Samples of live conversations
- Who picks the conversations
- Your customers, as they arrive
- Whose definition of resolved
- Yours, in writing
- When you can run it
- Only after go-live
- What it can miss
- Anything nobody samples
- Moves with you to a new vendor
- Only the samples you kept (Partly)
Use all three: the vendor's figure as the claim, the replay to check it, and live samples to catch what the replay misses.
Keep a copy of the setup
The agent's setup is everything that makes it yours: knowledge articles, written procedures, behavior rules, escalation rules and tone guidance. If it lives only in the vendor's admin screen, moving means rewriting it from memory. Keep the source in your own documents, treat the vendor's screen as the copy, and put exports of conversations, setup and scores into the contract.
Fin's evaluation guide also suggests a drill: time your support lead updating a return policy in the agent. More than 30 minutes, or a need for vendor help, signals high long-term running costs. The same drill works as an exit test. Time how long one procedure takes to rebuild on another platform, and you know what moving would cost before you need to move.
Before you sign
Before you sign, and every month after
BeforeCollect real conversations, hard ones included
Remove personal details. Keep escalations, complaints, reopened tickets and peak-season questions, in the proportions they arrive.
BeforeWrite down what resolved means
Name the base, the repeat-contact period and how abandoned conversations count, and use the same words in the contract.
BeforeGive every vendor the same set
Score a sample by hand to check each vendor's scoring, and keep every score you collect.
BeforeTime a policy change
Ask your support lead to update a return policy in each agent, and note who needed the vendor's help.
ContractKeep the setup and the exports
Hold the setup in your own documents, and secure exports of conversations, setup and scores.
MonthlyRe-run the set and add new failures
Compare the result with the rate on the invoice, and add each reopened ticket or complaint as a new case.
YearlyRun the set on one alternative
A second agent that also passes turns a change of vendor into a measured decision.
After go-live, someone also has to own that number. Our approach to digital workers begins in the same place: a written role, a test set from your own work, and accuracy and cost per task reported against it.
Start with one page: your definition of resolved, including the base and the repeat-contact period, agreed with the vendor before the first invoice.
Sources
- Salesforce, Salesforce Signs Definitive Agreement to Acquire Fin, 15 June 2026
- Eoghan McCabe, Salesforce signs definitive agreement to acquire Fin, Fin Ideas, 15 June 2026
- Fin, AI Resolution Rate: What It Is, 2026 Benchmarks, and How to Improve It, 17 June 2026; How to Evaluate an AI Agent for Your Ecommerce Store, 16 June 2026; Fin's Pricing Metric is Evolving, 12 March 2026
- OpenAI, Predicting LLM Safety Before Release by Simulating Deployment, 16 June 2026
- Lenny's Newsletter, How I AI, How Braintrust uses AI agents, evals, and CI to ship better software and the weekly roundup, 15 June 2026