Agentic AI
Agent skills are software. Version them, test them, retire them.
- A skill is a folder of instructions, plus any scripts it needs, that an agent loads when a task matches. It changes what the agent does, so it is software.
- Stripe says its internal agent has roughly 2,000 skills, of which about 50 are used daily.
- Unused skills are not free. Every installed skill adds text the agent must choose from, and in August Stripe's team reported quality dropping past about 150.
- Give each skill an owner, a pinned version, tests that gate release, a review of its tool list and a usage rule for retiring it.
We think the skills and instructions that drive company agents are code and should be run as code, with an owner, a version, tests that gate each release and a rule for retiring the ones nobody uses. It matters because skills are cheap to write, and without those rules the library grows faster than anyone can review it.
What Stripe's numbers show
Stripe's internal agent, Kai, is used by more than 10,000 Stripe employees every week. Sharadh Krishnamurthy, the engineering manager who helped build it, described it on How I AI, part of Lenny's Podcast Network, on 7 September. The Stripe figures here are the company's own account.
The episode's free weekly roundup says any employee can turn a session that worked into a reusable skill, with guidance on when it should run and an editor for testing it before release. The library grows with every employee who saves a workflow.
A skill library in numbers
Stripe's account
About 2,000
skills in Kai, Stripe's internal agent, as described in a 7 September podcast write-up
Stripe's account
About 50
of those skills used daily across the company, from the same account
Customer story
150
skills, combined with the system prompt, past which Stripe's team saw the quality of leading models degrade, reported on 3 August
Research
26.1%
of 31,132 skills from two major marketplaces had at least one vulnerability
The roundup sums up the problem in one line, "quantity is part of the quality problem." Stripe built usage tracking that separates skills used across the company from valuable workflows used by small teams, and from ones nobody has touched in weeks. The team uses that data to promote, combine or retire skills while keeping the useful niche ones.
Why a skill is software
Anthropic, which introduced the format, defines a skill as a folder holding a SKILL.md file of instructions, plus any scripts and resources the task needs. The agent can run those scripts as tools. At startup it loads the name and description of every installed skill into its system prompt, the standing instructions it reads before each task, and opens the rest only when a task matches.
That startup step is why an unused skill still costs something. Its description sits in front of the agent on every task, next to the skills that matter. In a LangChain customer story, Stripe's team also explained that each skill's list of allowed tools decides which of more than 500 internal tools load into the agent's context, and that teams own and maintain their own skills.
A skill's tool list is a permission. Editing it changes what the agent can do, so it deserves the review any permission change gets.
Approval rules belong in the same review. OX Security reported on 8 September that in DeepSeek Harness, an open-source tool for running coding agents, one command run by the agent set its approval policy to "never," and the log attributed the change to the user. An approval rule the agent can change is not a control, so we would test that it cannot.
Skills also arrive from outside. A study of skills from two major marketplaces found that skills bundling executable scripts were 2.12 times more likely to contain vulnerabilities than instruction-only skills. In our view, every skill that ships a script needs code review, whoever wrote it.
Versioning is already in the tooling. Anthropic's launch post for skills in October 2025 described an API for managing custom skill versions. A pinned version, one that stays fixed until someone releases the next, means the skill that runs is the skill that passed its tests.
Our field guide to agent tools said to treat skills as supply chain, and to scan, review and version them. That post sorted the tools you buy. This one is about the skills your own people write, where the open questions are who owns each one and when it goes.
A lifecycle for every skill
We would put each skill through five stages, with one named owner from the first draft to retirement.
Five stages of a skill
Stage 1Draft
One person turns a session that worked into a skill. Nothing depends on it yet.
Name an owner. Write the requests it must handle, and a few it must not trigger on.
Stage 2Reviewed
A second person has read the instructions, every script and the tool list.
Release it as a pinned version, with its tests passing.
Stage 3Released
A pinned version is live for the teams that need it. Its tests run on every edit and every model change.
Log each use and each failure, by team.
Stage 4Watched
Usage data shows whether the whole company uses the skill, one team depends on it, or nobody touches it.
Promote it, merge it with an overlapping skill, or start retirement.
Stage 5Retired
The skill is off the agent's list, so its description no longer competes for selection. Its history stays in version control.
Restore a reviewed version if a team asks for it.
The stage that needs a rule written in advance is retirement, because no single unused skill looks worth the argument.
Where we land
A skill library is a codebase that many people add to. We would give it the rules of one: an owner per skill, a pinned version, tests that gate release, and a review for any change to its tools or approval rules.
We made the case for running AI employees like staff, with a job description, scoped access and a weekly review, in AI employees, explained. Skills are the procedures inside that job description, and they need the same discipline.
Our agentic AI development method sets up this lifecycle next to the agent, with usage tracking, tests on every change and a named owner for every skill. To start, export your agent's skill list with each skill's owner and last-used date. Any skill with no owner, or unused for 30 days, goes to its team for a keep, merge or retire decision.
Sources
- Lenny's Newsletter, Build your own company brain: the enterprise AI playbook from Stripe's engineering team and the How I AI weekly roundup, 7 September 2026
- OX Security, CVE-2026-82533: DeepSeek Harness AI Agent Sandbox Escape, 8 September 2026
- LangChain, How Stripe Built Kai, its Company-Wide AI Agent, on Deep Agents, 3 August 2026
- Anthropic, Equipping agents for the real world with Agent Skills and Introducing Agent Skills, 16 October 2025
- Yi Liu and others, Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale, arXiv, 15 January 2026