There is an agent in your account that was deployed in March. It answers. It bills tokens every day. The person who specified it changed teams in May, the knowledge base it reads was last synced in June, and nobody has read one of its conversations since launch week.
The interesting question is not whether it is broken. It is that you have no procedure that would tell you either way — and the same sentence is true of most agents any organisation is currently running.
Three questions nobody can answer about their own fleet
On 26 August 2026, in the McKinsey Talks Talent episode Your AI agents need performance management, too, senior partner Kate Smaje — McKinsey's global leader of technology and AI — reduces the problem to three questions an operator should be able to answer: "How many agents do I actually have? Who is responsible or accountable for their performance? What value are they creating?"
In the same conversation she names the failure mode directly. Agents "need strong life cycle management because they can't exist in perpetuity. A wonderfully fit-for-purpose agent that existed six months ago could possibly be a significant liability today." Her term for what accumulates is abandonware: "the things in your networks that have been forgotten about and should no longer exist."
Senior partner Brooke Weddle adds the assumption underneath it: "There's a bias today that any agent is a good thing, but that's simply not true."
The framing is borrowed from HR, and that is the useful part rather than a flourish. Organisations know how to hire for a capability, review against it, correct, and end the arrangement when it stops fitting. Almost none apply any of that to software that decides on their behalf — because software has not historically needed it, and an agent does: it degrades while sitting perfectly still.
An inventory answers none of the three
The first instinct is a list of agents. It answers question one approximately and the other two not at all, because the unit of observation is not the agent. It is the run — one execution, from trigger to final output — and the record it leaves. "Value" is a ratio, and you cannot compute a ratio from a list.
| Recorded per run | Without it you cannot answer |
|---|---|
| Agent identity and version | Anything, because every metric silently averages two different agents |
| Terminal status, and the failure class when it failed | Whether it works |
| Tool calls attempted against tool calls succeeded | Whether it works for the reason you think |
| Tokens in and out, attributed to the step that spent them | What it costs to produce one outcome |
| The end user's verdict, where one was given | Whether the outcome was any good |
| The tenant and the human owner it belongs to | Who to tell |
The version field is the one most often left out and the one that breaks everything downstream. A run that does not name the definition it executed cannot be compared with the run before it: edit a prompt, and last month's satisfaction rate becomes a blend of two populations. It is the discipline any team applies to a release tag, skipped for agents because the edit feels like writing rather than deploying.
Two of those rows also give you the denominator for real unit economics: cost per outcome rather than cost per token is the only form in which question three survives contact with a finance team.
Accountability is a column, not a policy
Smaje is explicit that this should not land on the CTO: "I would love to see the business owners who use these agents for their workflows take on that responsibility. I don't think this is a technology problem." The head of finance who looks after the humans doing month-end close should look after the agents doing it.
The reason it usually does not work is more mechanical than the org-design conversation around it. If "who owns this agent" is answered by someone's memory of who asked for it, the answer is nobody — formally nobody, the first time that person changes teams. Federated ownership needs three unremarkable things before it can be an operating model:
- An owner field that cannot be null at creation. Not a description, not a tag — a resolvable identity that alerts route to.
- A risk posture the owner did not write. Smaje's condition for federating is "guardrails, standards, and clear life cycle management across your organization." The finance lead chooses what their agent does, not whether it logs, whether its outputs are checked, or how long its data is kept.
- A review that arrives without being requested. Ownership that activates only when something breaks is indistinguishable from no ownership, because the failure mode below breaks nothing.
The split is not new — it is how an operator runs boundaries across many organisations, one level down.
Agents decay while sitting perfectly still
A service nobody touches keeps doing exactly what it did. An agent whose definition nobody touches does not, because four things move underneath it:
- The model moves. The provider deprecates the version you pinned, or silently improves the one you did not, and a prompt tuned against the old behaviour is now tuned against nothing.
- The corpus drifts. Retrieval returns the best match in the index, always, with the same confidence whether the index was synced this morning or last quarter. Staleness has no error state.
- The tools change contracts. A field becomes optional, an enum gains a value, an endpoint starts paginating. The agent adapts — it invents a plausible reading of the new shape — which is worse than crashing.
- The process changes and the agent does not. The discount policy changed in April. The agent is still executing March's, fluently, with citations.
Only the third tends to raise an alert. The others produce output that looks exactly like correct output, which is why they are found by sampling conversations rather than by monitoring uptime. The dangerous state for an agent is not "down". It is "confidently obsolete" — and the only instrument that detects it is a human reading a handful of transcripts on a schedule.
Retirement is a state, not a delete
The standards work here predates the agent conversation and lifts wholesale. The NIST AI Risk Management Framework — NIST AI 100-1, released 26 January 2023 and currently under revision — makes decommissioning a first-class control. Subcategory MANAGE 2.4 reads: "Mechanisms are in place and applied, responsibilities are assigned and understood to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use." MANAGE 4.1 lists it alongside incident response and change management as a component of post-deployment monitoring, not a cleanup task.
The accompanying playbook is more specific, and the verb matters: "Decommission and preserve system components that cannot be updated to meet criteria for redeployment." Preserve, not delete — the conversations an agent produced are referenced by audits, disputes and other agents' memory, and deleting the definition behind them turns a year of history into unexplainable output.
So the minimum lifecycle is four states and one date:
| State | What it means | Who moves it |
|---|---|---|
| Draft | Exists, not reachable by any channel | Anyone with edit rights |
| Active | Serving traffic, carries a review date | Owner, on a passing check |
| Deprecated | Still answering, flagged for replacement, no new bindings | Owner |
| Retired | Answers nothing; definition, versions and runs preserved | Owner, recorded |
The date does the work. An active state with no review date is no lifecycle at all — it is the condition that produces the agent from March. Set it at deployment, show it on the agent, and let it expire loudly.
What a machine may decide on its own
Three signals are safely mechanical. Consecutive failures — a scheduled agent that errors N times in a row should pause itself and notify its owner, the counter resetting on any success so an intermittent provider does not trip it. Idleness — zero runs in a window is a strong signal and a weak verdict: it flags for review, never more. Cost or volume anomalies against the agent's own trailing baseline, a better comparator than any fleet-wide threshold.
What must not be automated is the state transition: retirement, scope changes and any relaxation of a guardrail carry a person's name. The asymmetry is the rule worth copying — automation may reduce what an agent is allowed to do, never expand it. A system that pauses an agent on its own is a safety mechanism. One that resumes it on its own has removed the human from the only decision they were there for.
When the measurement becomes the problem
This discipline has a cost, and it is possible to overshoot it in two directions.
Measuring the wrong thing is worse than measuring nothing, and there is a specific wrong thing everyone reaches for first. Smaje again: "I get nervous when I see scorecards with adoption metrics such as 'What's the adoption rate of tool A versus tool B versus tool C, market by market?' That's a highway to nowhere. It doesn't tell you how the technology is being used, let alone how it's creating value." Run counts and token totals are the same metric in an operations costume: they tell you an agent is busy, and a busy agent answering the wrong question costs more than an idle one.
Every human check is a queue. Sampled transcript review, approval gates, review dates — each is work landing on a real calendar, and Smaje notes the second-order effect: the people working most intensively with these systems are "exhausted — not because AI is failing them, but because it's working," left with only the judgement-heavy residue. A cadence nobody sustains becomes no review at all, which is where you started, now with a policy document. Set the sample at what the owner will actually read.
And the case for not building any of this: if one person built and operates three agents, they are the lifecycle. The machinery starts paying when ownership crosses a person boundary — when whoever answers for an agent is not whoever wrote it. Below that, the register is ceremony. Above it, the alternative is the March agent.
The monthly fleet review
One hour, one agenda, every item answerable from records you keep or should start keeping. No item is easier to pass on one platform than another — that is deliberate.
- How many agents are active, versus last month? A number that only goes up is not a fleet, it is an accumulation.
- Which agents ran zero times in 30 days? Each gets an owner decision on the spot: retire, or write down why it stays.
- Which have no owner, or an owner who has left? These are the abandonware. Assign or retire before the meeting ends.
- Which are past their review date? If more than a quarter, the cadence is wrong, not the fleet.
- Read three real conversations per material agent. Not summaries, not the thumbs-down ones. Randomly sampled.
- What changed underneath them — model versions, tool contracts, corpus syncs — and which agents sit downstream of each.
- Cost per completed outcome, by agent. Ratio, not total — a rising total with a falling ratio is success.
- What was retired? A review that never retires anything is not a review. "Nothing" twice running means item 2 is not being answered straight.
Smaje's three questions are diagnostic more than operational: an organisation that cannot answer them does not have a measurement gap. It has been hiring without a performance review, indefinitely, and calling it velocity.
