Back to blog Perspectives

Accuracy Per Step Doesn't Add Up: Why You Have to Measure Whole Cases

Phinite teamNov 5, 20263 min read

Ask a team how their triage agent is doing and you will hear step accuracies. It gets the category right 97% of the time, the priority 95%, the routing 99%. Those numbers are true. They are also almost useless for the question the process owner is asking, which is how many tickets were handled correctly.

The arithmetic nobody does

The share of tickets where all three steps are right is not 97%. It lands somewhere between 91% and 95%, and where it lands depends on whether the errors hit the same tickets or different ones. If the agent tends to miss category and priority on the same awkward cases, the combined rate is closer to 95%. If the misses are spread out, it is closer to 91%.

You cannot know which from step accuracies. You only find out by measuring whole cases, from intake to confirmed action.

What counts as a handled case

So we use a stricter test. A case is verified autonomous only if all five of these are true:

  1. It was in scope for the workflow.
  2. The required business action happened and was confirmed in the target system. The agent reporting "success" is not enough.
  3. Every critical check passed on the final result.
  4. No person had to step in.
  5. No one corrected it afterward, within an agreed window.

Each case counts once, however many agent runs it took. A ticket that bounced between two agents and a retry is still one ticket.

The ratio is then simple: verified autonomous cases divided by in-scope cases.

Correction belongs in the measurement

Here is where whole-case measurement changes what you build. Suppose 1,000 in-scope tickets arrive. The triage agent gets 750 fully right on the first pass. A correction agent reviews the other 250, fixes 200 and sends 50 to a person.

  • First-pass rate: 750 ÷ 1,000 = 75%. That is what one agent did.
  • Verified autonomous rate: (750 + 200) ÷ 1,000 = 95%. That is what the business got.

Both numbers are true. Only the second describes the result the process owner received, and only whole-case measurement produces it. The 50 tickets that needed a person are not a failure statistic; they are the map of where to aim the next fix.

(The figures are illustrative, not customer results.)

What this does not tell you

Verified autonomous says the case was handled correctly without a person. It does not say the customer's problem was solved, or that the customer was happy. Those are separate metrics with their own scope, and they should be reported separately rather than folded into the autonomy number to make it look like more than it is.

How to put it into practice

Whole-case measurement needs three things the agent platform has to provide. First, per-workflow checks that no log can infer: was this case in scope, did the required action actually happen, did every critical check pass. Second, a record of later corrections, which means watching the target system for a window after the agent finished. Third, a trace that ties every run, check and human step back to one case and one version.

With those in place, the headline number is one the business recognizes, the guardrails sit beside it, and the failed cases are a list you can open rather than a percentage you have to explain.

Written byPhinite teamWrites on building, running and governing AI agents in production.

Prove what your agents changed in the business.