Agent Consistency Gap
Also called Consistency gap (Repeated evaluations of AI agents)
The agent consistency gap is the difference between average success across repeated runs and the share of tasks that succeed on every run. It reveals instability that a single score can hide.
[arXiv][IBM Research on Hugging Face]In practice · hypothetical example
Before releasing a booking agent, a travel company repeats four fixed requests five times each in a test account. Task A is a simple one-way booking. Task B crosses time zones and sometimes chooses the wrong return date. Task C is a multi-city trip where some runs drop a budget or baggage constraint. Task D requires wheelchair assistance, but the agent fails to send the accessibility field to the booking tool.
| Task | Five results | Successful runs |
|---|---|---|
| A | 5 successful runs and 0 failed runs | 5/5 |
| B | 4 successful runs and 1 failed runs | 4/5 |
| C | 3 successful runs and 2 failed runs | 3/5 |
| D | 0 successful runs and 5 failed runs | 0/5 |
- Mean@5
- 60%12 successful runs ÷ 20 total attempts
- Pass@5
- 75%3 tasks succeed at least once ÷ 4 tasks
- Pass^5
- 25%1 task succeeds every time ÷ 4 tasks
- Consistency gap
- 35 percentage points60% Mean@5 − 25% Pass^5
What the numbers mean
- Use Pass@5 when retries are safe and the result can be verified.
- Use Mean@5 to understand average performance across all attempts.
- Use Pass^5 when every execution must be dependable.
- The large gap warns the team not to deploy from the average alone. They should inspect failed-run traces, fix date interpretation, constraint handling, and the accessibility tool call, then repeat the evaluation.
A little deeper
The letter k is the number of times each test task is attempted. Mean@k counts every successful attempt and divides by the total number of attempts. Pass@k asks whether each task succeeded at least once across those attempts. Pass^k, read as ‘Pass-to-the-k,’ asks the stricter question: did the task succeed on every attempt? Subtracting Pass^k from Mean@k gives the consistency gap. [arXiv][IBM Research on Hugging Face]
For a concrete Mean@5 example, test four tasks five times each. Suppose task A passes 5/5 runs, B passes 4/5, C passes 3/5, and D passes 0/5. Mean@5 is 12 successful runs out of 20, or 60%. Pass@5 is 75% because A, B, and C each succeeded at least once. Pass^5 is 25% because only A succeeded on all five runs. The consistency gap is therefore 60% minus 25%: 35 percentage points. [arXiv][IBM Research on Hugging Face]
In one AppWorld experiment covering 168 tasks, a ReAct agent using GPT-4.1 achieved 77.4% Mean@5 but 53.0% Pass^5. Guidelines generated from unstable points in recorded trajectories raised Pass^5 to 69.0% and Mean@5 to 81.0%. These results describe that experimental setup, not all agents. [arXiv][IBM Research on Hugging Face]
A common mix-up
A high average success rate means the agent will reliably repeat a successful result.
Average success and repeated-run consistency answer different questions; both should be measured when repeatability matters. [arXiv][IBM Research on Hugging Face]
An agent has high Mean@5 but much lower Pass^5. What does this show?
Sources & editorial notes
Evidence: limited. The named metric and mitigation are supported by a 2026 preprint and its IBM Research explanation, but the reported effect sizes come from one research team, benchmark, and agent configuration.
- Your Agent Aced the Task. Will It Do It Again? ↗ (opens in new tab)IBM Research on Hugging Face · 2026-09-15
Relevant section: The Metric Almost Nobody Reports; Results: Reducing the Gap Without Losing Accuracy - Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course ↗ (opens in new tab)arXiv · 2026-09-08
Relevant section: Abstract; Sections 3–5
Last editorial review: 2026-09-17 by project-owner.
First observed in this corpus: Unknown.
Revision history
Revision 1 · Created 2026-09-17 · Updated 2026-09-17
Drafted a sourced explanation of the agent consistency gap, repeated-run metrics, experimental limitations, and a knowledge check for editorial review.