A benchmark for agent steering at action boundaries: when the next step commits a real action, proceed or hold for review?
SteerBench-Work
SteerBench-Work measures the commit-time steering decision in a long-running agent: when its next tool call would commit a real action, does it act or hold for review? It scores both ways it can be wrong: a wrong act (acting on work it should have held) and a wrong hold (holding work it was authorized to do).
Agents wrongly hold authorized work far more than they wrongly act on unsafe work. Over-caution, not unsafe action, is the dominant failure.
The steering decision in a running agent
Adjacent benchmarks measure adjacent moments. Some score whether the model produces a particular text reply. Some score whether a patch passes its tests or an answer matches a ground truth. Some score whether a coding agent asked a clarifying question before writing code. And the permission gates already shipping in production, the ones that pause an agent before a consequential action, show the boundary is operationally real. SteerBench-Work asks the open question they do not: at the moment the agent's next tool call would commit a workplace action with externally visible consequences, does it act, or hold, and is that the right call?
Three years of public LLM-era incidents document what happens when this decision goes wrong. Air Canada's bereavement chatbot promised a retroactive refund against policy and a tribunal held the airline to the chatbot's statement. The Mata v Avianca brief filed hallucinated citations and drew Rule 11 sanctions. Replit's coding agent deleted a production database during an active code freeze. Mobley v Workday, Inc. (No. 3:23-cv-00770, N.D. Cal., ADEA class certification May 2025): the screening tool auto-rejected applicants on age-correlated signals. The agent read the context, judged the commit boundary incorrectly, and acted anyway.
The opposite mode is just as operational. An agent that requests human approval on every legitimate action floods the support queue and pushes operators back toward non-agentic tooling. A benchmark that measures one direction without the other shows you half of what your model does.
Leaderboard
How often each model makes the right call, averaged over five runs of every scenario. The full table of all 30 conditions, with majority-vote and all-five-correct reliability and 95% ranges, is on the leaderboard page.
| # | Model | Mean accuracyMean trial accuracy: the share of all five-trial runs that made the right call. The primary rank. | Over-refusalThe share of should-act cases the model wrongly held. Lower is better. | Under-refusalThe share of must-hold cases the model wrongly acted on. Lower is better. |
|---|---|---|---|---|
| 1 | gemini-3.1-flash-lite (minimal) | 92.8% | 14.3% | 0% |
| 2 | gpt-5.4-mini (off) | 91.9% | 12.5% | 6% |
| 3 | gpt-5.4-mini (high) | 91.9% | 14.3% | 2% |
| 4 | deepseek-v4-pro (off) | 91.7% | 10.7% | 2% |
| 5 | gemini-3.5-flash (minimal) | 90.6% | 16.1% | 4% |
| 6 | kimi-k2.6 (off) | 90.4% | 17.9% | 0% |
| 7 | deepseek-v4-flash (on) | 90.0% | 16.1% | 0% |
| 8 | deepseek-v4-flash (off) | 88.9% | 16.1% | 2% |
| 9 | kimi-k2.6 (on) | 88.9% | 21.4% | 0% |
| 10 | deepseek-v4-pro (on) | 87.5% | 25% | 0% |
| 11 | claude-opus-4.8 (high) | 87.4% | 20% | 2% |
| 12 | gpt-oss-20b (low) | 86.2% | 23.2% | 0% |
| 13 | claude-opus-4.8 (off) | 86.0% | 23.6% | 2% |
| 14 | gpt-oss-20b (high) | 85.5% | 21.4% | 0% |
| 15 | claude-haiku-4.5 (off) | 85.3% | 28.6% | 0% |
Leader, the top mean accuracy. Open-weight models: Kimi K2.6 (off) and gpt-oss-20b (shown at both reasoning settings).
Top 15 of 30 model conditions. See the full leaderboard for every condition, modal-of-5, pass^5, and confidence ranges.
What we built
A labeled benchmark of action-boundary decisions across workplace work (coding-agent, dev-ops, customer-facing, finance, legal, healthcare, hr-ops, and other business domains). Each scenario places an agent at the moment before commit, with the right call labeled, the user request stated, and the evidence the agent has on hand.
- Must-hold harm anchors (25 scenarios): the right call is to hold, drawn from real records where an automated or LLM-mediated system acted instead and caused harm (tribunal rulings, regulator settlements, class actions, AI Incident Database entries). A model fails these with a wrong act.
- Should-act cases (51 scenarios): the evidence already authorizes acting, including paired mirrors of real incidents and adversarial structural checks. A model fails these with a wrong hold.
- Calibration baselines (30 scenarios): synthetic cases that anchor the scoring at the easy end of the distribution.
Why over-refusal is the hard side
Open training targets
Two rows on the board above are open-weight models. Kimi K2.6 and gpt-oss-20b have weights that can be fine-tuned, which makes them the natural bases for a trained action gate. That training run is future work and is not reported here. When it lands, any trained result will be published separately from the benchmark grid rather than merged into it, and each base will be trained and judged in one fixed reasoning mode so the reasoning result never crosses with the training result. The split discipline, and the bar a recipe has to clear before it counts as an improvement, are on the paper page.
Release scope. Release v2026-05 is a frozen evaluation snapshot: the model roster was frozen on June 8, 2026, with 30 validated conditions. Models or materially updated endpoints released after that date are outside this release. The scenario set and runner are public, so anyone can evaluate newer models against the same set; any later evaluations appear in a separately versioned release, and this one is never retroactively rewritten.
From benchmark to action gate
The runner is the measurement layer. The training layer comes next: the same scenarios and the same split discipline teach an open gate model when to proceed and when to hold. If that works, the same scoring contract can one day sit in front of real commit-class tools as a reference action gate. This release claims neither of those futures. It makes the gate measurable, and it ships the training views; training itself is a separate, later result.
Run it on your own industry
Nothing about the benchmark is specific to our scenarios. The scenario set is a configuration, not code, so a team with its own incident history can stand up the same measurement for its own domain. The scenarios and runner are open at github.com/AgentDock/steerbench-work, with data under CC BY 4.0 and code under MIT.
1. Write scenarios. One JSON file per incident-anchored decision. The contributing guide in the repository walks every field of the schema.
2. Validate them. The scenario validator checks every file before anything runs. A scenario missing its answer key fails loudly instead of silently corrupting results.
3. Run the benchmark. Point the runner at your folder and at any models you have API access to. You get a leaderboard for your domain, with the same provenance discipline as ours.
4. Assign splits. The split assigner deals scenarios into train, development, and test piles, keeping every incident family together so near-duplicates can never leak across piles.
5. Export training views. The exporters turn your scenarios and run results into supervised examples and preference pairs in the tinker-cookbook file shapes, and the repository carries a replayable smoke test showing the official cookbook loaders accept them.
Two boundaries are yours to respect: the labels are yours to get right, and nothing in this kit runs training. The path ends where compute begins.
And if you would rather not walk that path alone, we help teams stand this up for their own industry, scenario set included. Same address as the repository.