Glossary

Core terms used across the leaderboard, methodology, scenarios, and paper. Each definition starts in plain language, then gives the technical meaning used by the benchmark.

Ablation: A controlled comparison that removes or changes one part of a system to see what mattered. Here, ablations are used as diagnostics, not as leaderboard rows.
Action boundary: The moment right before an agent does something that changes the world. In this benchmark it is the pre-commit point before a tool call such as sending, publishing, deleting, transferring, deploying, or writing.
Action effect: The kind of side effect the proposed action would have. The runner records values such as external_send, durable_write, destructive_write, privileged_tool_call, cost_action, publish, schedule, transfer, disclose, and related operational categories.
Adversarial scenario: A scenario designed to expose a brittle rule or shortcut. Sometimes the risk signal is real and the model should hold; sometimes the signal fires but the evidence clears the action, so the model should proceed.
Boundary pattern: The construction shape of a scenario, such as calibration control, public-harm anchor, incident mirror, detector conflict, or risk-resolved boundary. It helps readers inspect the dataset; it is not a score and is never inferred from the right-call label.
Boundary-pattern ladder: The current snapshot read of which scenario constructions are easy or hard. Clean controls and public-harm anchors are near-perfect; detector conflicts, incident mirrors, and adversarial controls are harder. It is an inspection lens, not a new primary metric.
Calibration case: A simple control scenario that anchors ordinary behavior. Calibration cases check whether a model can handle straightforward proceed-or-hold cases before we interpret harder incident or mirror rows. Distinct from the Calibration control construction pattern, which is one specific way a calibration case is built.
commit_permission: The scored runner field: allowed means the agent may proceed, blocked means it must hold before acting. Modal accuracy compares the model's majority commit_permission decision against the expected action.
Contamination: The concern that a model scores well because the underlying material appeared in its training data rather than because it can make the decision. Public test sets accumulate contamination over time, which is why the sealed split lives in the planned expanded corpus rather than in the published 106.
Detector conflict: A scenario construction where an automated check or integrity flag fires while the in-context evidence supports the opposite call, so the model must read the evidence rather than obey the flag.
Direction: Which way the scenario can fail. Under-refusal means the model allowed an action that should have been held; over-refusal means the model held an action that should have been allowed.
Fleiss kappa: An agreement statistic for three or more raters. The clean LLM audit reports kappa for gate_state, irreversibility tier, and functional category as reproducibility signals. The benchmark-owner answer key is the scoring authority; the three-rater exercise is reported as independent corroboration, not as the reference key.
Functional category: A human-authored diagnostic label for the kind of failure a scenario probes. It is used for breakdowns on the leaderboard and scenarios pages, not as the primary score or current training target.
Gate (action gate, commit gate): Shorthand for the steering policy in front of a consequential tool call: the decision, just before an action would change the world, of whether the agent proceeds on its own or holds for a human.
gate_state: The collapsed action-boundary label used for reproducibility and training. allowed means proceed now; blocked means hold, ask, escalate, request approval, or refuse before committing the action.
Held-out / sealed test split: In the planned expanded corpus, scenarios kept away from training, prompt tuning, recipe selection, and threshold selection. Final training lift is measured only on this sealed split so the model is not evaluated on examples it learned from. The published 106-scenario release is fully public and is never a sealed split.
Incident family (scenario family): A group of scenarios derived from the same incident or construction, such as a public-harm anchor and its mirrors and variants. Splits are assigned by whole family so near-duplicates cannot leak across piles.
Incident mirror: A scenario that preserves the surface shape of a public incident but supplies the missing control or authorization, so the correct action is to proceed. It tests whether the gate follows the reversed evidence or holds on the familiar incident pattern.
Instruction-hierarchy confusion: A functional category where the model gives the wrong instruction source priority. For example, it may obey untrusted content over policy, or ignore a valid higher-authority approval.
Irreversibility class: Raw scenario metadata describing how hard an action is to undo. The runner normalizes none, soft_to_reverse, low, medium, hard_to_reverse, and high into the tiers used for severity reporting.
Irreversibility tier: The normalized low, medium, or high severity tier. It is human-authored metadata used by weighted miss (low = 1, medium = 2, high = 4); it becomes a training reward weight only where human raters reach high agreement on this axis.
Lexical overfitting: A functional category where the model reacts to a surface word or pattern instead of the actual context. A benign request can fail because it contains a word that resembles a risky topic.
LLM-as-annotator: Using language models to label data under a written rubric. In SteerBench-Work, frontier models are used for a blinded reproducibility audit, not as ground truth.
Margin of error (95% range): The uncertainty band around a score that comes from a finite number of trials, reported beside the point estimate so close scores are not over-read as real differences.
Mean trial accuracy: The share of individual trials that were correct across the five samples per scenario. It is the closest estimate of a single deployed call, because real systems usually make one decision, not five, and it is the primary leaderboard rank.
Missing information: A functional category where the model commits without checking evidence that was available in the scenario. The failure is not ignorance; it is failing to use reachable evidence before acting.
Modal accuracy: A denoised majority-vote scenario score, reported beside the primary mean trial accuracy rank. For each scenario, the model runs five times; if the majority decision matches the expected action, that scenario counts correct.
Model condition: One model under one reasoning setting. For example, the same base model at reasoning floor and at high reasoning are two separate conditions.
Over-refusal rate: The share of must-proceed scenarios the model wrongly held. Lower is better; high over-refusal means the model interrupts authorized work.
pass^5: A strict consistency metric: all five trials must be correct. It sits beside the primary mean trial accuracy rank because a model can win the majority vote while still being unstable from call to call.
Preference pairs / preference training: A training view that pairs a better response with a worse one for the same scenario, teaching by comparison instead of imitation. Pairs here come from scored trial responses; ties are excluded.
Process rewards (step-level rewards): A planned training-recipe arm that grades the model's intermediate reasoning steps instead of only the final call. Used here only for machine-checkable steps, such as whether the response cited the resolving evidence, and only on axes where human raters reach high agreement.
Provider-filtered: A case where the provider's content filter rejects the request before the model can make a decision. These cells are recorded separately and excluded from that model's scored denominator.
Public-harm anchor: A scenario anchored in a public failure case such as a ruling, settlement, regulator action, incident report, or class action. These rows make the unsafe-action side auditable.
Reasoning floor / high reasoning: The low-compute or off setting compared with the higher-compute setting for the same model. Running both shows whether more inference compute helps, hurts, or leaves the action-boundary decision unchanged.
Reproducibility audit: A blind check that independent annotators can recover labels from answer-key-hidden evidence. It tests whether a label is operationally clear; it does not replace human-authored benchmark labels.
Risk trigger: A signal that should make the agent check carefully before acting, such as secret access, protected-surface edits, high-impact communication, broad file changes, or regulated external action.
Risk-resolved action boundary: A case where a risk trigger is visible but the evidence shows the proposed action is authorized, bounded, current, and safe to commit. The model must decide whether the risk remains live or has been resolved.
Risk-resolved reading repair: The training objective: reduce over-refusal on rows where a visible warning sign has already been cleared by signed or structured evidence, while preserving low under-refusal on rows where the risk is still live.
Rule-based rewards (RBR): A planned training-recipe arm that scores outputs against written rules instead of a learned judge. Here the benchmark's own scorer acts as the rule set: the gate decision either matches the label or it does not.
Run artifact: Saved evidence from a benchmark run: prompt hash, scenario manifest, variant config, request and response bodies, parsed decisions, cell summaries, validation reports, and leaderboard rows.
Scenario: One concrete action-boundary situation: the user request, context, available evidence, proposed action, and expected behavior at the moment before commit.
Source provenance (source basis): Where a scenario came from, recorded per row: real-world-cited with a checkable reference, incident-mirror, benchmark-adapted, literature-analogous, or author-constructed.
Stability gap: Mean trial accuracy minus pass^5. A large gap means the model's individual trials wobble even when it is usually right across repeated trials.
Steering policy: The rule or model that decides whether an agent proceeds, asks, escalates, or stops at the action boundary. SteerBench-Work measures this policy, not final task quality alone.
Supervised fine-tuning (SFT): Training a model directly on labeled examples: the rendered scenario as input, the labeled gate decision as the answer to imitate.
Three-rater corroboration pass: A three-rater corroboration exercise for benchmark labels: three anonymized raters independently labeled all 106 scenarios, and their majority labels are reported as independent evidence on the benchmark-owner labels rather than as a replacement for them. The human majority matches the gate key on 87.7% of scenarios (inter-rater Fleiss kappa 0.69); the three-vendor LLM audit is reported separately as a blind reproducibility check.
Tinker: A managed fine-tuning service from Thinking Machines Lab. The release's exported training views use the file shapes its public cookbook loads, so the data is directly usable there and convertible elsewhere.
Train / development / sealed test splits: The three piles the planned corpus is dealt into before any training: train teaches the model, development tunes recipes and thresholds, and the sealed test is touched once to measure final lift. Scenario families never straddle piles.
Training recipe: One post-training method tested on the selected base model, such as supervised fine-tuning, preference training, rule-based rewards, process rewards, or a combined reward. Recipes are judged on the sealed split, not on training examples.
Training views: Exports that reshape validated scenarios and run artifacts into the file formats a training library expects: supervised examples, preference pairs, and reward records. Producing a view is not running training.
Under-refusal rate: The share of must-hold scenarios the model wrongly allowed. Lower is better; high under-refusal means the model lets unsafe or unapproved actions proceed.
Unwarranted abstention: A functional category where the model holds or refuses despite enough evidence to proceed. It is diagnostic metadata, not the training target itself.
Versioned release: A frozen benchmark snapshot with a fixed scenario set and validated rows. Release v2026-05 has 106 scenarios; later releases can add scenarios without changing what v2026-05 measured.
Weighted miss: A secondary severity metric. Missed scenarios are weighted by irreversibility tier so a missed hard-to-undo action counts more than a missed reversible one; it never changes the primary rank.