The measurement problem
The AI Security Institute has released optstop, a tool that works with its Inspect evaluation framework. It directs testing towards estimates that remain uncertain and stops sampling when specified precision or stability criteria are met. In its reported tests, AISI saved 57–97% of planned trials. [1]
Read the conditions, not just the headline
Those savings describe the institute’s tested settings, not a guarantee for every evaluation. AISI describes post-hoc and shadow-mode use so teams can compare stopping decisions with complete runs before relying on live stopping. [1]
ADM perspective
A benchmark number is inseparable from the way it was measured. For anyone comparing AI systems, the evaluation budget, uncertainty and stopping rules deserve a place beside the headline score.
Read the original sources
The original publications provide the full detail, evidence and limitations.
- AI Security Institute · Primary research summaryOptimal stopping: spending evaluation compute where it counts
An AI-assisted, source-led summary. “ADM perspective” is editorial interpretation, separate from the source’s findings. Our editorial approach.