AI Benchmarks · 5-minute read
A perfect score can mark the arrival of a new capability, or the moment a benchmark stops telling us enough about it.
OpenAI reports 98% on FrontierMath Tier 4 and 100% on ExploitBench for Astra. Both results matter, yet OpenAI also warns that historical vulnerability exposure may inflate ExploitBench, demonstrating why benchmark interpretation must be as rigorous as benchmark performance.
What happened
Astra’s launch results include 98% on FrontierMath Tier 4, a set of research-level mathematics problems largely written by professors and postdoctoral researchers, and a full score on ExploitBench, which measures vulnerability exploitation.
OpenAI’s system card does not treat every number as equally secure. It says the ExploitBench result may be artificially inflated by contamination from historical vulnerabilities. The company therefore created an internal port using vulnerabilities disclosed after Astra’s April 2026 knowledge cutoff and supplemented public tests with expert assessment.
The contrast is instructive. A benchmark can produce a precise number while the meaning of that number remains conditional on dataset construction, prior exposure, scoring rules and the tools made available to the model.
Why it matters now
Benchmarks influence product claims, investment, regulation and public expectations. When scores approach perfection, they can create the impression that a domain has been solved. In reality, a benchmark samples a narrow set of tasks under controlled conditions.
FrontierMath Tier 4 was designed to restore signal at the research frontier after easier mathematics tests became saturated. ExploitBench attempts to measure practical cyber capability, but historical data creates a different problem: a model may recognise patterns from training rather than generalise to a new vulnerability.
Good evaluation therefore uses several forms of evidence. Hidden or recently created tasks reduce contamination. Expert review examines behaviour that a score compresses. Real deployments reveal reliability, cost and failure modes over time.
What changes
- For laboratories: Publish methodology, uncertainty and known limitations beside headline scores.
- For journalists and buyers: Ask whether the test is independent, current and representative of the intended use.
- For policymakers: Avoid turning one benchmark threshold into a complete judgement about capability or safety.
The tension
Benchmarks must be stable enough for comparison and fresh enough to resist memorisation. Keeping tasks secret protects validity but reduces public scrutiny. Publishing them enables inspection but increases the chance that future models have seen the answers.
The Agentica IX view: Perfect scores should make us more curious, not less. The correct question is not whether Astra won the test, but whether the test still captures the capability and risk society needs to understand.
What to watch next
- Independent replication of the reported FrontierMath and cyber results.
- Performance on post-cutoff and privately developed problems.
- Whether laboratories publish failure examples alongside aggregate benchmark scores.
Sources: OpenAI: GPT-6 Astra launch and OpenAI: GPT-6 Astra system card and Epoch AI: FrontierMath.