Frontier Reasoning Benchmarks

Short answer. Four benchmarks get quoted as evidence of “frontier reasoning”: HLE, GPQA Diamond, AIME and ARC-AGI. They test four different capabilities, they are reported under four different conventions, and only two of them have Chinese-model results in this database. Reading them as a single scoreboard is the most common mistake in Chinese model evaluation.

What each one actually tests

Benchmark	What it rewards	Format	Items
HLE	Breadth of expert knowledge, resistance to search	Mixed MCQ and short answer	2,500
GPQA Diamond	Graduate-level science reasoning	4-option MCQ, 198 questions	198
AIME	Competition mathematics, end-to-end derivation	15 integer-answer problems	15 per year
ARC-AGI	Abstraction from a few examples	Grid-transformation puzzles	400 public, private eval

The distinction matters commercially. A model that leads HLE is broad; one that leads GPQA is scientifically precise; one that leads AIME is a strong mathematical reasoner; one that leads ARC-AGI generalises from tiny samples. None of these is “general intelligence”, and a buyer selecting for a contract-review pipeline should weight none of them heavily.

HLE

Humanity’s Last Exam was built as a frontier-resistant knowledge test: expert-written questions across disciplines, designed so that search alone does not answer them. Vendors commonly report two numbers — with and without tool access — and the gap between them is often larger than the gap between models.

The practical reading: HLE is a breadth test, and breadth is cheap to improve with retrieval. A high no-tools score is the more interesting number; a high with-tools score mostly tells you the model is good at using a browser.

GPQA Diamond

198 graduate-level science questions written to be Google-proof, drawn from physics, chemistry and biology. It is the most stable of the four because the item count is large enough for the score to be meaningful and the format is uniform.

Its limit is domain narrowness: strong GPQA performance says a model handles technical reasoning carefully, not that it handles your domain. Chinese labs report it consistently, which makes it one of the few genuinely comparable numbers across vendors — provided the tool-access setting is stated.

AIME

Fifteen problems, integer answers, no partial credit. The exam is public and annual, so the year is the version: “AIME 90%” without a year is uninterpretable, and the shift toward AIME 2025 reflects contamination pressure as much as difficulty.

AIME’s real value is diagnostic: it isolates multi-step mathematical derivation from knowledge recall. Its real weakness is sampling noise — one problem is 6.7 percentage points.

ARC-AGI

ARC-AGI asks a model to infer a transformation rule from two or three examples and apply it to a new grid. It is the least saturated benchmark in this group and the hardest to game with scale alone, which is why it attracts attention disproportionate to its commercial relevance.

For buyers it is close to useless as a selection criterion — almost no production workload resembles ARC puzzles. Its value is as a canary: a lab that improves on ARC-AGI is probably improving genuine abstraction rather than memorisation.

Comparability

Three rules apply across all four:

1. Tool access must be stated. HLE and GPQA both have with-tools variants; mixing them creates fake improvements.
2. Item counts bound precision. 15 items cannot support a claim of a 2-point difference; 198 can support roughly a 7-point one.
3. Year and version are part of the name. AIME by year, HLE by tool setting, GPQA by split (Diamond vs main), ARC-AGI by public or private set.

Our read

HLE is the most over-weighted benchmark in the Chinese model market. It measures breadth of expert knowledge, which retrieval systems solve, and it is reported in two modes that vendors choose between when publishing. A buyer who selects on HLE is selecting on a number that partly measures browsing skill.

GPQA Diamond is the most useful of the four for technical buyers. It is large enough to be stable, narrow enough to be honest, and consistently reported. If you must pick one reasoning benchmark to track across Chinese vendors, pick this one.

AIME and ARC-AGI are diagnostic, not commercial. They tell you something real about a lab’s capability trajectory and almost nothing about whether a model will work in your product. Track them for signal about the vendor, not for selection.

The panel beats any single number. A model strong on GPQA and weak on AIME is a careful reasoner without mathematical depth; the reverse suggests a maths specialist. Reading two together tells you more than reading either twice.