Chinese Language Benchmarks

Short answer. Chinese-language benchmarks exist to measure what English benchmarks miss: native fluency, Chinese professional knowledge and culturally specific reasoning. The three most cited — C-Eval, CMMLU and SuperCLUE — are also the most contaminated, and almost no Chinese vendor publishes results in a form that can be compared.

The three

Benchmark	Scope	Format	Status here
C-Eval	52 subjects, 13,948 questions	Multiple choice	No tracked results
CMMLU	67 subjects, Chinese professional knowledge	Multiple choice	No tracked results
SuperCLUE	Composite across many task types	Mixed	No tracked results

That all three columns read “no tracked results” is not an oversight. It is the state of disclosure: Chinese labs report these numbers in launch blog posts with inconsistent settings, and rarely in a form that can be recorded against a model version.

C-Eval

Released in 2023 as a Chinese counterpart to MMLU, covering humanities, social science, STEM and professional subjects. It quickly became the default Chinese knowledge benchmark and just as quickly became contaminated: the questions are public and later models trained on Chinese web data have seen them.

Its subject breakdown is more useful than the aggregate — a model can look strong overall while being weak in, say, law or medicine, which is exactly what a domain buyer needs to know.

CMMLU

67 subjects with an emphasis on Chinese professional and cultural knowledge. Same structural strengths and the same contamination problem.

For buyers, the interesting question is not the total but whether a model handles the specific domain: accounting standards, clinical terminology, Chinese legal citations. No aggregate answers that.

SuperCLUE

A composite that mixes knowledge, reasoning, safety and Chinese-language generation, published as a leaderboard. Its breadth is its value and its weakness: the weighting is not fully published, so a composite score cannot be decomposed into the capabilities a buyer cares about.

The disclosure problem

Three practices make Chinese-language evaluation unusually hard:

1. Launch-post reporting. Numbers appear in a blog post, tied to a date rather than a model version.
2. Aggregate-only disclosure. Subject-level results, where the signal is, are rarely published.
3. Inconsistent prompting. Few-shot counts and chain-of-thought settings change scores materially and are rarely stated.

Our read

The absence of comparable Chinese-language results is a real gap in the market, and an opportunity. Every serious buyer of a Chinese model needs to know how it handles Chinese contracts, Chinese medical text or Chinese customer service — and no public benchmark answers that credibly.

Contamination makes aggregate Chinese scores close to useless for model selection. A 2023 benchmark evaluated on a 2026 model measures memorisation as much as capability. The honest use is as a floor: a model that scores badly is probably bad; a model that scores well may simply have seen the questions.

Buy your own evaluation set. Fifty real documents from your domain, scored by your own criteria, will outperform any of these three benchmarks for selection — and it is the only approach that survives contamination.

Subject-level disclosure is the metric to demand from vendors. A vendor that will share per-domain results is telling you something about both its model and its transparency; one that shares only a composite is telling you something too.