Coding and Terminal Benchmarks

Short answer. Four benchmarks dominate Chinese coding claims: SWE-bench, Terminal-Bench, DeepSWE and LiveCodeBench. They measure different things — repository issue resolution, command-line work, end-to-end software engineering and competitive programming — and their scores are only comparable within the same harness version.

The four, side by side

Benchmark	Task	Unit of work	What a high score means
SWE-bench	Fix a real GitHub issue	One repository, one patch	Can localise and repair code in a large codebase
Terminal-Bench	Complete a task in a shell	Container session	Can operate a terminal, install, inspect and recover
DeepSWE	Multi-step software engineering	Repository task, multiple steps	Can plan and execute across files
LiveCodeBench	Solve a contest problem	Single function	Strong algorithmic reasoning, low contamination

The distinction is practical. A model that leads SWE-bench may fail Terminal-Bench, because the second requires environment manipulation rather than code reading. A model that leads LiveCodeBench may be a poor agent, because contest problems are single-shot.

SWE-bench

The benchmark that made “agentic coding” measurable: 2,294 real GitHub issues, with a Verified subset of 500 human-checked items. The model must produce a patch that makes the failing tests pass without breaking others.

Three things decide the number: the subset (Verified vs full), the harness (which tools the agent may use, how many attempts), and the model’s context handling on a large repository. Vendors frequently report “SWE-bench” without stating any of the three.

Terminal-Bench

Tasks are completed inside a container: install a package, fix a config, recover a broken service. Scoring is task success within a time limit.

This is the closest public proxy for “can the model operate a computer”, and it is also the most harness-sensitive: time limit, retry policy and network access change results materially. A Terminal-Bench score with no stated budget is a demo, not a measurement.

DeepSWE

Multi-step software engineering tasks that require planning across files rather than a single patch. Its value is that it separates “can write a function” from “can carry out a change”, which is the actual job.

Because the harness varies, DeepSWE numbers should be read as a relative signal between models evaluated by the same lab, not as an absolute capability level.

LiveCodeBench

Contest-style problems published after model training cut-offs, which makes it the least contaminated coding benchmark in the set. The trade-off is narrowness: a single function against hidden tests measures algorithmic reasoning, not engineering.

Its date window moves, so a LiveCodeBench score is only comparable within the same problem range.

The harness problem

Every benchmark in this group is really a benchmark of a system: model plus tools plus retry policy plus time budget. Two vendors reporting the same benchmark name may be measuring different systems, and neither may publish enough detail to tell.

Practical rule: compare only numbers published with the same harness version, and treat any cross-vendor comparison without harness disclosure as indicative at best.

Our read

SWE-bench Verified is the only coding number worth tracking across vendors — and only when the harness is stated. The full set is contaminated and the agentic variants are incomparable. A vendor that reports “SWE-bench 70%” without saying Verified, harness and attempts has produced a marketing figure.

Terminal-Bench is the better predictor of agent reliability, and almost nobody publishes it properly. Real work happens in a shell: installs fail, paths are wrong, services need restarting. Models that score well here tend to survive production; models that only score on SWE-bench often do not.

LiveCodeBench’s contamination control is its whole value. If you need to know whether a model can reason about algorithms rather than recall solutions, this is the cleanest signal available — provided the date window is stated.

Judge coding models on failure recovery, not first-attempt success. None of these benchmarks measures what buyers actually experience: how the model behaves after a test fails. That gap is why leaderboard leaders can disappoint in a real repository.