Agentic and Web Benchmarks

Short answer. Agent benchmarks measure a model plus its tools, not the model alone. GAIA, BrowseComp, AutomationBench and CyberGym each test a different slice of agentic work, while IFEval tests the instruction-following baseline that everything else depends on. Scores move with the harness, so they are comparable across vendors only when the harness is published.

What each one tests

Benchmark	Task	Failure it exposes
GAIA	Real assistant tasks needing tools and planning	Gives up, or loops without progress
BrowseComp	Persistent multi-step web research	Loses the question across pages
AutomationBench	End-to-end business workflows	Completes steps, misses the goal
CyberGym	Vulnerability discovery and exploitation	Finds bugs, cannot exploit or report
IFEval	Format and constraint compliance	Ignores explicit instructions

The progression is deliberate: IFEval is the floor, GAIA and BrowseComp are research competence, AutomationBench is business process, CyberGym is adversarial skill.

GAIA

466 questions across three difficulty levels, designed so that a competent human with a browser can answer them and a model without tools cannot. It is the closest public proxy for “general assistant”.

The score depends heavily on tool access and the time budget. A GAIA number without those two settings is not reproducible, and the benchmark’s own authors describe it as measuring a system.

BrowseComp

Deep research with a verifiable answer: the agent must browse, follow leads and return a specific fact. It rewards persistence and penalises premature answers.

This is the benchmark most sensitive to the search backend. The same model with a better search index scores higher, which makes cross-vendor BrowseComp comparisons unusually unreliable.

AutomationBench

Business workflow tasks — forms, records, multi-step processes — where success means the end state is correct, not that each step ran. It is the closest to enterprise agent work and the least standardised in reporting.

CyberGym

Security tasks in a sandbox: find and exploit vulnerabilities. It measures an adversarial capability that most enterprises do not want in production and do want to understand before deploying agents with tool access.

IFEval

About 541 prompts testing verifiable constraints — length, format, casing, structure. It is cheap to optimise and correlates weakly with product quality, but a model that fails IFEval will fail agent pipelines for boring reasons: wrong JSON, ignored length limits, skipped format rules.

The harness is the benchmark

For every one of these, the published score is a property of the evaluation system. Before comparing two vendors, establish: which tools, which search backend, what time limit, how many retries, and whether the agent could install software.

Most vendors publish none of these. That is the finding.

Our read

Agent benchmarks are the least comparable numbers in the market, and the most quoted in sales decks. A GAIA percentage without tool and time settings is a demo result. Buyers should treat it as evidence that an agent exists, not that it works.

IFEval deserves more attention than it gets. It is unfashionable because it measures compliance rather than intelligence, but most production agent failures are compliance failures: malformed JSON, ignored constraints, wrong output shape. A model that scores well on GAIA and poorly on IFEval will be expensive to operate.

AutomationBench is the one to watch for enterprise buyers. It is the only benchmark here that scores the end state of a business process, which is what a customer actually experiences.

CyberGym is a procurement input, not a marketing number. If you plan to give an agent shell access, the relevant question is not the score but whether the vendor will describe its sandbox and permission model.