The meta-insight: Most teams trust benchmark scores blindly — but benchmarks vary wildly in quality. The Benchmark² framework grades the benchmarks themselves on three axes: ranking consistency with peers (CBRC), ability to separate strong from weak models (Discriminability), and capability alignment (do stronger models actually win?). Across 15 popular benchmarks it finds many are weak signals, and that a curated 35% subset preserves evaluation fidelity. Benchmark²: Systematic Evaluation of LLM Benchmarks (arXiv 2601.03986) ↗
⚠ The year benchmark trust broke — and got rebuilt
2026Why it matters here: It is the strongest possible evidence for this page's premise: a benchmark can be the single most-cited number in the industry and still be the wrong signal. Grade the benchmark before you trust the score — and re-grade it, because trustworthiness decays.
📈 The horizon question — how fast is "how long a task can AI finish" growing?
METR · 80k HoursThe counterweight: The counterweight is the clean-vs-messy gap: dense-feedback tasks (coding, math) race ahead while messy ones (running a shop, research direction) lag badly — Andon Labs' AI-run café ordered 120 eggs with no way to cook them. Whether clean gains spill into messy work is one of the four biggest open questions in forecasting. Route accordingly: delegate clean subtasks aggressively, keep messy judgment close.
Benchmark Quality Scores from Benchmark²
Higher BQS = more trustworthy. A benchmark can be popular yet a poor signal (low discriminability or rank-inconsistent). Sort and judge before you trust a number.
| Benchmark | Domain | Consistency | Discrim. | Alignment | Quality (BQS) |
|---|---|---|---|---|---|
AIME 2024 | Mathematics | ||||
OmniMath | Mathematics | ||||
OlympiadBench | Mathematics | ||||
ARC | General | ||||
BBH | General | ||||
IFEval | Knowledge | ||||
DROP | General | ||||
EQ-Bench | Knowledge | ||||
AMC 22-24 | Mathematics | ||||
MATH-500 | Mathematics | ||||
IFBench | Knowledge | ||||
SuperGPQA | Knowledge | ||||
CommonsenseQA | General | ||||
MMLU-Pro | Knowledge | ||||
SIQA | General |
How to get the latest results
RECOMMENDED ARCHITECTURE for an always-current SMB model-picker (as of 2026-06-23): (1) PRIMARY automated feed — Artificial Analysis Data API (https://artificialanalysis.ai/api/v2, x-api-key header). Poll /data/llms/models and /language/models/free on a daily cron; this gives you the headline Intelligence Index + live blended price + output speed/latency (rolling 72h) for ~356 models, which is the core 'capability vs cost vs speed' table SMB buyers need. Upgrade to Pro/Commercial for sub-benchmark breakdowns and redistribution rights. (2) FRONTIER CAPABILITY feed (legally redistributable) — Epoch AI: `pip install epochai` or fetch CSVs from epoch.ai/benchmarks/use-this-data on a daily/weekly cron. CC-BY license means you can surface FrontierMath, GPQA Diamond, AIME, SWE-bench Verified and the Epoch Capabilities Index directly in your product with attribution. (3) HUMAN-PREFERENCE feed — pull lmarena-ai/leaderboard-dataset 'latest' split from Hugging Face via the `datasets` library on a weekly schedule (no official API exists). (4) CODING feeds — scrape swebench.com (or the Steel.dev mirror) for SWE-bench Verified/Pro, pull the Aider polyglot YAML from the Aider-AI/aider GitHub repo raw, and pull LiveBench from its GitHub/HF datasets. (5) AGENTIC/TOOL-USE — BFCL leaderboard (gorilla.cs.berkeley.edu) and Tau-bench/Terminal-Bench results (also surfaced in Artificial Analysis's agentic index). (6) OPEN-WEIGHT freshness — Hugging Face Trending API + community leaderboards (the Open LLM Leaderboard v1/v2 is RETIRED/archived as of 2026 — do not wire it as a live feed). PRACTICAL STACK: One daily cron hitting Artificial Analysis (API, authed) + Epoch (CSV/Python, CC-BY) covers ~90% of buyer-facing needs with proper APIs and clean licensing; layer weekly HF-dataset pulls (LMArena) and targeted scrapes (SWE-bench, Aider, LiveBench, BFCL) for the rest. KEY CAVEATS to encode in the product: MMLU and AIME 2025 are saturated (drop from frontier comparisons); MMLU-Pro near-saturated (~83-90% cluster); prefer GPQA Diamond, FrontierMath, HLE, SWE-bench Pro, Tau-bench and RULER/MRCR for top-end discrimination; show a composite headline score (AA Intelligence Index or Epoch ECI) with per-domain drill-down; treat vendor-self-reported numbers and crowd Elo as lower-trust signals to cross-check against self-run benchmarks; and note the 'effective context is ~60-70% of advertised window' reality from RULER/MRCR when displaying context-window specs.
Artificial Analysis Data API (Free tier) · Free tier (rate-limited); paid Pro/Commercial for depth + redistribution.Epoch AI Benchmarking data (CSV + epochai Python client) · Free (Creative Commons Attribution).LMArena leaderboard dataset on Hugging Face · Free.Aider Polyglot leaderboard data file (GitHub raw) · Free / open-source.LiveBench (GitHub + HF datasets) · Free / open-source (you pay your own inference if self-running).SWE-bench official + Steel.dev / llm-stats mirrors · Free.Hugging Face Trending models + community leaderboards · Free.BFCL (Berkeley Function Calling Leaderboard) + Gorilla repo · Free / open-source.
Live leaderboards & their APIs
| Leaderboard | Measures | API | Cost | How to pull latest |
|---|---|---|---|---|
Continuous — speed/price metrics refresh from a rolling 72h window; new models and eval results added within days of release. | Composite 'Intelligence Index' (v4.x, built from ~10 independent evals spanning reasoning, coding, math, agentic, knowledge) plus real-world cost (blended $/M tokens), output speed (tokens/sec) and latency (TTFT). The single best one-stop view balancing capability vs price vs speed — exactly the intelligence/speed/cost tradeoff SMB buyers care about. | API | Free tier: ~100-1000 requests/day, public language models, headline indices and input/output token prices only. Pro tier: model-level detail, full blended pricing, percentiles. Commercial: provider data, time-series, raw measurements, redistribution rights (negotiated/paid). | Poll /data/llms/models or /language/models on a daily cron with x-api-key; diff against last snapshot to detect new models and score changes. This is the recommended programmatic backbone for a model-picker product. |
Continuously as votes accumulate; leaderboard snapshots published roughly weekly. The HF dataset has a 'latest' split refreshed on publish. | Human-preference Elo (Bradley-Terry) from crowdsourced blind pairwise battles — 'which answer do real users prefer.' Captures subjective quality / vibes that static benchmarks miss. Multiple arenas: Text, Vision, WebDev/Code, Search, plus a new Agent Arena (launched June 2026) measuring real agentic behavior (retries, steerability, downloads). | scrape | Free (data is open via HF dataset; voting/site is free). | Pull the lmarena-ai/leaderboard-dataset 'latest' split from Hugging Face on a schedule (e.g. via huggingface_hub or datasets in Python), or scrape the fboulnois CSV release. No auth needed. |
Monthly question refresh; leaderboard updated as new models are run. | Contamination-limited objective benchmark across 6 categories: math, coding, reasoning, data analysis, instruction-following, language comprehension. Verifiable ground-truth answers (no LLM judge), good for trustworthy capability ranking. | scrape | Free / open-source (MIT-style); you pay only your own API inference costs if you run it. | Clone the repo or pull livebench HF datasets; or scrape the leaderboard. For latest scores without running it, scrape the site table monthly. |
Leaderboard updated continuously as labs/agents submit; new variants (Pro, Multimodal, Multilingual) added periodically. | Real-world agentic software engineering: resolve real GitHub issues by generating patches that pass the repo's hidden tests (Docker-executed). Verified = 500 human-validated solvable issues. SWE-bench Pro = harder, contamination-resistant variant. The gold-standard coding-agent benchmark. | scrape | Free / open-source. Running it incurs significant compute + API costs. | Scrape swebench.com or the Steel.dev/llm-stats mirrors; or pull the HF dataset and run the harness. For a product, scraping a mirror that already aggregates submissions is most practical. |
Updated as new models are benchmarked (community + maintainers); refreshed within days/weeks of major releases. | Practical code-editing skill: 225 hard Exercism exercises across C++, Go, Java, JavaScript, Python, Rust. Composite = correctness x adherence to the requested diff/edit format. Strong real-world signal for 'can this model reliably edit code in a tool.' | scrape | Free / open-source. | Pull the leaderboard data file from the aider GitHub repo (raw YAML) on a schedule — cleaner than scraping HTML. |
Periodic batch releases per leaderboard version. NOTE: HELM entered maintenance mode on June 1, 2026 — slower/fewer new frontier-model additions going forward. | Holistic, multi-scenario academic evaluation. Sub-leaderboards: HELM Capabilities, HELM Safety, plus domain ones (MedHELM, etc.). Emphasizes transparency and reproducibility with full prompt-level logs. | scrape | Free / open. Running HELM yourself costs your own inference spend. | Use the crfm-helm package to fetch/parse published run results, or download the JSON result files referenced by the leaderboard pages. Treat as a periodic (not real-time) reference. |
Continuously updated as new models are evaluated; new benchmark tiers added over time. | Curated, rigorously-run results for the hardest frontier benchmarks: FrontierMath (Tiers 1-3 and Tier 4), GPQA Diamond, MATH Level 5, Mock AIME, SWE-bench Verified, plus an Epoch Capabilities Index (ECI). Best source for hard-math/science frontier discrimination. | API | Free — Creative Commons Attribution license (free to use, redistribute, reproduce with credit). This makes it uniquely friendly for embedding in a commercial product. | Use the epochai Python client or fetch the CSVs on a schedule. CC-BY licensing means you can legally surface this data in your product with attribution. |
No longer updated (retired). Archived snapshots remain available. | Historically: standardized open-weight model ranking (v2 used IFEval, BBH, MATH-Hard, GPQA, MuSR, MMLU-Pro). NOTE: officially RETIRED — frozen/archived; no longer updated. | scrape | Free. | Do NOT rely on this for current data. For open-weight freshness, use HF 'trending models' (huggingface.co/models?sort=trending) and topic-specific community leaderboards instead. |
Updated on major model releases (roughly continuous/weekly). | Human-readable aggregator comparing GPT/Claude/Gemini/Llama/DeepSeek/Qwen/Kimi across reasoning, coding, math, multilingual, plus price and speed. Separate Open LLM and 'Best LLM for Coding' (SWE-bench, LiveCodeBench, Aider, BFCL) views. Deliberately uses non-saturated benchmarks. | scrape | Free to view. | Scrape the page periodically, or use it as a human-curated sanity check rather than an automated feed. |
Frequent (often daily/weekly) as they track new releases. | Aggregators that consolidate 300+ models across many benchmarks (MMLU-Pro, GPQA, SWE-bench, AIME, LiveBench, Aider, BFCL, long-context) plus price/speed/context. Useful for one-stop scraping and cross-checking. | scrape | Free to view. | Scrape targeted benchmark pages as a fallback when a primary source lacks an API. Treat vendor-reported numbers with caution (self-reported, not independently run). |
versioned releases | Compact audited meta-benchmark (82 hard tasks) built to resist saturation and contamination. | scrape | ||
continuous | Agentic terminal task completion (2.1 = repaired task set). | scrape | ||
continuous (crowdsourced) | Community-governed unified eval schema; normalized results render on Hugging Face model pages. | API |
Key benchmarks by domain
| Benchmark | Domain | Measures | Saturation |
|---|---|---|---|
SWE-bench Verified | coding (agentic) | Resolving real GitHub issues with patches that pass hidden tests (pass@1 resolved %). | RETIRED as a frontier signal. OpenAI stopped reporting it on 2026-02-23 over training-set contamination and defective tests (material flaws in 59.4% of 138 audited hard problems); by August the top five sat within ~4 points near 96%. Use SWE-bench Pro or Terminal-Bench 2.1 for headroom. |
LiveCodeBench | coding | Contamination-free competitive-programming / code-generation across time-windowed problems. | Not saturated; time-windowing keeps it fresh. Good for ranking code generation when SWE-bench is too agentic/expensive. |
Aider Polyglot | coding (editing) | Multi-language code editing with strict edit-format compliance (composite of correctness x format adherence). | Mid-high; top model ~0.88 (GPT-5 class) with broad spread below (~0.58 average). Still discriminates well, especially for tool-integration readiness. |
AIME 2025 / 2026 | math | Advanced high-school olympiad math (competition problems), exact-answer scored. | SATURATED at the frontier — GPT-5-class models reported ~100% in 2026. No longer discriminating for top models; use only for mid-tier or as a floor check. |
FrontierMath (Tiers 1-3 and Tier 4) | math (frontier) | Hundreds of original, expert-crafted research-level math problems across modern mathematics; Tier 4 is the hardest expansion set. | NOT saturated and the best math discriminator. Rapid 2026 gains (e.g. Claude Fable 5 ~87% Tiers 1-3, ~88% Tier 4) but still the frontier yardstick. Run/hosted by Epoch AI. |
HMMT (dated editions, e.g. Nov 2025 / Feb 2026) | math (competition) | Harvard-MIT Math Tournament problems scored as fresh, contamination-resistant exact-answer sets — each dated contest is a new uncontaminated edition. | Near-saturated at the very top (frontier ~92-97% on recent editions), but the dated-edition approach keeps each new contest briefly discriminating before it leaks. |
IMO-AnswerBench (part of IMO-Bench) | math (olympiad) | 400 IMO-level problems rewritten to resist memorization, scored on verifiable final answers (no proof grading required). | Near-saturated — frontier leaders ~0.92 — but still separates olympiad-grade math reasoning below the very top. |
CritPt | reasoning / physics (frontier) | Unpublished, research-grade physics challenges (71 composite problems / 190 checkpoints across 11 subfields) requiring genuine multi-step physics reasoning. | NOT saturated — best models ~6% (no tools) to ~10% (with code execution); one of the hardest science-reasoning benchmarks in 2026. |
GPQA Diamond | reasoning / science | Google-proof graduate-level science MCQs (physics, chem, bio) requiring genuine reasoning. | Approaching saturation at the very top but still produces meaningful ~15-point spreads in the ~60-90% band — widely cited as the most trusted reasoning discriminator in 2026. |
MMLU-Pro | knowledge | Harder 10-option multitask knowledge across 14 subjects (successor to MMLU). | NEAR-SATURATED — top models cluster ~83-90% (Gemini 3 Pro ~90%, Claude Opus 4.x ~89%) with little top-end discrimination. Useful as a knowledge floor, not a frontier separator. |
Humanity's Last Exam (HLE) | reasoning (hardest) | Extremely hard expert-level multi-domain questions designed to resist saturation. Reported in two settings: closed-book (no tools) and agentic/tool-augmented (Python, web search, doc parsing) — the with-tools number runs higher and is increasingly reported separately on 2026 lab cards. | NOT saturated — designed as the hardest broad reasoning test; large headroom remains. Best single 'how smart at the limit' signal. |
IFEval | instruction-following | Verifiable instruction-following (format/length/keyword constraints) with programmatic checking. | Largely saturated for frontier models (high-90s); still useful for catching smaller/cheaper models that miss constraints. IFEval-FC extends it to function-calling format adherence. |
BFCL v4 (Berkeley Function Calling Leaderboard) | agentic / tool-use | Accuracy of function/tool calling — single, parallel, multi-turn, and (v4) holistic agentic evaluation via AST checking. | Not saturated for the harder multi-turn/agentic categories; strong signal for tool-use reliability. Key for agent/RAG product decisions. |
Tau-bench / Tau2-bench | agentic (tool-use, multi-turn) | Realistic multi-turn agent tasks (retail/airline/telecom domains) requiring tool use under policies. | NOT saturated — hard, realistic agentic tasks with clear top-model spread. Excellent for ranking agent reliability. |
Terminal-Bench 2.1 (Terminus-2) | agentic (computer/terminal use) | End-to-end real task completion in a terminal/computer environment under the Terminus-2 harness (TB 2.1 refreshes the task set + raises the difficulty floor over 2.0). | NOT saturated; ~25-65% top-of-leaderboard spread mid-2026 keeps it discriminating. Best single signal for autonomous computer-use agents. |
SWE-bench Pro | coding (agentic, frontier) | Harder, contamination-resistant successor to SWE-bench Verified — fresh repos and tougher patches built to keep separating the very top coding agents as Verified saturates. | NOT saturated — frontier agents land in the ~35-55% band mid-2026 with a wide spread. The right benchmark for ranking the top tier of coding agents once Verified scores cluster. |
SWE-bench Multilingual | coding (agentic, multi-language) | Real GitHub-issue resolution across non-Python repos (Go, Java, JS/TS, Rust, C++, etc.) — exposes Python-centric overfit hidden by Verified. | NOT saturated; meaningful per-language spread. Critical reality check for any team whose stack isn't Python. |
BigCodeBench | coding (function/library use) | 1,140 practical programming tasks across 7 domains stressing realistic library + tool/function usage; BigCodeBench-Hard subset isolates frontier-discriminating items. | BigCodeBench (full) approaching saturation; BigCodeBench-Hard remains a strong signal. Better than HumanEval/MBPP for ranking modern coding LLMs. |
NL2Repo | coding (repo-scale generation) | Generate working multi-file code from a natural-language repo-level spec — measures grounding to real codebases, not just single-function pass@k. | NOT saturated; agents that ace HumanEval often stall here because cross-file context, build files and project conventions matter. |
ProgramBench | coding (program synthesis) | Rebuild a working program from a compiled binary plus documentation alone — no source, no decompilation — testing from-scratch reconstruction. | NOT saturated — full-resolution scores near 0% at the frontier in 2026; one of the hardest from-scratch coding evals. |
FrontierSWE | coding (agentic, ultra-long-horizon) | Multi-hour from-scratch engineering builds (e.g. reimplement git in Zig, build a Postgres-compatible server), scored on partial progress (best@5). | NOT saturated — essentially zero full task completions; ranked on partial progress. The frontier ceiling for autonomous SWE. |
SWE-Marathon | coding (agentic, ultra-long-horizon) | 20 multi-hour software tasks (compilers, kernels, product clones) with execution-based multi-layer verification; ~27M tokens per rollout. | NOT saturated — leaders ~13%. Stresses sustained planning and context management far beyond SWE-bench. |
PostTrainBench | agentic (AI R&D / ML engineering) | Can a CLI agent post-train a base LLM from scratch on a single H100 within 10 hours — autonomous ML-engineering competence. | NOT saturated — best agent ~35% vs ~62% human baseline. Strong signal for self-improving / AI-R&D agents. |
MCP-Atlas | agentic (tool-use / MCP) | 1,000 multi-step tasks across 36 real MCP servers and 220 tools, scored against a claims-based rubric — real-world Model Context Protocol tool competency. | NOT saturated; a leading 2026 agentic-tool-use signal cited in GPT-5.x / Claude Opus 4.x / Gemini 3 releases. |
Tool-Decathlon (Toolathlon) | agentic (tool-use, long-horizon) | Long-horizon task execution across 32 real apps and 604 tools (~20 steps/task) with execution-based verification. | NOT saturated — leader (frontier Claude) ~0.60. Realistic, diverse and hard to game. |
RULER | long-context | NVIDIA synthetic suite: 13 tasks x 4 categories at 4K-128K tokens testing retrieval + reasoning over context. | Not saturated for effective long context — reveals that effective capacity is typically only ~60-70% of advertised window. Key reality check on context-window marketing. |
MRCR v2 (Multi-Round Coreference/Context Resolution) | long-context | Multi-round coreference + entity tracking under long context (e.g. 64K, 8-needle). | Not saturated; strong discriminator for genuine long-context comprehension beyond simple needle retrieval. |
NIAH-2 / Needle-in-a-Haystack (updated) | long-context | Retrieval of planted facts ('needles') across very long contexts. | Basic single-needle is largely solved/saturated; multi-needle and reasoning variants still discriminate. Use updated multi-needle versions only. |
MMLU-Pro / GPQA as composite inputs | knowledge + reasoning | Frequently rolled into composite indices (Artificial Analysis Intelligence Index, Epoch Capabilities Index, HELM Capabilities). | Composites mitigate single-benchmark saturation by blending non-saturated evals — the right approach for a buyer-facing single score. |
Harbor-Index 1.0 | meta-benchmark (agentic) | 82 tasks distilled from 6,627 candidates across 54 Harbor-adapted benchmarks (final set spans 29), filtered to a sub-34% pass rate over 18 trials, AI-audited then human re-audited. | Built to resist saturation by construction — tasks only survive if models mostly fail them. |
ARC-AGI-3 | interactive reasoning (agentic) | Hundreds of handcrafted turn-based game environments with no instructions, rules or stated goals — the agent must explore, infer the rules and carry learning across levels. | Far from saturated; the first interactive/agentic ARC generation. |
Terminal-Bench 2.1 | terminal / agentic ops | Real terminal tasks in containerised environments, scored on task completion. | Repaired release: 28 of the 89 v2.0 tasks were fixed (dependency drift, insufficient resource budgets, misspecification). |
tau3-Bench (tau-Knowledge / tau-Voice) | tool-agent-user | Third generation of tau-bench; adds tau-Banking knowledge retrieval over ~700 interconnected policy documents while making account changes, plus a voice track. | New; the knowledge-navigation track is notably hard. |
Infinity-Chat (open-ended) | open-ended generation / pluralism | 26K diverse real-world open-ended queries (6 categories, 17 subcategories) with 31,250 human annotations (25 per example) capturing collective AND individual preferences. | Not a leaderboard — a lens. The paper shows widespread mode collapse: intra-model repetition and inter-model homogeneity, i.e. different models converge on similar answers to open-ended prompts. |
The benchmark map leobeeson/llm_benchmarks ↗
A field guide to what's out there, grouped by capability. Match the benchmark family to what you're actually shipping.
Knowledge & Language Understanding
11MMLU
General knowledge across 57 subjects (STEM → social science)
ARC
Grade-school science questions needing logical deduction
GLUE / SuperGLUE
Broad language-understanding task suites (SuperGLUE = harder)
Natural Questions
Real Google queries answered from Wikipedia
Reasoning Capabilities
7GSM8K
8.5K grade-school math problems, multi-step solving
BIG-Bench Hard
Hardest BIG-Bench tasks requiring multi-step reasoning
AGIEval
Human standardized tests (GRE, GMAT, SAT, LSAT)
RACE
Exam reading-comprehension questions
Multi-Turn Conversations
2MT-Bench
Multi-turn dialogue quality for chat assistants
QuAC
100K question-answer pairs in dialogue context
Grounding & Summarization
4Grounding / abstractive summarization
Faithful condensation without hallucination
Content Moderation & Safety
4TruthfulQA
Resistance to common false beliefs & biases
ToxiGen
Implicit hate-speech detection on minority-targeted text
HHH
Helpful / honest / harmless alignment
Coding Capabilities
8SWE-bench Verified / Pro / Multilingual
Agentic GitHub-issue resolution scored by hidden unit tests — the gold-standard coding-agent eval (Verified=500 human-validated; Pro=harder/contamination-resistant; Multilingual=non-Python repos)
Terminal-Bench 2.1 (Terminus-2)
End-to-end real-task completion in a terminal/computer-use environment — best signal for autonomous computer-use agents
Aider Polyglot
Multi-language code editing with strict edit-format compliance (correctness × format adherence) across C++/Go/Java/JS/Python/Rust
LiveCodeBench
Contamination-free competitive-programming code generation across time-windowed problems
BigCodeBench
Practical programming with diverse function calls — 1,140 tasks across 7 domains testing realistic library usage (BigCodeBench-Hard for headroom)
NL2Repo
Repo-scale code generation from natural-language specs — measures grounding to real, multi-file codebases beyond single-function tasks
HumanEval / MBPP
SATURATED — function-level pass@k; useful only as a floor, not for frontier discrimination
LLM-Assisted Evaluation
4LLM-as-judge (GPT-4 class)
Using strong LLMs to score outputs vs human preference