Understand AGI › Syllabus › Module 2
A benchmark score is a real number about a real thing. It is almost never the thing the headline says it is.
A benchmark is a fixed set of problems with known answers, run against a system, scored. That is all. Its usefulness rests entirely on one assumption: that doing well on this particular set of problems indicates something broader about the system.
That assumption is called construct validity, and it is where nearly every misreading of a benchmark result begins. The score is not in dispute. What the score stands for is where the argument lives.
Six specific things go wrong. Knowing them is enough to read most capability claims sensibly.
Every benchmark has a ceiling. As systems approach it, the benchmark stops discriminating: the difference between a good system and an excellent one compresses into the last couple of percentage points, which are also the points most likely to be noise or mislabelled answers.
This has happened repeatedly. A benchmark is built, becomes the field’s reference, gets saturated within a few years, and is replaced by a harder one - which is then saturated in turn.
The practical consequence is counter-intuitive: a flattening curve on a saturated benchmark is weak evidence that progress is slowing, because the instrument ran out of room before the thing it was measuring did. You would see the same flat line either way.
Modern systems are trained on very large quantities of text scraped from the internet. Benchmarks are published on the internet. If the test questions and their answers were in the training data, the score measures recall rather than capability.
This is not a hypothetical. It is a routine, well-documented problem, and careful evaluations now spend real effort on detecting and controlling for it: holding out fresh questions, checking whether a system can complete a test item it was only shown part of, comparing performance on problems published before and after a training cutoff.
When you read a benchmark result, the question “could this have been in the training data?” is not cynicism. It is the first thing a competent evaluator asks.
Once a benchmark matters commercially, it starts shaping what gets built. Not necessarily by cheating - by the ordinary process of choosing what to improve. Teams measure what they are judged on, and systems get better at exactly that.
Repeated evaluation against the same held-out set erodes it too. Every time a decision is made on the basis of a test set, a little information leaks from the test into the system, even if nobody ever trains on it directly.
This is Goodhart’s law arriving on schedule: when a measure becomes a target, it stops being a good measure.
Two systems can be reported on the same benchmark and not be comparable at all, because the procedure differed.
A score might be from one attempt, or the best of many attempts, or a majority vote over many attempts. It might allow the system to use tools, or not. It might allow it to think at length before answering, or not, and the amount of that thinking might differ by an order of magnitude between two reported numbers. Every one of those choices moves the score, sometimes by a lot.
None of this is hidden - it is generally stated in the technical report. It just rarely survives into the headline. When two numbers are being compared, the question is whether they were produced the same way.
“Better than human” requires a human number, and human numbers are slippery. Which humans? Given how long? With what resources? Motivated how?
An expert with a week is a different baseline from a crowdworker with ninety seconds, and benchmark human baselines are much more often the latter. A system can clear that bar comfortably while remaining far below what a competent professional would produce.
The claim is usually true as stated and much narrower than it sounds.
This is the one that matters most outside research, and the one benchmarks are worst at capturing.
A benchmark is a distribution of problems. Real work is a different distribution: messier inputs, ambiguous goals, missing context, consequences for being wrong, and no answer key. Performance frequently does not survive the move, and it can fall off sharply rather than gracefully.
This is why a system can be measurably excellent and practically unreliable at the same time, and why the two facts are not in tension. They are measurements of different distributions.
None of this means benchmarks are worthless. It means a score is evidence about something specific and you need to know what.
Before accepting a capability claim: what exactly does this benchmark test? Could the answers have been in the training data? How near is the ceiling? Under what procedure was the number produced? What was the human baseline, precisely? And how far is the benchmark distribution from the work you actually care about?
A claim that survives all six is worth taking seriously. Most do not get past the second.
Module 3 looks at who is producing these numbers and what else is acting on them.
Last reviewed
Enrol early
We will email you when each module publishes. You will also get AGI, Made Clear, keeping you up to date on AGI and ASI developments in a field that moves faster than anyone can follow.
Double opt-in. One-click unsubscribe. We never share your address.
AGI Made Clear covers what is happening, the evidence behind it and the claims that are not supported. This site teaches the subject. That one tracks it.
Visit AGI Made Clear →