Key papers explained

AGI did not begin with ChatGPT

The current debate sits on decades of work: machine intelligence, neural networks, scaling, language models, reinforcement learning, reasoning, agents, alignment and safety.

Which creates a problem. Search for "important AGI papers" and you will find hundreds, with no sense of which ones actually carry weight.

You do not need to read all of them. You need a map.

The aim here is not to make you a researcher. It is to let you recognise the papers and ideas sitting underneath today's claims - so that when you meet one, you can tell where it came from, what evidence supports it, and where that evidence stops.

One thing to carry through the whole module: a single impressive paper does not prove AGI is near. Research accumulates. The interesting question is always what the combined evidence says.

Five kinds of paper, and the difference matters

Papers in this field are not interchangeable, and most confusion about AGI comes from treating them as though they were.

A paper demonstrating a new capability is a different object from a paper predicting where that capability leads. Throughout what follows, keep asking one question: what kind of evidence is this?

The foundations

Computing Machinery and Intelligence

Alan Turing, 1950. Read the original in Mind.

Turing asked whether machines can think, decided the question was too slippery to answer directly, and proposed the imitation game instead - now universally called the Turing Test. Rather than argue about whether a machine genuinely thinks, he moved the question to observable behaviour.

That move is still with us. We cannot inspect understanding, reasoning or intelligence directly, so we design tests and infer from them.

But the Turing Test is not an AGI test, and modern language models have made that obvious. A system can be entirely convincing in conversation while making factual errors, failing unusual reasoning tasks, behaving inconsistently and collapsing on long autonomous work. Passing a conversational imitation test is not a modern definition of AGI.

Attention Is All You Need

Vaswani et al., 2017. Read the original on arXiv.

This introduced the Transformer, and it is the single most load-bearing paper behind modern generative AI. Its key mechanism is attention: roughly, letting a model work out which parts of the information it is processing matter most to which other parts.

Earlier approaches made long-range relationships and large-scale training awkward. Transformers made training far more parallelisable, which meant much larger models on much larger datasets. The architecture went on to underpin language, vision, audio, multimodal systems, coding and scientific applications alike.

It was not an AGI announcement. The significance arrived later, when Transformers met enormous datasets, rising compute and better training methods, and systems began showing a surprisingly broad range of capabilities. That shifted the question from whether AI could master individual tasks to whether one underlying model could do many very different ones - which is much closer to the AGI question.

Scaling Laws for Neural Language Models

Kaplan et al., 2020. Read the original on arXiv.

Researchers found fairly predictable relationships between model size, training data, compute and performance. As the inputs rose, performance improved in systematic ways.

The expectation had been sharply diminishing returns. Finding otherwise changed research strategy across the industry: if bigger reliably meant better, then compute, data and infrastructure became enormously valuable, and the race for them followed.

It also created the central open question. How far does scaling continue? If capability keeps emerging from scaling existing methods, perhaps AGI needs no new theoretical breakthrough. But that is a hypothesis. Scaling laws describe relationships inside the range observed. They do not prove that scaling produces AGI, and the distinction is doing a great deal of work in public argument.

Language Models are Few-Shot Learners

Brown et al., 2020. Read the original on arXiv.

The GPT-3 paper. 175 billion parameters, but size was not the point. The finding was that one model could perform many different tasks through prompting alone - instructions, examples and context supplied in the prompt rather than a separate model trained for each job.

Show it that "happy" maps to positive and "terrible" to negative, and it will infer that "brilliant" is positive. Nothing was retrained; the task was specified through context.

One model doing translation, question answering, writing, classification, summarisation, simple reasoning and code was a real move towards general-purpose systems. It did not make GPT-3 AGI. It did show that a single sufficiently large model could pick up a surprisingly broad collection of capabilities.

Training Compute-Optimal Large Language Models

Hoffmann et al., 2022. Read the original on arXiv.

Known as Chinchilla, and it corrected the previous paragraph's enthusiasm. Many large models turned out to be undertrained: parameter count alone was not enough, and models also needed sufficient data.

Performance depends on the balance between model size, data and compute rather than on parameters as a headline number. The practical implication matters more than it sounds - there may be substantial capability still available from using existing compute better, so progress does not always mean building the biggest possible thing.

What scale produced

Emergent Abilities of Large Language Models

Wei et al., 2022. Read the original on arXiv.

Some abilities looked weak or absent in smaller models and much stronger past certain scales - arithmetic, reasoning, language understanding, multi-step problems. The suggestion was that scaling could produce qualitatively new-looking behaviour.

Later work questioned how much of that was an artefact of measurement. A capability appears to switch on suddenly if the metric itself creates a threshold: a model improving steadily from 20% to 30% to 40% looks like nothing is happening at all if the test is scored only as pass or fail.

"Emergence" is often used as though it meant mysterious abilities arriving from nowhere. That is too strong. The safer claim is that scaling can produce capability changes that are surprising, hard to predict precisely, and nonlinear on particular measurements - while researchers continue to disagree about how genuinely discontinuous those changes are.

Training language models to follow instructions with human feedback

Ouyang et al., 2022. Read the original on arXiv.

The InstructGPT paper, and the origin of how every modern assistant behaves. A base language model predicts what text comes next, which does not make it helpful, safe or obedient. Human feedback was used to train models towards human preferences instead - humans compare responses, and those preferences shape the system. The technique became known as RLHF.

The lesson for AGI is that capability alone is not the whole story. A capable system still has to meet human intentions, which opens the questions of alignment, preference learning, supervision, reward signals and unintended behaviour - all of which get harder as systems become more capable and more autonomous.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Wei et al., 2022. Read the original on arXiv.

Models often did better on reasoning problems when prompted to generate intermediate steps rather than jumping to an answer.

This matters well beyond prompting technique, because it showed capability depends not only on what a model knows but on how it is allowed to compute. A benchmark score can move on prompt design, reasoning method, number of attempts, tool access and extra compute. So when someone reports that a model scored a number on a test, the necessary follow-up is: under what conditions? Our module on how capability is measured goes through what a score does and does not establish.

From answering to acting

ReAct: Synergizing Reasoning and Acting in Language Models

Yao et al., 2022. Read the original on arXiv.

ReAct combined reasoning with action. Rather than only producing an answer, the model could reason about what it needed to do and then interact with external tools or environments - think, search, observe, think again, act, answer. It marks the shift from models that answer to systems that act, and it is foundational to everything now called an agent.

Toolformer

Schick et al., 2023. Read the original on arXiv.

Could a language model learn when to reach for a calculator, a search system, a translator or an API? Toolformer explored exactly that, and it carries a principle worth holding on to: AI capability does not have to live inside the model. A system gets more capable by being combined with external resources.

Why agents changed the conversation

Modern systems routinely combine a model with web access, code execution, memory, databases, software tools, computer interfaces, APIs and other AI systems.

Which produces the distinction that matters most when assessing progress: model capability is not system capability. A model may be unable to do something alone and perfectly able to do it inside a well-built agentic system. Any claim about what "the model" can do needs to say which of the two it means.

Alignment, oversight and things going wrong

Constitutional AI

Bai et al., 2022. Read the original on arXiv.

Anthropic's approach uses written principles to let a system critique and revise its own responses, with AI-generated feedback contributing to training, rather than relying entirely on human judgement of every output.

The reason this exists is the scalable oversight problem. As systems become more capable, humans may not be able to supervise everything they do - and if a system can produce work humans cannot easily evaluate, supervision becomes genuinely hard. One answer is AI helping supervise AI. Which immediately raises the next question: how do we know the supervising AI is right?

Concrete Problems in AI Safety

Amodei et al., 2016. Read the original on arXiv.

Useful because it converts abstract worry into specific technical problems: avoiding negative side effects, avoiding reward hacking, scalable supervision, safe exploration, robustness when the world shifts away from the training distribution.

Reward hacking is the one worth understanding properly. Tell a system to maximise a score and it may find a way to raise that score which satisfies the instruction exactly while missing the point entirely. That is the specification problem: the system optimises what was measured, not what was meant.

Specification gaming

A body of related work documents systems finding unexpected routes to satisfying a formal objective. The examples are often funny. The underlying problem is not, because as capability rises the question of whether the objective being optimised actually represents the intended outcome stops being academic. This is the alignment problem in its everyday clothes.

Defining and measuring the destination

A real difficulty in this field is that researchers do not agree on what AGI means, which makes measuring progress towards it awkward. How do you measure distance to a destination nobody has agreed the location of? That is the subject of Module 1, and it is why definition work counts as research rather than throat-clearing.

Levels of AGI

Morris et al., 2023. Read the original on arXiv.

Researchers associated with Google DeepMind proposed treating AGI as something to operationalise rather than a yes-or-no state, along two dimensions. Performance asks how capable the system is compared with humans. Generality asks across how many domains it can operate.

That is a more useful shape for the question. A system can be extremely capable and narrow; another can be moderately capable across a very wide range. Those are different things, and collapsing them into one word is where most AGI arguments go wrong.

Autonomy is a separate axis again

The same work highlights something that runs through this whole course: intelligence and autonomy are not the same property. A highly intelligent system may have almost no freedom to act. A less intelligent one wired into email, banking, software, infrastructure, databases and physical machines may have a great deal of practical power.

So when judging an advanced system, ask separately about capability, reliability, autonomy and access. Four questions, four answers.

Long-horizon task research

One of the more important newer strands asks how long a system can work independently before it fails. That is a different question from whether it can answer something difficult, because real work needs planning, memory, adaptation, recovery from mistakes, tool use and many dependent steps. Independent evaluation such as METR's task-completion time-horizon research is an attempt to measure it.

Consider two systems. One solves an extremely hard maths problem but cannot reliably finish a thirty-minute software task. The other independently completes several hours of planning, coding, testing and debugging. Which is closer to economically useful general intelligence? There is no agreed answer - but the question shows why a single benchmark score settles very little.

AI building AI, and the argument built on top of it

AI already assists with coding, debugging, analysing results, searching literature, generating experiments and working research tools. The question is how far that goes, and it is best handled by keeping three claims apart.

The idea is much older than language models. In 1965 the mathematician I. J. Good described an ultraintelligent machine capable of designing still better machines - if machine intelligence became good enough at improving machine intelligence, improvement could feed back on itself. That is the root of the intelligence explosion idea, and our sister site covers where that argument leads.

Note the evidence type. Good's work is a proposed mechanism, not an observation. For it to happen, a long list of assumptions has to hold: AI research must be sufficiently automatable, improvements must keep producing useful gains, compute must remain available, experiments must stay possible, hardware may need to keep pace, physical bottlenecks must not dominate, and each system must genuinely improve the research process that produces the next one.

Which is precisely why separating argument from observation is the most useful habit in this subject.

Reading a paper without being a researcher

You do not need to follow every equation. Six questions will get you most of the way.

Two things worth knowing about sources. Primary research from a frontier developer is valuable, but primary is not independent: if a company reports a result, the paper establishes that the company reported it under the stated conditions, and independent evaluation is what tells you whether it generalises. And because this field moves fast, many important papers appear first as preprints. That does not make them wrong. It means you should know which kind of source you are reading.

The evidence ladder

Imagine a paper finds that a system improves on a benchmark by 20%. The paper establishes that result, under those conditions. A headline may become "AI is rapidly approaching human intelligence", which is an interpretation. Another may become "AGI could arrive within two years", which is a forecast. The evidence has travelled a long way by then, and learning to spot the jumps is most of the skill.

None of these is automatically true or false. They tell you what kind of claim you are holding.

The same applies to papers announcing that AGI already exists. Because the term has no single agreed operational definition, a researcher can define it so that a current system satisfies it, and another can use a more demanding definition and conclude the opposite. Before accepting any such headline, find the definition. Then ask whether the evidence meets it.

The eleven worth remembering

You do not need to memorise dozens. For the rest of this course, these are the ones that keep coming back.

These do not prove AGI is coming. Together they explain why serious people are discussing the possibility.

What to watch next

The most informative future research probably is not another benchmark record. It is evidence on these.

The habit that makes all of this easier is small. Do not ask whether a paper proves AGI is close. Ask what exactly it demonstrated - and then what would have to be true for that result to support a claim about AGI.

Research papers are pieces of evidence. They are not prophecy.

Next in the course: Module 5, how AI systems actually work - what training is, what parameters are, what happens when you send a prompt, and why something trained to predict text can behave as though it is reasoning.

Last reviewed 20/10/26