The Small Prediction Behind AI’s Biggest Leap
How Next-Token Prediction Built AI
The Small Prediction Behind AI’s Biggest Leap
Systems that write reports, translate languages and generate computer code are built around a task that sounds almost trivial: predict what comes next. The surprise is not that the machines can complete a sentence, but that scaling this narrow objective has produced systems able to imitate reasoning, adapt to new instructions and operate across domains that their designers did not programme separately.
That achievement has encouraged grand claims about artificial minds. Some are justified by the machines’ performance; others are marketing dressed up as metaphysics. The evidence shows a technology of extraordinary breadth, but not a machine that has settled the old questions of understanding, judgement or consciousness. To see what has actually happened, it is necessary to look past the chatbot interface and follow the mathematics back to its most basic instruction: choose the most plausible next token.
The apparently modest task that changed the field
A language model is trained by showing it pieces of text and asking it to estimate what should follow. In everyday speech this is usually called next-word prediction, although the technical unit is often a token: a word, part of a word, punctuation mark or other small fragment. A sentence such as “The capital of France is” supplies a context. The model assigns probabilities to possible continuations, with “Paris” receiving a high score. It then compares its prediction with the text that actually followed and adjusts itself when it was wrong.
Nothing in that description sounds like a theory of intelligence. It sounds like an advanced autocomplete system. That description is not false, but it is radically incomplete. A useful prediction depends on more than the immediate word before it. To complete “The capital of France is” reliably, a system must connect geography, language and the conventional form of a factual statement. To complete a legal argument, a software function or a scientific explanation, it must recognise the patterns that make those forms coherent.
The central argument of modern language-model research is therefore not that the machine has been secretly handed a list of rules called “reasoning”. It is that language contains traces of the reasoning used to produce it. Text records facts, procedures, explanations, disputes, narratives and examples of people solving problems. A system trained to predict text is pushed towards representing the regularities that make those forms of text possible.
That does not mean every prediction is an act of thought. A parrot can produce a meaningful phrase without mastering its meaning, and a statistical model can reproduce a pattern without knowing why the pattern matters. The point is narrower and more consequential: when the prediction problem is made sufficiently large, solving it requires internal structures that behave as if the system has learned grammar, facts, categories and some methods of inference.
This is why the phrase “just autocomplete” is both a useful warning and a poor explanation. It warns users not to confuse fluent output with guaranteed knowledge. It fails as an explanation because the word “just” hides the scale and complexity of the problem. Predicting the next token across a large share of human writing is not a single lookup operation. It is a demand to compress an immense variety of relationships into a numerical system and then use that system to extend unfamiliar sequences.
The architecture that made scale practical
The decisive engineering change came in 2017 with the Transformer, introduced in the paper “Attention Is All You Need”. Earlier language systems relied heavily on recurrent architectures that processed sequences step by step. The approach matched the way people often describe reading: take one word, update an internal state, then move to the next. But that sequence imposed a bottleneck. Computation had to wait for earlier computation, and long-distance relationships were difficult to preserve.
The Transformer replaced much of that sequential machinery with self-attention. The model can examine the relationships among many positions in a sequence and calculate which parts deserve emphasis for the task at hand. When the word “bank” appears in a sentence, the relevant clues may be several words away. Terms such as “loan” and “interest” point in one direction; “shore” and “river” point in another. Attention gives the network a way to weigh those connections rather than treating the sentence as an unbroken chain.
The innovation mattered for two reasons. It improved the model’s ability to represent context, and it allowed much more of the training calculation to run in parallel on specialised hardware. The original paper reported strong results on machine translation while requiring less training time than leading systems of the period. Its importance was larger than its initial benchmark scores suggested: it supplied the basic design that later language models would enlarge, modify and combine with other training methods.
Attention should not be confused with human concentration. The model does not decide, in a conscious sense, to focus on a word. It performs matrix operations that assign changing numerical weights to elements of the input. Yet the mechanism produces a functional result that is useful for language: information from different parts of a context can be brought together as the network builds a representation of what comes next.
The architecture also made the economics of scale more attractive. A model that can process training examples in parallel can use more examples and more computing power without being trapped by the pace of a single sequential path. This is one reason a technical paper about translation became the foundation for systems that now write essays, answer questions and generate images or code. The breakthrough was not a sudden insertion of common sense. It was a more efficient way to learn relationships from data.
What the model learns while trying not to be wrong
During training, the model’s parameters are adjusted by a process known as gradient descent. The details are formidable, but the principle is straightforward. The model makes a prediction, calculates how far that prediction is from the observed answer, and sends a mathematical correction through the network. Repeated across vast numbers of examples, those corrections alter billions of numerical weights.
A parameter is not a fact stored in a neat compartment. It is closer to a small setting in a huge web of interacting values. One setting may contribute to many associations, while a single concept may depend on thousands of settings working together. The result is not a database in which “Paris” sits beside “France” under a labelled entry. It is a distributed pattern that can influence the probability of many related tokens in many contexts.
That arrangement helps explain why language models can generalise. If training exposed the system to numerous examples of explanations, it may learn more than the exact sentences it saw. It may learn that a definition is often followed by an example, that a proof proceeds through linked steps, or that a computer function must obey the syntax of a programming language. The model is not merely retrieving one phrase; it is using a network of learned regularities to generate a new continuation.
But generalisation has a darker companion. The training material contains errors, propaganda, stereotypes, private information and mutually contradictory accounts. A model trained to predict what tends to appear in text can reproduce the weaknesses of that text. It does not begin with a reliable truth filter. If a false claim is repeated often, repetition can make the claim statistically familiar without making it true.
That is why fluent output should not be mistaken for verified reporting. The model’s core objective rewards a plausible continuation, not an independent investigation. Later stages of training can improve instruction-following, safety and usefulness, while external retrieval systems can supply current evidence. Those additions matter. They do not erase the basic fact that the underlying engine learned by modelling patterns in language.
The tension is visible in the same response. A model can explain a complicated idea in clear prose because it has absorbed the patterns of explanations. It can also invent a citation in a confident tone because it has learned what citations look like without necessarily checking whether a source exists. The machine’s strength and weakness come from the same place: it is exceptionally good at producing a continuation that fits the context.
When numbers became capabilities
The industry’s most important discovery was that performance improved as researchers increased the size of the model, the amount of training data and the computing power used to process it. Work on scaling laws found that language-model loss—the measure of how poorly a model predicts text—often falls in predictable power-law patterns as these resources rise. The finding transformed model development from a search for a clever trick into an industrial programme of systematic enlargement.
Parameter counts became the headline measure. GPT-3, described in its 2020 research paper, had 175 billion parameters and showed stronger few-shot performance than earlier systems. The model could be given a task and a small number of examples in the prompt rather than receiving a new training process for every individual application. It could translate, answer questions, unscramble words and perform some forms of arithmetic without updating its weights for each task.
The significance of few-shot learning was not that the model had become a miniature human. It was that a general-purpose system could infer the format of a task from context. A user could show examples of a desired transformation and ask for another. The model would continue the pattern. This ability made the same trained network useful as a writing assistant, translator, programmer and question-answering system.
Size, however, is not a magic number. Bigger systems cost more to train and operate, demand more electricity and require more data. A later line of research showed that model size and training data must be balanced. The Chinchilla study found that models of a given computing budget could be significantly undertrained if companies concentrated on enlarging parameters while holding the amount of data too low. A smaller model trained on more tokens could outperform a larger but undertrained competitor.
This matters because the story of AI is often told as if intelligence were purchased by adding zeros to a specification sheet. The reality is less glamorous and more expensive. Progress depends on the quality and mixture of data, the architecture, optimisation, evaluation, hardware, inference methods and the post-training process that teaches a model to follow instructions. Scale creates opportunity, not certainty.
There is also a practical distinction between what the model knows in its parameters and what it can do at the moment of use. A model may have learned a broad range of mathematical and linguistic patterns, yet fail on a simple question because the prompt is ambiguous or the relevant pattern is not activated. Conversely, a carefully written prompt can make a capability visible that was not obvious in casual use. The result can look like a sudden change in intelligence when it is partly a change in access.
The disputed drama of emergence
Researchers and commentators have used the word “emergence” to describe abilities that seem to appear once a model reaches a certain scale. A smaller model may fail a task, while a larger one performs it with striking competence. The popular interpretation is that quantitative growth has produced a qualitative jump: enough parts, enough data and enough training have generated a new kind of behaviour.
There is something real behind the observation. Larger models can represent more information and more complicated relationships. They can maintain longer dependencies, recognise subtler instructions and combine patterns that smaller systems cannot hold together. In practical use, a model can cross a threshold at which it becomes reliable enough to be useful. A system that gets an answer right one time in five is experienced differently from one that succeeds four times in five.
But the strongest claims about sudden emergence have been challenged. A 2023 study argued that some apparent jumps were produced by the way researchers measured performance. If a task is scored with an all-or-nothing metric, small improvements in the underlying probability can remain invisible until enough answers cross the scoring threshold. The graph then looks like a cliff even when the underlying capability has improved gradually.
Another study proposed that supposed emergent abilities could reflect a mixture of in-context learning, memorised information and linguistic knowledge rather than a new power appearing from nowhere. That interpretation is less dramatic, but science is not improved by choosing the most dramatic graph. If a model becomes more capable because it can use context more effectively, that is still a major achievement. It simply tells us that the mechanism is different from the story of an inner light switching on.
The argument matters beyond academic terminology. If capabilities improve smoothly, developers may be able to forecast performance and evaluate systems more responsibly. If capabilities arrive unpredictably, safety testing becomes harder and corporate claims become easier to inflate. The language of emergence can describe a real change in what users see while still misleading them about how that change occurred.
The sober view is that scale can reveal new combinations of existing competencies without proving consciousness or general intelligence. A model may learn enough language, facts and procedures to perform a task that no engineer wrote as a dedicated rule. That is emergence in the ordinary sense of a complex behaviour arising from simpler components. It is not evidence, by itself, of a mind behind the output.
Reasoning may be a learned pattern, but it is not a free gift
Language models appear to reason when they break a problem into stages, compare alternatives or draw a conclusion from premises. The appearance is not empty. Training data contains worked examples, arguments, proofs, explanations and code. A sufficiently capable model can reproduce the structure of those activities and apply it to a new prompt.
Yet reasoning in a model is not guaranteed by eloquence. A paragraph can be grammatically perfect and logically defective. The system may produce a convincing chain of steps that rests on a false assumption, changes the meaning of a term midway through the answer or reaches the right conclusion for the wrong reason. It can also solve a familiar problem through a memorised pattern while failing when the wording changes.
This is where the distinction between knowledge and process becomes important. The parameters encode associations and strategies, but generation still proceeds token by token. The model chooses a continuation based on the current context, adds that continuation to the context and predicts again. A long answer can therefore be coherent because each step supports the next, but errors can compound as the sequence grows.
Researchers have developed prompting and post-training methods to encourage more reliable multi-step work. Asking for intermediate reasoning, supplying examples or allowing the system to use external tools can improve performance on some tasks. A calculator can handle arithmetic more reliably than an unaided language model. A search system can check a current fact. A code interpreter can execute a programme instead of merely describing what it should do.
Those tools do not prove that the base model has acquired human-like reasoning. They show that a language model can serve as part of a larger computational system. The same is true of human reasoning, which depends on notebooks, diagrams, calculators, institutions and other people. But the comparison should not be used to excuse mistakes. A machine deployed in a hospital, court or newsroom must be judged by the consequences of its failures, not by the elegance of its explanations.
The strongest case for these systems is not that they think exactly as people do. It is that they can perform useful cognitive work through a different mechanism. The strongest case against overclaiming is that useful performance does not settle what the system understands, intends or knows about the world. Both statements can be true at once.
Generation is probability made visible
When a user submits a prompt, the trained model converts the text into tokens and numerical representations. Those representations pass through layers of attention and other transformations. At the end, the system produces a probability distribution over possible next tokens. It selects one, attaches it to the context and repeats the process until it reaches a stopping point.
The model is not writing an entire essay in one hidden act. It is generating a sequence. That sequence can still show planning-like behaviour because the model’s current output changes the context for its next decision. If the first sentences establish a particular argument, later predictions are constrained by the language already produced. Coherence emerges from the interaction between the learned network and the growing text.
Generation also involves a choice about randomness. A system can select the highest-probability continuation, which tends to produce predictable language, or sample from a wider range of candidates. The temperature setting changes how sharply the probability distribution is used. Lower values favour conventional answers; higher values permit more variation and can produce more original but less reliable prose.
This explains an everyday puzzle: the same question can receive different answers. The model is not necessarily consulting a fixed page in a digital memory. It is sampling a continuation from a context-sensitive distribution. Even when a service makes generation more repeatable by fixing settings, the answer remains the product of a probabilistic process rather than a guaranteed retrieval of truth.
Randomness is useful for creative work, but it has no respect for the boundary between imagination and fact. A model can invent a fictional character on request and then, under a slightly different prompt, invent a historical quotation in the same polished voice. The form of the language does not announce the reliability of the claim. Users must supply that discipline through source checking, constrained workflows and clear instructions about when the system is allowed to speculate.
The interface hides these mechanics. A typing animation makes the output feel like a person thinking aloud. The speed is impressive, but it can also encourage an unjustified intimacy with the machine. What appears to be a stable speaker is a system repeatedly calculating likely continuations. Its voice can be consistent because the model has learned stylistic patterns and the product adds rules around it. The apparent personality is a useful interface, not proof of an inner self.
The old philosophical objection has acquired a new target
The debate over machine understanding long predates large language models. In 1980, philosopher John Searle presented the Chinese Room thought experiment. He imagined an English-speaking person who did not know Chinese locked in a room with instructions for manipulating Chinese symbols. By following the instructions, the person could send back appropriate answers, leading people outside to believe that a Chinese speaker was inside.
Searle’s point was that correct symbol manipulation might simulate understanding without creating it. Syntax, in this argument, is not semantics. A system can handle formal marks according to rules while lacking any grasp of what those marks mean. The thought experiment was aimed at strong claims about artificial intelligence, but it now reads like an unusually direct challenge to chatbots that produce fluent language without demonstrating a human body, biography, or conscious experience. Of course, ‘understanding’ could also be this kind of manipulation, but using biological neural networks?
The replies are not trivial. One response says the person is only part of the system; the room, instructions and stored information together may constitute the understanding. Another says that a system connected to the physical world through sensors and actions could acquire the grounding that a sealed room lacks. A further response argues that we already attribute minds to other people on the basis of behaviour, so refusing to attribute understanding to a machine that behaves intelligently may apply an impossible standard.
Large language models sharpen the disagreement because their capabilities are no longer confined to canned demonstrations. They can answer follow-up questions, revise a draft, explain code and transfer patterns across tasks. These behaviours make the phrase “mere symbol manipulation” sound less decisive than it did when computer programs followed narrow hand-written scripts.
But improved performance does not end the philosophical dispute. A system can behave as if it understands while its internal process remains unlike human experience. The practical question is often more modest: does the system perform a task with enough reliability to be useful under specified conditions? The philosophical question is different: does the system possess meaning, awareness or intentionality? Confusing the two leads either to empty scepticism—nothing counts unless it is human—or to empty triumphalism—anything fluent must be a mind.
The honest position is uncomfortable. We can measure what models do far better than we can settle what they are. Their behaviour warrants serious study and careful use. It does not warrant pretending that a probability distribution has confessed to having a soul.
The cost hidden behind the simple prediction
Next-token prediction is mathematically simple to state, but modern training is an industrial operation. The data may include books, websites, code, articles and conversations, assembled and filtered at enormous scale. The model must process that material many times or across a vast number of distinct training examples while specialised processors carry out the required calculations.
The hardware bill is only one part of the cost. Training facilities need electricity, cooling, networking and engineering staff. Models must be evaluated for quality and safety, then refined so they respond to instructions rather than merely continuing a document. Once deployed, they consume resources every time a user asks for a response. Long prompts and long answers increase the computation required for inference.
That economic reality shapes the technology. The largest systems are developed by organisations able to assemble capital, data and computing infrastructure. The public often sees the chatbot as a weightless service in a browser, but the underlying process depends on physical machinery and a supply chain of advanced chips, data centres and electricity. The phrase “just predict the next token” conceals a large industrial base.
There are also social costs that cannot be reduced to power consumption. Training data can include copyrighted work and personal information. Models can reproduce biases present in the material on which they were trained. Automated text can flood search results, customer-service channels and public debate. When a model generates a false accusation or fabricated source, the cost is borne by the person who must correct it, not by the probability distribution that produced it.
None of this makes the technology a fraud. It makes the technology material. The most important AI systems are not disembodied minds floating above society; they are products of data selection, commercial incentives, labour, energy and infrastructure. Their abilities reflect what was included in training, what was excluded, what was rewarded after training and what users are prepared to tolerate.
That is why claims about inevitable progress deserve scrutiny. Scaling may continue to improve performance, but every additional gain requires choices about data, money, power and risk. A model can be more capable and still be less trustworthy in a high-stakes setting. Intelligence, if that is the word, is not the only variable that matters.
Why fluent machines still fail in obvious ways
The most persistent mistake in public discussion is to treat competence as a single scale. A model that writes a persuasive explanation of a scientific idea may fail to count letters in a word. A system that produces useful code may misread a date. One that translates a common sentence accurately may mishandle a rare dialect or an ambiguous legal phrase. These are not contradictions once the model is understood as a pattern-learning system rather than a person with a unified store of common sense.
Human beings also make mistakes, but the comparison is often made carelessly. People can ask what they do not know, observe the world, remember personal experience and revise their beliefs when evidence changes. A language model, without access to tools or current information, has no direct route to the world beyond the patterns encoded in its training and the context supplied by the user.
Its confidence is therefore a poor guide. The model’s probability distribution concerns which token fits the context, not whether the proposition is true in the external world. A false statement can be highly probable because it resembles familiar text. A correct but unusual answer can receive less support because it appears rarely in the training material.
Evaluation must take account of this difference. A benchmark score can reveal progress on a defined task, but it cannot certify general intelligence. Tests may overlap with training data, reward memorisation or measure a narrow skill with a convenient answer format. When researchers change the metric, the apparent level of capability can change with it. That does not make benchmarks useless; it makes them evidence rather than verdicts.
Users can reduce some failures through retrieval, citations, structured prompts, independent checks and tool use. Organisations can add review requirements and restrict systems from making unsupervised decisions. But no workflow can turn a language model into an oracle. The correct response to a confident error is not surprise. It is recognition that the system has succeeded at producing plausible language while failing at verification.
The lesson is not to dismiss the machine. It is to assign it work that matches its strengths and build safeguards around its weaknesses. Drafting, summarising, translation, coding assistance and exploration can benefit from speed and breadth. Medical diagnosis, legal conclusions, financial decisions and factual reporting require stronger evidence and accountable human oversight.
From research trick to general-purpose machine
The historical importance of next-token prediction lies in the range of tasks that can be reached through language. Once a system learns enough patterns in written instructions, examples and explanations, the prompt becomes a general interface. A user can describe a programming problem, a translation request or a writing task in ordinary language and ask the same model to attempt it.
This is a major departure from older software, which usually required a dedicated interface and explicit rules for each function. The language model does not need a separate button for every possible transformation. It can infer the requested format from the prompt. That flexibility is why a model trained on text can act as a rough programmer, editor, tutor or research assistant.
But general-purpose does not mean general intelligence. A Swiss Army knife is versatile because its tools are built into one object; it is not a doctor, carpenter or pilot. The language model’s versatility comes from the fact that human knowledge is expressed in language and that many tasks can be described as transformations of text, code or structured instructions.
The next phase is likely to combine language prediction with other capabilities. Models can be connected to search engines, databases, calculators, software tools, cameras and robots. The result may be more useful than an isolated text generator because the system can retrieve information, act on the world and check some of its own work. Yet each connection adds new failure modes, from stale data to unsafe actions and ambiguous permissions.
The transformation of work will therefore be uneven. Some tasks will become faster without disappearing. Others will be reorganised around reviewing and directing machines. In fields where output is mostly routine language, the pressure may be intense. In fields where trust, responsibility, physical judgement or human relationships matter, automation will be more limited or will require a human to remain visibly accountable.
The machines’ achievement should be described accurately. They are not empty tricks, because the patterns they learn can support impressive transfer and problem-solving. They are not settled artificial minds, because performance alone does not establish understanding or consciousness. They are large predictive systems whose simple objective, pursued with extraordinary data and computation, has produced a surprisingly broad form of machine competence.
That is the real story behind the spectacle. The leap did not begin with a computer being taught to think. It began with engineers asking a machine to be less wrong about what comes next—and discovering that, to do so across the range of human language, the machine had to learn far more about our world than the instruction seemed to require.
Sources
This post contains affiliate links. If you purchase through these links, I may earn a commission at no extra cost to you.
Leave a Reply