The Machines Can Pass the Test. That Does Not Mean They Understand It.
Why AI Still Struggles to Reason

The Machines Can Pass the Test. That Does Not Mean They Understand It.
Artificial intelligence systems can write legal prose, generate working code and solve demanding examination questions. Yet, new evidence suggests that their apparent reasoning remains brittle—and that the industry may be mistaking fluent performance for understanding. The consequence is not merely philosophical: companies are deploying systems that can sound certain while failing when a problem is reworded, a fact is missing, or a judgment cannot be reduced to a single correct answer.
The Scoreboard Has Become the Story
The modern AI industry has built its case for machine intelligence around scores. A model reaches a new mark on a mathematics test, improves on a coding benchmark or performs well on a professional examination, and the result is presented as evidence that the machine has crossed another threshold of thought. Investors interpret the graph as a forecast. Executives describe a capability jump. Commentators begin discussing which occupations will disappear next.
The numbers are useful, but they are not self-explanatory. A test score tells us what a system did on a defined set of tasks under defined conditions. It does not, by itself, tell us how the system arrived at an answer, whether it can transfer the underlying skill to a new setting or whether it knows when its answer is wrong. Those are not minor qualifications. They are the difference between a reliable reasoning system and a very sophisticated answer generator.
That distinction is being forced into view by research on the weaknesses of language-model evaluations. Apple researchers examining mathematical reasoning found that changing the numerical values in familiar word problems could reduce performance, even when the structure of the task remained the same. Adding a clause that appeared relevant but did not alter the solution could cause performance to fall by as much as 65 percent across leading models.1
A person who understands the problem should be able to recognize it in different clothing. The names can change. The numbers can change. An irrelevant sentence can be removed. The problem remains the problem. A system that has learned the pattern of a familiar question, by contrast, can be thrown off by the packaging.
This is the first uncomfortable fact about the current debate: impressive performance and general reasoning are not synonyms. They overlap, but they are not interchangeable. The industry has often treated them as if they were because the first is easy to measure and the second is hard to define.
That choice has consequences. Once a benchmark becomes a public scoreboard, the scoreboard becomes a target. Researchers train on examples that resemble the test. Model builders refine prompts and post-training methods to produce the kind of answers a grader rewards. Users learn which formulations make a system perform best. A measure that began as an instrument of discovery becomes part of the environment being optimised.
There is nothing improper about improving a score. The problem begins when a higher score is treated as proof of a broader mental capacity that the test never measured. A thermometer can tell us the temperature. It cannot tell us whether the room is comfortable, safe or worth living in. The AI industry keeps asking the scoreboard to answer questions about the mind.
When the Wording Changes, the Confidence Does Not
The weakness exposed by the Apple work is not that language models never solve difficult problems. They plainly do. The weakness is that their success can depend on features that a human regards as incidental. Numbers, clause order, familiar phrasing and the visual shape of a question can carry more weight than the logical relationship the question is supposed to test.
Apple’s GSM-Symbolic project was designed to make that problem visible. Instead of relying on a fixed collection of grade-school mathematics questions, the researchers used symbolic templates to generate multiple versions of the same underlying problem. The approach allowed them to vary the surface details while preserving the mathematical structure. The models did not behave as a stable reasoner would be expected to behave. Their answers varied across different instantiations, and extra clauses could destabilise performance.
The result matters because benchmark familiarity is not the same as contamination in the narrow sense of a model having memorised a specific answer. A system may not have seen the exact question and still have learned the statistical shape of the problem. It may recognise the cadence of a word problem, the likely sequence of steps or the language associated with a particular answer. That can produce a correct response without the flexible grasp that allows a person to solve a new version.
This is why claims about “reasoning models” deserve more scrutiny than a final answer. A model may produce a long chain of intermediate text, but a chain of text is not automatically a chain of valid thought. It can be a record of plausible-looking steps, some correct and some invented, generated because that is what a successful solution usually sounds like.
Humans are not flawless reasoners. People also rely on shortcuts, recognise familiar forms and make mistakes when problems are poorly presented. But human competence has a feature current models often lack: a model of the situation outside the wording. A child who learns what a cup is can identify one made of glass, metal or plastic. A lawyer who understands a legal principle can apply it to a case with unfamiliar names. A mechanic can diagnose a fault even when the description is imprecise because the mechanic is reasoning about a machine, not merely matching a sentence to a stored response.
The distinction is especially important as AI systems move from answering questions to taking action. In a low-stakes setting, a fragile answer is an annoyance. In a financial, medical, legal or engineering setting, it becomes a liability. An agent that misreads a changed condition while retaining the same confidence can make a mistake at machine speed and repeat it across thousands of decisions.
The industry’s response has often been to add more computation: let the model generate a longer answer, check itself, ask another model to criticise it or run the same prompt several times. Those techniques can improve results. They can also create the appearance of reflection without establishing that the system has acquired a stable understanding. Repetition is not verification. A committee of statistical guesses is still a committee of statistical guesses.
The Old Lesson of ELIZA
The temptation to confuse convincing language with understanding is not new. In the mid-1960s, MIT computer scientist Joseph Weizenbaum built ELIZA, a programme that simulated a Rogerian psychotherapist through simple pattern-matching rules. It did not possess a theory of psychology, a memory of the patient’s life or an inner account of the distress described to it. Yet people often responded to its replies as if a mind were present.
MIT’s historical account describes ELIZA as a “pattern-matching conversational machine”.2 The programme could take a statement such as “I am unhappy” and turn it into a question about unhappiness. The effect was powerful because the setting did much of the work. A person expects a therapist to listen and ask questions. Once the conversation followed that script, the user supplied meaning that the software did not contain.
ELIZA was primitive. Today’s models are not. They can maintain context across long exchanges, combine information from many fields, produce coherent explanations and interact with tools. The comparison is not an argument that nothing has changed. It is a warning about what has not changed: fluency can cause people to attribute understanding where there is only an effective response mechanism.
In ELIZA’s case, the trick was visible once the programme’s rules were inspected. In large language models, the mechanism is buried beneath billions of learned parameters. That opacity makes the illusion more persuasive, not less. The machine can discuss its limitations, explain a decision and apologise for an error. None of those behaviours proves that it understands the words in the human sense. They show that it has learned how such language is used in conversations about knowledge and error.
The historical parallel also exposes a weakness in the way the Turing Test is often discussed. Alan Turing’s 1950 proposal was a brilliant attempt to turn an abstract question into an observable test: if an interrogator cannot reliably distinguish a machine from a person through conversation, what practical reason is there to deny the machine the label of intelligence? The test was never designed as a complete theory of mind. It was an operational thought experiment.
It becomes misleading when it is treated as a final scientific verdict. Passing a conversation test may show that a system can imitate human linguistic behaviour. It does not settle whether the system understands causes, maintains a grounded model of the world, forms intentions or knows when it is outside its competence. A stage magician can make an object appear to vanish. The performance is real; the disappearance is not.
The lesson of ELIZA is not that imitation has no value. Imitation can be useful, entertaining and commercially important. The lesson is that usefulness does not remove the need for accurate description. A navigation system does not need consciousness to guide a car. A translation system does not need a human biography to render a sentence. But a company that sells either system as a general reasoner invites customers to trust it beyond its evidence.
“Emergence” Often Means We Do Not Know
As models have grown larger, researchers have observed capabilities that were weak or absent in smaller systems. The language of “emergence” has become common: intelligence, planning, abstraction or reasoning is said to appear when enough parameters, data and computing power are combined.
Some of these changes are real. Larger systems can perform tasks that smaller systems cannot. But the word emergence can conceal a simple admission: the researchers do not yet know which internal mechanisms produced the change. It describes the timing of an observation more readily than it explains the cause.
That uncertainty is manageable in a narrow tool. If a system translates accurately, the user can judge the translation. If a programme sorts images correctly, the output can be tested. It becomes more serious when “emergence” is used to fill the gap between capability and intelligence. A model produces a correct answer, and its developers infer that a new form of thought must have appeared inside it. The inference may be right. The result alone does not establish it.
Interpretability research is an attempt to close that gap. Researchers are trying to identify the internal features and circuits associated with concepts, linguistic relationships and particular behaviours. The ambition is reasonable: instead of judging a model only by its outputs, inspect the machinery that generated them.
But interpretability is not a magic certificate of understanding. Finding a circuit associated with deception, a factual association or a sequence of planning steps would tell us something important about how the model operates. It would not, by itself, answer the philosophical question of whether the model’s internal representation means what the human concept means. A map of electrical activity is not automatically an account of experience. A description of a circuit is not automatically a theory of intelligence.
There is also a practical problem. The larger and more distributed a system becomes, the harder it is to assign a simple meaning to any one component. Features may be represented across many units. The same units may participate in different behaviours. A model may use one route in one context and another route in a slightly altered context. Even a successful explanation of one failure does not establish that the rest of the system is understood.
This does not make interpretability pointless. On the contrary, it is one of the few research programmes aimed at replacing confident storytelling with evidence about mechanisms. The mistake is to demand that it prove more than it can. Interpretability can reveal what a model is calculating. It may eventually help establish whether the calculation supports robust abstraction, causal reasoning and generalisation. Until then, “emergent intelligence” should be treated as a hypothesis, not a settled fact.
There is an irony in the argument. Human beings routinely infer minds from behaviour, because behaviour is the evidence available to us when dealing with other people. We cannot open another person’s consciousness and inspect it directly. But machines are designed artefacts. Their builders have a responsibility to show more than a convincing surface. The more authority they ask a system to exercise, the less acceptable it becomes to say that its internal logic is unknowable.
What a Better Machine Looks Like
The most serious criticism of large language models is not that they are incapable of reasoning in every sense. It is that they combine great strength in some forms of pattern recognition with serious weakness in the properties that make reasoning dependable.
One such property is compositional generalisation. A robust system should be able to combine known concepts into a new arrangement without needing to have seen that exact arrangement many times. It should understand that a red square is an instance of both redness and squareness, even if that specific phrase was absent from its training. It should be able to transfer a rule from one context to another while preserving the relationship that matters.
A second property is causal understanding. Correlation tells us what tends to occur together. Causation helps us reason about intervention: what would happen if a particular condition changed? A system that merely associates smoke with fire may answer questions about their co-occurrence. A system with a causal model can reason about smoke from a stage machine, a fire without smoke or a broken alarm that creates a false signal.
A third is robustness under distribution shift. The world does not present information in benchmark format. People use unfamiliar terminology, omit relevant details, mix reliable and unreliable evidence and change their minds. A reliable reasoner must identify which changes matter and which do not. It must not confuse a new surface form with a new underlying problem.
A fourth is efficient learning. Humans can acquire a useful concept from a small number of examples, a definition or a brief demonstration. Current models typically absorb broad capabilities from enormous quantities of data and then require further training or prompting to adapt. Scale can compensate for a lack of efficiency, but the compensation is expensive and fragile. A system that needs millions of examples to learn what a child can infer from one is not learning in the same way.
A fifth is honest uncertainty. Human experts do not become reliable by answering every question. They become reliable by knowing when evidence is weak, when a conclusion depends on an assumption and when a specialist is needed. Language models can be prompted to express uncertainty, but a cautious tone is not the same as calibrated knowledge. A system may say “I may be mistaken” and then provide a completely fabricated citation.
The legal case of Mata v. Avianca remains a compact illustration of the problem. Lawyers submitted nonexistent cases and quotations generated with ChatGPT, and a federal judge imposed sanctions. The court’s order made the point that a fake opinion is not existing law and that relying on it damages the legal process.3 The system’s language was not the issue. The issue was the absence of a dependable link between language and reality.
That link is the standard AI systems must meet if they are to be trusted in consequential work. They need not think like humans. They do need to preserve structure when circumstances change, distinguish evidence from invention and expose the basis on which a decision rests. A machine that cannot do those things may still be valuable. It should not be described as a general intelligence because it writes an excellent paragraph.
The Case for Hybrid Systems
One promising response is not to make language models carry every burden alone. Neural systems are good at recognising patterns, proposing possibilities and navigating messy information. Symbolic systems are good at applying explicit rules, checking consistency and producing proofs that can be inspected. A hybrid architecture gives each part a job that suits its strengths.
Google DeepMind’s AlphaGeometry offers a concrete example. The system combines a neural language model with a symbolic deduction engine. The neural component suggests useful geometric constructions; the formal engine tests those suggestions and carries out deductions according to defined rules. On a set of 30 olympiad geometry problems, AlphaGeometry solved 25, compared with 10 solved by the previous best computer method and an estimated 25.9 by an average human gold medallist on the same set.4
The result deserves respect, but it also deserves accurate description. AlphaGeometry is not evidence that an unrestricted chatbot has acquired human general intelligence. It is evidence that a system designed around complementary methods can solve a difficult class of problems and produce machine-verifiable proofs. That is a more modest claim, and in some ways a more impressive one.
The architecture resembles how skilled people often work. A mathematician may have an intuitive idea about which construction could unlock a proof, then use formal reasoning to test it. A scientist may generate a hypothesis and turn to instruments and calculations. An engineer may use experience to identify likely faults and then verify them against measurements. Creativity supplies candidates; rules and evidence decide which candidates survive.
Language models alone can imitate both stages in text, but imitation creates a dangerous ambiguity. When the proposed idea and the verification are generated by the same probabilistic process, there may be no independent check. A model can suggest a proof and then produce a plausible explanation of why the proof works, even when a formal verifier would reject it. The words can remain smooth throughout the failure.
Symbolic systems have limitations. They can be brittle, slow and difficult to build for domains whose rules are incomplete or contested. They do not automatically solve the problem of deciding which goals matter. A formal engine can prove that a statement follows from premises; it cannot decide whether the premises describe the real world or whether the question was worth asking.
That is why the most credible path is unlikely to be a simple victory for either neural networks or old-fashioned rules. The useful systems will probably combine learned perception with explicit memory, external tools, formal checks, causal models and mechanisms for asking for help. The industry’s obsession with a single model that can do everything may be less scientific than commercial. A collection of specialised components is harder to market as a magical mind, but easier to audit as a machine.
The Slop Problem Is a Judgment Problem
The debate about reasoning has a visible counterpart in the flood of AI-generated content. The internet is filling with articles, images, videos and marketing materials that are grammatically competent, technically formatted and almost entirely forgettable. The problem is not simply that machines can produce bad work. Humans produce bad work too. The problem is that generation has become cheap while selection remains difficult.
Graphite, a research-driven growth agency, reported that AI-generated articles surpassed human-written articles in publication volume in November 2024.5 The estimate concerns the web sample and methodology used by the researchers, not every piece of writing in existence. Even with that qualification, it captures a change that readers can recognise without a spreadsheet: more content is being made, and less of it appears to have been considered.
Generative systems tend to move towards likely continuations. For factual questions, that tendency can be useful when the likely answer is also correct. In design, writing and cultural work, likelihood is a poor substitute for judgment. The most probable colour scheme is often the one already used everywhere. The most probable opening paragraph sounds like an announcement. The most probable metaphor is familiar because it has been repeated.
This produces what critics call mode collapse: a narrowing of the range of outputs towards safe, average forms. Different models can converge on the same language because they have absorbed overlapping material and are rewarded for producing responses that broad groups of users find acceptable. A user asks for something distinctive and receives a polished version of the median.
The economic appeal is obvious. A company can generate ten thousand alternatives at little marginal cost. The bottleneck shifts to the people who must decide which alternative deserves to survive. If those people rank outputs without explaining their standards, the system learns that approval is the goal. It may become better at pleasing evaluators while becoming less capable of producing something original.
In subjective fields, the average opinion is not always the best signal. A film critic, a type designer, a novelist and a brand strategist may disagree about a visual or a sentence. Their disagreement can be informative because each sees a different dimension of the work. A system that averages them into one score may erase the very distinctions a creative product needs.
There is a broader cultural cost. If millions of people use tools that offer the same safe suggestions, those suggestions can shape what people consider normal, professional or beautiful. The tool does not merely reflect taste; it trains taste through repetition. A world saturated with plausible averages will not necessarily become more creative because it contains more output. It may become less able to recognise the difference between a finished thought and a template.
Taste Labs Wants to Sell the Missing Filter
That is the market Taste Labs is pursuing. Founded by Thais Castello Branco, the company emerged from stealth in June 2026 with an $18.5 million seed round co-led by Amplify Partners and CRV. Its stated mission is to end AI slop by building data and infrastructure that give models better judgment in subjective domains.6
The company’s premise is blunt: frontier models have become good at generating almost anything, but they remain poor at deciding what fits, what feels distinctive and what is worth keeping. “AI has nailed objective domains and made it easy to generate anything,” Castello Branco said at the launch. “But it still feels off. Now, the challenge is judgment.”
The business problem is real. Mathematics and code offer relatively clear feedback. A calculation can be checked. A programme can run or fail. A formal proof can be verified. A design, a piece of writing or a video does not arrive with an interpreter that declares whether it has achieved its purpose. Quality depends on audience, context, history and intention.
For Taste Labs, the answer is to make parts of taste measurable rather than pretending that the whole of taste can be reduced to one universal score. A brand may specify colours, fonts, spacing, motion and textures. A designer can then judge whether an output follows those rules while also asking whether it feels coherent. The first question is easier to verify than the second, but the two together are more useful than a vague instruction to “make it great”.
The company’s investor description says Taste is building annotation infrastructure for subjective tasks, not merely collecting more generic preferences. Its proposed network of tastemakers is meant to capture not only what an expert likes but who that expert is, what standards the expert applies and in which context the judgment is relevant.
That distinction matters. Asking a large group of people to choose between two designs can reveal a preference. It does not necessarily reveal expertise. An expert can identify why a composition fails, connect the criticism to a specific element and explain when a different choice would be appropriate. The explanation is not decorative. It is the data that turns a reaction into a trainable signal.
There is a danger in the language of “taste”, however. Taste can be a useful description of trained judgment, but it can also become a fashionable word used to hide authority. If a company says its experts know what is good, customers need to know who selected them, what communities they represent and how disagreement is handled. A system that replaces the average user with an unaccountable elite has not solved subjectivity. It has moved the decision behind a more expensive curtain.
Taste Labs therefore faces a test that is larger than its funding announcement. It must show that expert curation improves outputs outside the examples used to train the system, that the resulting models can preserve distinct points of view and that “better” does not simply mean more aligned with the preferences of the people who paid for the dataset.
Can Judgment Be Scaled Without Being Flattened?
The central difficulty is that subjective quality contains both shared standards and legitimate disagreement. A poorly aligned interface may be objectively confusing. A colour contrast may fail accessibility guidelines. A sentence may contain a factual error. But once those failures are removed, several good choices can remain, each suited to a different audience.
Traditional data operations are built to minimise disagreement. Annotators receive instructions, complete tasks and are measured against a reference answer or against one another. That approach is sensible for clear labels. It becomes counterproductive when variation is part of the signal. If a fashion editor, a children’s illustrator and a luxury-brand designer evaluate the same image differently, forcing them into consensus may produce an output that belongs to none of them.
The more useful model is not a single taste score but a map of preferences and principles. The system should be able to say that one design is strong for a particular brand, audience or cultural setting, while another is stronger for a different purpose. It should retain the reason for the judgment rather than merely recording the winner.
That requires careful expert selection. A network of people with strong portfolios is not automatically a network of good evaluators. Some creators are excellent at making work but poor at explaining why it succeeds. Others have forceful opinions that are really personal habits. The data must separate expertise from confidence and taste from status.
Quality control also has to distinguish useful disagreement from bad annotation. If experts disagree about whether a button is misaligned, the problem may be measurement or instruction. If they disagree about whether a particular composition feels elegant, the disagreement may be the point. The system must know which kind of disagreement it is seeing.
This is where Taste Labs’ proposed “measurability principle” becomes more than a slogan. The company argues that progress follows the ability to decompose a fuzzy goal into components that can be checked. In a design system, brand colours, typography and spacing can be specified. In a writing system, factual accuracy, audience, voice and structure can be evaluated separately. None of those measures captures the whole work. Together, they can prevent the model from failing in obvious ways before a human makes the final judgment.
The approach also changes what it means to improve an AI system. Improvement is not necessarily a higher score on a general benchmark. It may mean fewer generic outputs, fewer visual contradictions, stronger adherence to a user’s established style or better explanations of why one option was selected. Those outcomes are harder to advertise because they do not collapse into one number. They may be closer to what users actually want.
There is no guarantee that taste can be made into a product without becoming another form of conformity. But the current alternative is not neutral. Models already encode the preferences of their data, their annotators and their reward systems. The choice is not between influence and no influence. It is between influence that is examined and influence that is hidden behind the phrase “the model learned it”.
Scaling Is Not a Theory of Mind
The most expensive assumption in the industry is that more scale will eventually resolve every limitation. Bigger models have brought real gains. More data can improve coverage. More computing can allow a system to search longer or generate more candidates. But those facts do not prove that scale alone will create general intelligence.
The disagreement is often presented as a quarrel between optimists and pessimists. That is too simple. The real argument is about architecture. One side believes that sufficiently powerful prediction can develop the representations needed for reasoning. The other argues that prediction, even at extraordinary scale, does not automatically supply the machinery for logic, causation, durable memory or grounded action.
The evidence points in both directions. Large language models display abilities that would have seemed extraordinary a decade ago. They can summarise technical material, write software, translate between languages and assist with research. At the same time, they can fail on a minor change to a mathematical question, invent a source, reverse a constraint or lose track of a multi-step task. The correct description is not “stupid” and not “human”. It is broad competence with uneven foundations.
Investors are entitled to bet on a breakthrough. They are not entitled to treat a bet as a result. Billions of dollars have flowed into the assumption that the next increase in training scale will deliver systems capable of replacing broad categories of human work. If the assumption fails, the financial damage will not be limited to companies that overpromised. Schools, employers and public agencies may have redesigned decisions around capabilities that existed mainly on demonstrations.
There is a familiar historical pattern here. A new technology arrives, performs a striking set of tasks and attracts claims that its remaining limitations are temporary. When the limitations persist, funding and confidence retreat. The technology does not disappear; the language becomes less grand. The useful applications remain, while the promise of universal transformation is revised.
That revision would not be a defeat for AI. It would be a correction of category. A system can be valuable without being a mind. A model can accelerate a researcher, help a programmer explore alternatives or give a designer a fast starting point without understanding the world as a person does. The commercial opportunity may be stronger when products are built around what the systems reliably do rather than around what their fluency invites users to imagine.
The practical question is not whether a model can pass a test. It is whether the model remains dependable when the test is changed, the context is unfamiliar and the cost of error is real. Until the industry measures those conditions, the scaling curve remains a graph of output—not a proof of thought.
The Chinese Room Problem still exists
In spite of all the advances, the thought is that this is just a smarter scaled version of the Chinese Room thought experiment. The very limits of large language models echo John Searle’s 1980 Chinese Room thought experiment. Searle argued that a system can flawlessly manipulate formal symbols—syntax—without ever grasping their meaning, or semantics. An AI coding assistant, however fluent its output, remains inside that room: it rearranges tokens according to statistical regularities learned from vast corpora, yet possesses no genuine understanding of the domain, the design trade-offs, or the architectural intent. That is precisely why the five skills Pocock enumerates—grilling one’s own design concepts, enforcing ubiquitous language, practicing test-driven development, favoring deep modules, and retaining strategic oversight—cannot be delegated. They are the human acts that inject semantics into the syntax the machine so readily produces. Without them, teams risk multiplying not merely code, but code that looks correct while remaining, in the deepest sense, uncomprehended.
Yet the Chinese Room argument itself invites a further, more unsettling question. The human brain, after all, is no less a physical system than the hardware that runs a large language model; both are ultimately governed by the same laws of physics. Why, then, do we refuse to apply Searle’s critique to ourselves? The conventional reply points to embodiment, continuous sensorimotor feedback, and the evolutionary accumulation of meaning through lived experience—features that current LLMs conspicuously lack. In the software-engineering context that Pocock addresses, this distinction is not merely philosophical. It underscores why the five essential skills remain non-negotiable: only a system that is already outside the room—able to feel the force of a design constraint, to inhabit a domain’s ubiquitous language, to suffer the consequences of shallow modules—can inject the semantics that pure symbol manipulation cannot generate. Until artificial systems acquire something functionally equivalent to that lived grounding, the human developer’s role as the source of meaning, architecture, and strategic oversight is not optional; it is the indispensable bridge between physics and purpose.
The Standard the Industry Has Yet to Meet
A serious account of AI progress must hold two ideas together. The systems are powerful. The systems are not yet reliably intelligent in the broad human sense implied by the industry’s promises.
That position is not a compromise for its own sake. It follows from the evidence. Benchmark scores demonstrate capability within an evaluation. Apple’s research shows that mathematical performance can be sensitive to superficial changes. ELIZA shows that the appearance of conversation has long been able to provoke false confidence. AlphaGeometry shows that combining neural suggestions with symbolic verification can produce a system that is both capable and inspectable. The legal sanctions in Mata show what happens when fluent invention is mistaken for evidence.
Taste Labs’ bet adds another dimension. The next bottleneck may not be generation but selection. The world does not need an infinite number of plausible images, articles and interfaces. It needs systems that can distinguish the useful from the disposable, the coherent from the confused and the distinctive from the average. That requires a richer account of human judgment than a popularity poll or a single reward score.
The best path forward will therefore be less magical and more disciplined. It will use evaluations that resist memorization, tests that vary the surface while preserving the structure, tools that verify claims against external reality and architectures that separate proposal from proof. It will treat uncertainty as a feature to measure, not a tone to add after the answer. It will ask whose judgment is being encoded and what happens when reasonable experts disagree.
Most of all, it will stop using one flattering word to cover several different abilities. Pattern recognition is not the same as understanding. Fluency is not knowledge. A long chain of tokens is not a verified argument. A high benchmark score is not a general mind.
That may sound like an unnecessarily strict standard. It is not. The stricter the language used to describe a system, the stricter the evidence should be. If the industry wants machines trusted with law, medicine, infrastructure, education and culture, then “it usually sounds right” is not enough. The machine must show that it can carry meaning across changing circumstances, recognise the limits of its evidence and submit its most consequential claims to checks outside its own voice.
Until then, the most honest description of today’s leading systems is also the least glamorous: they are extraordinarily capable generators whose reliability depends on context, verification and human judgment. The next breakthrough will not be proved by a more impressive performance on a familiar test. It will be proved when the machine can face a new problem, explain what it knows, show how it knows it, and remain correct after the wording changes.
References
- Apple Machine Learning Research, “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models.” ↩
- MIT CSAIL, “Early Artificial Intelligence Projects,” section on ELIZA. ↩
- United States District Court for the Southern District of New York, Mata v. Avianca, Inc., Opinion and Order on Sanctions. ↩
- Google DeepMind, “AlphaGeometry: An Olympiad-level AI system for geometry.” ↩
- Graphite, “More Articles Are Now Created by AI Than Humans.” ↩
- Sarah Catanzaro, Amplify Partners, “Kill the Slop: announcing our investment in Taste.” ↩
This post contains affiliate links. If you purchase through these links, I may earn a commission at no extra cost to you.
Leave a Reply