LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks
On this page
arXiv:2609.34187v4 Announce Type: replace-cross Abstract: The strong version of the stochastic parrot argument claims that, although large language models (LLMs) may exceed rote regurgitation, they cannot move beyond statistical pattern matching into abstraction or reasoning, remaining ontologically near the lower bound of pattern reuse despite producing alluringly fluent text. We test this hypothesis using conlang-like tasks. Several LLMs are given only natural-language descriptions of fictional languages that subvert prominent superficial patterns in training data by combining statistically uncommon and unattested features. Crucially, no example outputs are given. We argue that if the models exhibit rule-following behaviour, they cannot be relying solely on superficial statistical patterns; such patterns often work against the correct output. Instead, successful performance requires representations of the constraints specified in the prompt. Across three complementary task families, models systematically move in the meaning-predicted direction: they distinguish prompt exposure from instructed use, alter semantic relationships in response to novel constraints, and sometimes produce exact matches to complex translation answer keys. Although performance varies across the spectrum of models used, these results provide evidence for meaning-mediated abstraction in LLMs and refute the strong stochastic parrot hypothesis. Our work shows that, under appropriate architectural and contextual constraints, statistical learning can produce meaning-mediated abstractions, although generation remains strongly constrained by superficial plausibility. We discuss implications for model development and for understanding how increasingly abstract representations may emerge from plausible-text-generation objectives.
Why It Matters
The “stochastic parrot” argument claims LLMs cannot move beyond statistical pattern matching into abstraction or reasoning. This paper presents evidence from conlang-like tasks that models exhibit meaning-mediated abstraction, directly challenging that strong claim. It bears on whether LLMs are genuinely reasoning or only regurgitating, a debate central to how we interpret and trust their capabilities. See Large Language Models, Knowledge, Reasoning, and Semantics, and AI Phenomena and Public Discourse.
Sources
Back to Recent News · Sub-article index.