Specialization, Diversity, and Scaling in Child-Scale Language Models
On this page
arXiv:2610.36679v1 Announce Type: cross Abstract: Modern language models are trained on far more language than children experience, but we know much less about learning from small, naturalistic datasets. We train small GPT-2 models on transcripts of speech with children from dense home recordings (BabyView) and on synthetic child-directed dialogue (TinyDialogues), varying the amount, source, and diversity of training data. Models trained on one child’s language environment predict that environment substantially better than other children’s environments. Much of this specialization is lexical and distributional: a simple 5-gram model reproduces the cross-child pattern (r=.91). Training on data from multiple children clearly improves prediction of pooled held-out language relative to training on a single child’s input without a similarly clear improvement on grammatical benchmarks. Across data sizes, naturalistic and synthetic training inputs also produce different grammatical and lexical-semantic patterns, with important differences between grammatical phenomena. Together, these results show that small-data language learning depends not only on how much language a model sees, but on which environments it comes from and how well that input matches what the model is asked to predict.
Why It Matters
This study asks whether small, naturalistic language input—the kind a child actually experiences—can support language learning in small models. Training GPT-2 on dense home recordings and synthetic child-directed dialogue shows that a model learns its own environment’s language far better than another child’s, and that data diversity, not just volume, changes grammatical and lexical-semantic outcomes. It matters because it challenges the assumption that language learning scales with sheer data size and points to environment-matched training as a design lever. See Large Language Models, Cognitive Science, and Training Pipelines.
Sources
Back to Recent News · Sub-article index.