Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons
On this page
arXiv:2605.12049v2 Announce Type: replace-cross Abstract: Cortical neurons are complex, multi-timescale processors wired into recurrent circuits, shaped by long evolutionary pressure under stringent biological constraints. Mainstream machine learning, by contrast, predominantly builds models from extremely simple units, a default inherited from early neural-network theory. We treat this as a normative architectural question. How should one split a fixed parameter budget $P$ between the number of units $N$, per-unit effective complexity $k_e$, and per-unit connectivity $k_c$? What controls the optimal allocation? This calls for a model in which per-unit complexity can be tuned independently of width and connectivity. Accordingly, we introduce the ELM Network, whose recurrent layer is built from Expressive Leaky Memory (ELM) neurons, chosen to mirror functional components of cortical neurons. The architecture allows for individually adjusting $N$, $k_e$, and $k_c$ and trains stably across orders of magnitude in scale. We evaluate the model on two qualitatively different sequence benchmarks: the neuromorphic SHD-Adding task and Enwik8 character-level language modeling. Performance improves monotonically along each of the three axes individually. Under a fixed budget, a clear non-trivial optimum emerges in their tradeoff, and larger budgets favor both more numerous and more complex neurons. A closed-form information-theoretic proxy captures these tradeoffs and attributes the diminishing returns at two ends to: per-neuron signal-to-noise saturation and across-neuron redundancy. A hyperparameter sweep spanning three orders of magnitude in trainable parameters traces a near-Pareto-frontier scaling law consistent with the proxy. This suggests that the simple-unit default in ML is not obviously optimal once this tradeoff surface is probed, and offers a normative lens on cortex’s reliance on complex spatio-temporal integrators.
Why It Matters
This paper treats the simplicity of typical neural-network units as a normative architectural question, asking how a fixed parameter budget should be split between the number of units, per-unit effective complexity, and per-unit connectivity. It introduces the ELM Network, whose recurrent layer uses Expressive Leaky Memory neurons to mirror functional components of cortical neurons, allowing each factor to be tuned independently and trained stably across orders of magnitude. This matters because it connects scaling-law thinking to biological realism and offers a way to allocate parameters between width, complexity, and connectivity. It relates to deep-learning revolution, machine-learning algorithms, large language models, and training pipelines.
Sources
Back to Recent News · Sub-article index.