# From SGD to Muon: Adaptive Optimization via Schatten-p Norms

arXiv:2605.19781v3 Announce Type: replace Abstract: Modern optimizers, like Muon, impose matrix-wise geometry constraints on their updates. These matrix-wise constraints can be unified under Linear Minimization Oracle (LMO) theory. However, all current methods impose fixed LMO geometries for the update rules, chosen by-design or empirically, which are not necessarily optimal according to the problem's geometry. We introduce a novel efficient datadriven criterion for dynamically choosing proxy-optimal update LMO geometries on individual Deep Neural Network layers. Derived in closed form from gradient and activation statistics using a single-step random feature regression surrogate model, our criterion navigates a design space interpolating from SGD to Muon updates. Moreover, integrating parameter-wise preconditioning allows our framework to recover SGD, Muon, Adam, and MuAdam as specific extrema. To make this adaptive approach scalable, we pair it with efficient computational strategies, achieving only a $\sim$ 3% runtime overhead on highly optimized baselines. As a proof of concept, we show that this data-driven optimizer beats or remains competitive with the performance of the best performing optimizer between Muon and AdamW across three different training scenarios. Ultimately, this work provides evidence that LMO geometry can be successfully and efficiently adapted from runtime data, opening a new pathway for optimizer design beyond static geometries.

## Why It Matters

Modern optimizers like Muon impose matrix-wise geometry constraints on parameter updates. This work generalises that insight through Schatten-p norms, showing how adaptive optimization can be derived from spectral geometry rather than element-wise scaling. It connects optimizer design to the algorithms that underpin model training, with implications for pretraining efficiency. See [Algorithms Underlying AI](../ml-algorithms.md), [Training Pipelines](../training-pipelines.md), and [Large Language Models](../large-language-models.md).

## Sources

- rss: https://arxiv.org/abs/2605.19781

*Back to [Recent News](../../news.md) · [Sub-article index](../index.md).*