# Training an MoE LLM in MXFP4: How We Closed the Gap to BF16

Training large language models (LLMs) in sub-8-bit arithmetic promises substantial gains in memory bandwidth and compute throughput, yet FP4 formats introduce quantization error severe enough to destabilize optimization if left unaddressed. We present ALTO, an open-source training recipe for GPT-OSS-20B, a 20-billion-parameter mixture-of-experts (MoE) model, on the MLPerf Small MoE Training benchmark under the MXFP4 microscaling format. The recipe combines established techniques from Abecassis, Felix et al. [arXiv:2509.25149] (hybrid 1D/2D block quantization, randomized Hadamard transforms, and stochastic rounding) with a simplified weight de-oscillation scheme adapted from TetraJet-v2 [arXiv:2510.27527]. On C4 validation at global batch size 16, MXFP4 + RHT + SR + de-oscillation reaches a validation loss of 3.3350 at 16,128 steps, only 0.007 above the BF16 baseline (3.3283), closing roughly 50% of the gap left by MXFP4 without de-oscillation (3.3418). Measured by steps to reach the MLPerf quality target (validation loss 3.34), de-oscillation cuts the FP4 convergence overhead relative to BF16 from +20.0% to +5.0%. We also report negative end-to-end results from differential gradient estimation, outlier clipping, and macro-block scaling, despite each showing operator-level SNR gains. All experiments use simulated MXFP4 kernels on AMD MI300X hardware and wall-clock speed has not been optimized.

## Why It Matters

This work advances the encyclopedia's coverage of [Agents, Autonomy, and Tool Use](../agents-autonomy.md), [Training Pipelines](../training-pipelines.md), [Algorithms Underlying AI](../ml-algorithms.md), [Hardware Foundations](../hardware-foundations.md), [Emergent Abilities and Benchmark Evolution](../emergent-abilities.md), [Machine-Learning Algorithms](../ml-algorithms.md). The central idea—Training large language models (LLMs) in sub-8-bit arithmetic promises substantial gains in memory bandwidth and compute throughput, yet FP4 formats introduce q—connects to the broader arc of recent AI research tracked in [Recent News](../../news.md).

## Sources

- rss: https://arxiv.org/abs/2610.32229

*Back to [Recent News](../../news.md) · [Sub-article index](../index.md).*