Rongchai Wang
Feb 23, 2026 18:39
NVIDIA’s NVFP4 4-bit training format achieves 59% faster AI model training than BF16 while matching accuracy on Llama 3 8B benchmarks, per new research.
NVIDIA’s NVFP4 low-precision training format delivers up to 1.59x faster throughput compared to standard BF16 training while maintaining equivalent model accuracy, according to new benchmarks published by the company’s research team on February 23, 2026.
The results mark a significant milestone for 4-bit AI training, demonstrating that aggressive numerical compression doesn’t require sacrificing model quality when proper techniques are applied.
The Numbers That Matter
Testing on Llama 3 8B models trained across 1 trillion tokens, NVIDIA’s team measured throughput at 1,850 TFLOP/s per GPU with NVFP4 versus 1,165 TFLOP/s for BF16 baseline—a 59% improvement. The tests ran on GB200 NVL72 hardware using the company’s Blackwell architecture.
Downstream benchmark scores tell the real story. On MMLU, NVFP4-trained Llama 3 8B scored 45.64% compared to 45.98% for BF16. HellaSwag showed 75.59% versus 76.44%. These differences fall within noise margins for practical applications.
Memory efficiency gains enabled doubling the micro-batch size from 2 to 4 during pretraining, directly improving scalability for large-scale training runs.
Why 4-Bit Training Works Now
Previous attempts at ultra-low-precision training often resulted in model divergence or significant accuracy degradation. NVIDIA’s approach sidesteps these issues through a specific recipe that’s emerged from extensive testing.
The critical insight: keeping approximately 15% of the network in higher precision prevents training collapse. Specifically, the final four transformer layers must remain in BF16. Ablation studies confirmed that fully NVFP4 models diverge during training.
The format uses a two-level scaling strategy—micro-block scaling for groups of 16 elements combined with global FP32 scaling across full tensors. This hierarchical approach manages the limited dynamic range inherent in 4-bit representations.
Random Hadamard transforms smooth tensor spectrums and reduce outliers that would otherwise cause training instability. Stochastic rounding for gradients eliminates systematic quantization bias.
Comparison With Other Low-Precision Formats
NVFP4 isn’t the only option. FP8 with current scaling (FP8-CS) achieved 1.33x speedup over BF16, while MXFP8—a block-level scaling variant optimized for Blackwell—hit 1.32x. Both formats showed slightly better convergence tracking than NVFP4 during training, though final accuracy metrics remained comparable across all approaches.
MXFP8 demonstrated marginally better performance than standard FP8, likely due to finer-grained scaling that better captures local dynamic range within tensors.
Production Deployment
The techniques are available now through NeMo Megatron Bridge, NVIDIA’s open PyTorch-native library. Switching between precision formats requires changing a single configuration flag—no model code or optimizer logic modifications needed.
For teams running large-scale training workloads on Blackwell hardware, the throughput gains translate directly to reduced training time and compute costs. A model that previously required 10 days of training could potentially complete in under 7 days with NVFP4.
The recommended recipe for NVFP4: AdamW optimizer with epsilon=1e-8, learning rate decaying from 6e-4 to 6e-6, and global batch size of 768. These parameters represent the empirical sweet spot from NVIDIA’s extensive testing across multiple architectures and datasets.
Image source: Shutterstock
Credit: Source link
