Llama 3.1 405B Achieves 1.5x Throughput Boost with NVIDIA H200 GPUs and NVLink

The rapid evolution of large language models (LLMs) continues to drive innovation in artificial intelligence, with NVIDIA at the forefront. Recent developments have seen a significant 1.5x increase in the throughput of the Llama 3.1 405B model, facilitated by NVIDIA’s H200 Tensor Core GPUs and the NVLink Switch, according to the NVIDIA Technical Blog.

Advancements in Parallelism Techniques

The enhancements are primarily attributed to optimized parallelism techniques, including tensor and pipeline parallelism. These methods allow multiple GPUs to work in unison, sharing computational tasks efficiently. Tensor parallelism focuses on reducing latency by distributing model layers across GPUs, while pipeline parallelism enhances throughput by minimizing overhead and leveraging the NVLink Switch’s high bandwidth.

In practical terms, these upgrades have resulted in a 1.5x improvement in throughput for throughput-sensitive scenarios on the NVIDIA HGX H200 system. This system utilizes NVLink and NVSwitch to facilitate robust GPU-to-GPU interconnectivity, ensuring maximum performance during inference tasks.

Comparative Performance Insights

Performance comparisons reveal that while tensor parallelism excels in reducing latency, pipeline parallelism significantly boosts throughput. For instance, in minimum latency scenarios, tensor parallelism outperforms pipeline parallelism by 5.6 times. Conversely, in maximum throughput scenarios, pipeline parallelism delivers a 1.5x increase in efficiency, highlighting its capacity to handle high-bandwidth communication effectively.

These findings are supported by recent benchmarks, including a 1.2x speedup in the MLPerf Inference v4.1 Llama 2 70B benchmark, achieved through software improvements in TensorRT-LLM with NVSwitch. Such advancements underscore the potential of combining parallelism techniques to optimize AI inference performance.

NVLink’s Role in Maximizing Performance

NVLink Switch plays a crucial role in these performance gains. Each NVIDIA Hopper architecture GPU is equipped with NVLinks that provide substantial bandwidth, facilitating high-speed data transfer between stages during pipeline parallel execution. This capability ensures that communication overhead is minimized, allowing throughput to scale effectively with additional GPUs.

The strategic use of NVLink and NVSwitch enables developers to tailor parallelism configurations to specific deployment needs, balancing compute and capacity to achieve desired performance outcomes. This flexibility is essential for LLM service operators aiming to maximize throughput within fixed latency constraints.

Future Prospects and Continuous Optimization

Looking ahead, NVIDIA’s platform continues to advance with a comprehensive technology stack designed to optimize AI inference. The integration of NVIDIA Hopper architecture GPUs, NVLink, and TensorRT-LLM software offers developers unparalleled tools to enhance LLM performance and reduce total cost of ownership.

As NVIDIA persists in refining these technologies, the potential for AI innovation expands, promising further breakthroughs in generative AI capabilities. Future updates will delve deeper into optimizing latency thresholds and GPU configurations, leveraging NVSwitch to enhance online scenario performance.

Image source: Shutterstock