NVIDIA shares 3 methods to improve MoE inference performance
The story so far
NVIDIA found through analysis that decoding is a memory constraint factor, and MoE computation and output layer projection increase execution time, sharing three methods to improve MoE inference performance. Inclusion AI evaluated Ling-3.0-flash with batch size 1 on NVIDIA DGX Spark and found that decoding is memory-constrained: reading model weights drove execution time, with MoE computation and output layer projection being bottlenecks. NVIDIA improved the inference throughput of the 124B parameter Ling-3.0-flash model on DGX Spark by 42.5%, a speed increase of 3.5x, with no change in accuracy, through a new MoE backend, FP8 output layer, and built-in speculative decoding. This optimization is suitable for single-user desktop systems, with throughput increasing from 37.81 tokens/s to 53.89 tokens/s and latency reducing by 24.5% in 1,319 GSM8K problem tests.
AI-written from the reports · Updated 15 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- MarkTechPostNVIDIA Shows 3 Ways to Speed Up MoE Inference on DGX Spark, Lifting Ling-3.0-flash Throughput 42.5%
NVIDIA boosts the inference throughput of the 124B parameter Ling-3.0-flash model on DGX Spark by 42.5% and speed by 3.5x with a new MoE backend, FP8 output layer, and built-in speculative decoding, without compromising accuracy. This optimization is suitable for single-user desktop systems, improving throughput from 37.81 tokens/s to 53.89 tokens/s and reducing latency by 24.5% on 1,319 GSM8K problem tests.
- NVIDIA Technical Blog3 Ways to Boost MoE Inference on NVIDIA DGX Spark
Inclusion AI found that decoding is the memory bottleneck when evaluating Ling-3.0-flash with a batch size of 1 on NVIDIA DGX Spark, where MoE computation and output layer projection increase execution time. MoE computation and output layer projection are bottlenecks.