Diagnose Distributed AI Workload Failures with NVIDIA Attribution
Diagnose Distributed AI Workload Failures with NVIDIA Attribution
NVIDIA Attribution can help users locate the initial failure point in distributed AI workloads by analyzing NCCL errors, scheduler state changes, and downstream message cascading symptoms, revealing the source of anomalies rather than just reporting the stop location. This tool is suitable for diagnosing cross-node issues caused by GPU or network fabric events, by visualizing and tracing the root cause of problems through thousands of message links.
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state...
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began. In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence…
Source: NVIDIA Technical Blog · developer.nvidia.com