使用 NVIDIA Attribution 诊断分布式 AI 工作负载故障
Diagnose Distributed AI Workload Failures with NVIDIA Attribution
NVIDIA Attribution 可帮助用户定位分布式 AI 工作负载中的初始故障点，通过分析 NCCL 错误、调度器状态变化和下游消息等级联症状，揭示异常发生源头而非仅报告停止位置。该工具适用于诊断 GPU 或网络 fabric 事件引发的跨节点问题，通过可视化分析数千条消息链路追溯问题根源。
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state...
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began. In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence…
来源: NVIDIA Technical Blog · developer.nvidia.com