Skip to content
Live
Trending storyPast story

NVIDIA has launched automated workload recovery functionality

2 reports1 outlets16 days agoUpdate

The story so far

AI overview

NVIDIA has launched an automated workload recovery feature to address failures in distributed AI training, ensuring stable operation. When training ranking stagnates, GPU nodes stop responding to the scheduler, or slow nodes appear, it can trigger NVIDIA Collective Communications Library (NCCL) timeouts, requiring fast and reliable recovery. As distributed AI training expands, failures in hardware and software dependencies increasingly interrupt training processes, making fast and reliable recovery crucial. NVIDIA Attribution can help users locate the initial failure points in distributed AI workloads by analyzing NCCL errors, scheduler state changes, and downstream message cascading symptoms, revealing the source of anomalies rather than just reporting the stop location.

AI-written from the reports · Updated 16 days ago

Developments

2 developments

  1. 3 Nov, 07:39 · 1 reports
    NVIDIA Attribution
    NVIDIA Technical Blog:Diagnose Distributed AI Workload Failures with NVIDIA Attribution
  2. 3 Nov, 06:43 · 1 reports
    Automated Workload Recovery
    NVIDIA Technical Blog:Keep Distributed Training Running with Automated Workload Recovery

Coverage timeline

Follow the coverage to see every side of the story.

3 Nov
  1. NVIDIA Technical Blog
    Diagnose Distributed AI Workload Failures with NVIDIA Attribution

    NVIDIA Attribution can help users locate the initial failure point in distributed AI workloads by analyzing NCCL errors, scheduler state changes, and downstream message cascading symptoms, revealing the source of anomalies rather than just reporting the stop location. This tool is suitable for diagnosing cross-node issues caused by GPU or network fabric events, by visualizing and tracing the root cause of problems through thousands of message links.

  2. NVIDIA Technical Blog
    Keep Distributed Training Running with Automated Workload Recovery

    NVIDIA introduces automated workload recovery, providing fast and reliable recovery from failures such as NCCL timeouts, GPU node interruptions, or slow nodes, to ensure stable distributed AI training.