Amazon SageMaker HyperPod detects silent data corruption
The story so far
Amazon SageMaker HyperPod detects silent data corruption (SDC) by adding three checks in the PyTorch DDP training loop, including non-finite gradient checks, loop replay audits, and weight fingerprint comparisons. When issues are detected, suspicious nodes are isolated and replaced by HyperPod, having been tested and verified on a 4 ml.p5.48xlarge instance cluster.
AI-written from the reports · Updated 3 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- AWS Machine Learning BlogHow does Amazon SageMaker HyperPod detect and recover silent data corruption?
Amazon SageMaker HyperPod detects silent data corruption (SDC) by adding three checks in the PyTorch DDP training loop, including non-finite gradient checks, loop replay audits, and weight fingerprint comparisons. When issues are detected, suspicious nodes are isolated and replaced by HyperPod, having been tested and verified on a 4 ml.p5.48xlarge instance cluster.
Heat over time
- Now
- 3
- Peak
- 1012 Nov, 08:00
- 24h change
- -51%
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.