Amazon SageMaker HyperPod Enhanced Observability Feature Released
The story so far
Amazon SageMaker has released the enhanced observability feature for HyperPod, which compares the utilization, memory, temperature, power consumption, and SM clock frequency of each GPU in training jobs through the Amazon CloudWatch DetectGroupOutliers API, marks GPUs that deviate significantly from peers, and uses static diagnostic rules to help users quickly diagnose GPU training issues.
AI-written from the reports · Updated 7 hours ago
Coverage timeline
Follow the coverage to see every side of the story.
- AWS Machine Learning BlogIntroducing enhanced infrastructure observability in Amazon SageMaker HyperPod
Amazon SageMaker HyperPod has released enhanced observability features, comparing the utilization, memory, temperature, power consumption, and SM clock frequency of each GPU in training jobs through the Amazon CloudWatch DetectGroupOutliers API, marking GPUs that deviate significantly from peers, and using static diagnostic rules (such as GPU hardware failure, EFA transmission drop, thermal throttling, and overheating) to help users quickly diagnose GPU training issues.
Heat over time
- Now
- 8
- Peak
- 1020 Nov, 02:00
- 24h change
- –
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.