Google DeepMind introduces Streaming DiLoCo algorithm
The story so far
Google DeepMind and Google Research launched the Streaming DiLoCo algorithm, implementing cross-city distributed LLM training through the MaxText framework. The algorithm splits training into M independent computational groups, each performing H steps of local optimization before inter-group communication, significantly reducing the impact of cross-regional network transmission latency. In the MaxText framework, global optimization of weight differences is synchronized every H steps, while Streaming DiLoCo further splits model parameters into P slices, synchronizing one slice every Δh steps to smooth network load, reducing communication volume by 1-2 orders of magnitude while maintaining the H-step synchronization cycle.
AI-written from the reports · Updated 5 hours ago
Coverage timeline
Follow the coverage to see every side of the story.
- Google Developers BlogDistributed LLM Training Across Metros: Streaming DiLoCo in MaxText
The DiLoCo algorithm splits training into M independent computational groups, each performing H steps of local optimization before inter-group communication, significantly reducing the impact of cross-regional network transmission latency. In the MaxText framework, it achieves global optimization of weight differences synchronously every H steps. Streaming DiLoCo further splits the model parameters into P pieces, synchronizing one piece every Δh steps to smooth network load, while maintaining an H-step synchronization cycle and reducing communication volume by 1-2 orders of magnitude.
Heat over time
- Now
- 5
- Peak
- 620 Nov, 03:00
- 24h change
- –
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.