Skip to content
Live
Trending storyDeveloping

Google DeepMind introduces Streaming DiLoCo algorithm

1 reports1 outletsyesterdayUpdate

The story so far

AI overview

Google DeepMind and Google Research launched the Streaming DiLoCo algorithm, implementing cross-city distributed LLM training through the MaxText framework. The algorithm splits training into M independent computational groups, each performing H steps of local optimization before inter-group communication, significantly reducing the impact of cross-regional network transmission latency. In the MaxText framework, global optimization of weight differences is synchronized every H steps, while Streaming DiLoCo further splits model parameters into P slices, synchronizing one slice every Δh steps to smooth network load, reducing communication volume by 1-2 orders of magnitude while maintaining the H-step synchronization cycle.

AI-written from the reports · Updated 5 hours ago

Coverage timeline

Follow the coverage to see every side of the story.

19 Nov
  1. Google Developers Blog
    Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText

    The DiLoCo algorithm splits training into M independent computational groups, each performing H steps of local optimization before inter-group communication, significantly reducing the impact of cross-regional network transmission latency. In the MaxText framework, it achieves global optimization of weight differences synchronously every H steps. Streaming DiLoCo further splits the model parameters into P pieces, synchronizing one piece every Δh steps to smooth network load, while maintaining an H-step synchronization cycle and reducing communication volume by 1-2 orders of magnitude.

Heat over time

Now
5
Peak
620 Nov, 03:00
24h change
–

Hover or tap the chart to see each hour; use the arrow keys on a keyboard.