Google Cloud MaxText supports long-context training on TPUs
The story so far
Google Cloud has released MaxText v0.2.4, supporting long-context training on TPUs. Through MaxText Context Parallelism technology, the Llama 3.1 8B model achieves 70% MFU on sequences of 4 million tokens on TPU 5p, and can seamlessly scale to 14 million tokens on TPU 5p and TPU 7. Context Parallelism in MaxText splits long sequences across multiple chips to avoid a single chip holding the activation information for the entire sequence, and this feature is now available in MaxText v0.2.4.
AI-written from the reports · Updated 9 days ago
Coverage timeline
Follow the coverage to see every side of the story.
- Google Developers BlogAnnouncing Long-Context Training on TPUs with MaxText Context Parallelism
Google Cloud releases MaxText v0.2.4, supporting long-context training on TPUs, achieving 70% MFU for sequences of 4 million tokens with the Llama 3.1 8B model on TPU 5p using MaxText Context Parallelism technology, seamlessly scalable to 14 million tokens on TPU 5p and TPU 7