Skip to content
Live
Trending storyPast story

Google Cloud MaxText supports long-context training on TPUs

1 reports1 outlets10 days agoUpdate

The story so far

AI overview

Google Cloud has released MaxText v0.2.4, supporting long-context training on TPUs. Through MaxText Context Parallelism technology, the Llama 3.1 8B model achieves 70% MFU on sequences of 4 million tokens on TPU 5p, and can seamlessly scale to 14 million tokens on TPU 5p and TPU 7. Context Parallelism in MaxText splits long sequences across multiple chips to avoid a single chip holding the activation information for the entire sequence, and this feature is now available in MaxText v0.2.4.

AI-written from the reports · Updated 9 days ago

Coverage timeline

Follow the coverage to see every side of the story.

9 Nov
  1. Google Developers Blog
    Announcing Long-Context Training on TPUs with MaxText Context Parallelism

    Google Cloud releases MaxText v0.2.4, supporting long-context training on TPUs, achieving 70% MFU for sequences of 4 million tokens with the Llama 3.1 8B model on TPU 5p using MaxText Context Parallelism technology, seamlessly scalable to 14 million tokens on TPU 5p and TPU 7