Google announced TPU Sync

Google has released the open-source C++ library TPU Sync, designed to move KV cache and model weights on Cloud TPUs, aiming to solve the bottleneck of model state migration. The library treats TPU HBM, host DRAM, shared storage, and data center network as a unified memory hierarchy and includes a global KV store for prefix lookup, a transfer controller, and a KV cache manager. Google claims that the library reduces KV transfer latency by over 90% for MoE models with parameters ranging from 235B to 480B. The TPU Sync library is released under the Apache-2.0 license.

AI-written from the reports · Updated yesterday