AKV introduces caching project management for LLM intelligent entity key-value caching
The story so far
AKV manages LLM agent key-value cache through drop and repos operations, improving efficiency and accuracy. Experiments show that tool response rolling drop cache strategy increases average benchmark score by 1.7 points, reduces cost by 27.7%, maximizes output throughput by 1.2 times, and accelerates P95 request latency by 1.5 times. AKV provides code via GitHub repository https://github.com/thu-nics/AKV, claiming that introducing caching engineering management for LLM agent key-value cache does not alter trajectories but improves efficiency and accuracy. Researchers from Tsinghua University and HKU open-sourced AKV, offering KV cache engineering services for LLM agents.
AI-written from the reports · Updated 24 days ago
Developments
2 developments
- 26 Oct, 15:56 · 1 reportsTsinghua AI Open-Sources AKVMarkTechPost:Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang
- 22 Oct, 08:00 · 1 reportsCache Engineering for AgentsHugging Face Daily Papers:Cache Engineering for Agents
Coverage timeline
Follow the coverage to see every side of the story.
- MarkTechPostTsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang
Tsinghua University and HKU researchers open-sourced AKV, providing KV cache engineering services for LLM agents. By discarding selected cache entries and reusing the rest, an average cost reduction of 27.7% and an average task score improvement of 1.7 points were achieved in 3 models and 4 agent benchmark tests.
- Hugging Face Daily PapersCache Engineering for Agents
AKV proposes a caching engineering solution, managing LLM agent key-value caches through drop and repos operations, enhancing efficiency and accuracy. Experiments show that tool response rolling drop caching strategy improves the average benchmark score by 1.7 points, reduces costs by 27.7%, maximizes output throughput by 1.2 times, and accelerates P95 request latency by 1.5 times.