Skip to content
Live
Trending storyPast story

AKV introduces caching project management for LLM intelligent entity key-value caching

2 reports2 outlets24 days agoUpdate

The story so far

AI overview

AKV manages LLM agent key-value cache through drop and repos operations, improving efficiency and accuracy. Experiments show that tool response rolling drop cache strategy increases average benchmark score by 1.7 points, reduces cost by 27.7%, maximizes output throughput by 1.2 times, and accelerates P95 request latency by 1.5 times. AKV provides code via GitHub repository https://github.com/thu-nics/AKV, claiming that introducing caching engineering management for LLM agent key-value cache does not alter trajectories but improves efficiency and accuracy. Researchers from Tsinghua University and HKU open-sourced AKV, offering KV cache engineering services for LLM agents.

AI-written from the reports · Updated 24 days ago

Developments

2 developments

  1. 26 Oct, 15:56 · 1 reports
    Tsinghua AI Open-Sources AKV
    MarkTechPost:Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang
  2. 22 Oct, 08:00 · 1 reports
    Cache Engineering for Agents
    Hugging Face Daily Papers:Cache Engineering for Agents

Coverage timeline

Follow the coverage to see every side of the story.

26 Oct
  1. MarkTechPost
    Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang

    Tsinghua University and HKU researchers open-sourced AKV, providing KV cache engineering services for LLM agents. By discarding selected cache entries and reusing the rest, an average cost reduction of 27.7% and an average task score improvement of 1.7 points were achieved in 3 models and 4 agent benchmark tests.

22 Oct
  1. Hugging Face Daily Papers
    Cache Engineering for Agents

    AKV proposes a caching engineering solution, managing LLM agent key-value caches through drop and repos operations, enhancing efficiency and accuracy. Experiments show that tool response rolling drop caching strategy improves the average benchmark score by 1.7 points, reduces costs by 27.7%, maximizes output throughput by 1.2 times, and accelerates P95 request latency by 1.5 times.