Skip to content
Live
Trending storyPast story

NVIDIA's full-stack collaborative design improves AI memory efficiency

2 reports2 outlets13 days agoUpdate

The story so far

AI overview

NVIDIA enhances AI memory efficiency through full-stack collaborative design, optimizing memory usage across the GPU, CPU, network, storage, and software layers. The Blackwell Ultra GPU increases HBM capacity by approximately 3%, while the SemiAnalysis AgentX reduces agent inference cost by 30 times. NVIDIA proposes 'full-stack collaborative design' to break through AI memory bottlenecks. NVFP4 compresses model weights and KV cache, NVL72 distributes weights across 72 GPUs, Vera CPU provides buffering, Dynamo software migrates context, and RAG indices remain on the CPU and storage side. As large model parameter scale, context length, and agent applications grow rapidly, GPU memory is facing unprecedented pressure. HBM supply is tight and prices are rising, and this trend is expected to continue in the foreseeable future.

AI-written from the reports · Updated 12 days ago

Developments

2 developments

  1. 6 Nov, 07:30 · 1 reports
    NVIDIA breaks through AI memory bottleneck
    ITHome · AI:NVIDIA breaks through AI memory bottlenecks: HBM only increased by about 3%, with the maximum improvement in agent throughput per dollar up to 30 times.
  2. 6 Nov, 00:00 · 1 reports
    Full-Stack Codesign Solves AI’s Memory Challenge
    NVIDIA Newsroom:More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

Coverage timeline

Follow the coverage to see every side of the story.

6 Nov
  1. ITHome · AI
    NVIDIA breaks through AI memory bottlenecks: HBM only increased by about 3%, with the maximum improvement in agent throughput per dollar up to 30 times.

    NVIDIA proposed 'Full-Stack Co-Design' to break through AI memory bottlenecks. NVFP4 compresses model weights and KV cache, NVL72 distributes weights across 72 GPUs, Vera CPU provides buffering, Dynamo software migrates context, and RAG indices remain on the CPU and storage side.

  2. NVIDIA Newsroom
    More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

    NVIDIA enhances memory efficiency through full-stack co-design, optimizing memory usage across layers of GPU, CPU, network, storage, and software. The Blackwell Ultra GPU increases HBM capacity by about 3% per GPU, while the SemiAnalysis AgentX reduces agent inference cost by 30x.