Skip to content
Live
Trending storyDeveloping

Amazon SageMaker AI real-time endpoints support KV cache

2 reports1 outlets22 hours agoUpdate

The story so far

No overview yet — start with the coverage timeline below.

Developments

2 developments

  1. 18 Nov, 07:33 · 1 reports
    Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
    AWS Machine Learning Blog:Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
  2. 18 Nov, 05:11 · 1 reports
    Amazon SageMaker AI real-time endpoints now support KV cache
    AWS Machine Learning Blog:Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Coverage timeline

Follow the coverage to see every side of the story.

18 Nov
  1. AWS Machine Learning Blog
    Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

    Amazon SageMaker HyperPod Inference integrates NVIDIA Dynamo's distributed inference framework to address GPU memory shortages and low efficiency in processing long prompts in large language model inference

  2. AWS Machine Learning Blog
    Amazon SageMaker AI real-time endpoints now support KV cache aware routing

    Amazon SageMaker AI has launched KV cache-aware routing, which sends requests to instances that have cached relevant prompts and have moderate load, based on the cache status reported by the model container. It is suitable for applications of large language models (LLMs) that handle multi-round conversations, long shared prompts, or high load.

Heat over time

Now
5
Peak
1018 Nov, 08:00
24h change
–

Hover or tap the chart to see each hour; use the arrow keys on a keyboard.