Live
Amazon SageMaker AI real-time endpoints support KV cache
Trending storyDeveloping
Amazon SageMaker AI real-time endpoints support KV cache
2 reports1 outlets22 hours agoUpdate
The story so far
No overview yet — start with the coverage timeline below.
Developments
2 developments
- 18 Nov, 07:33 · 1 reportsManaged NVIDIA Dynamo on Amazon SageMaker HyperPod InferenceAWS Machine Learning Blog:Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
- 18 Nov, 05:11 · 1 reportsAmazon SageMaker AI real-time endpoints now support KV cacheAWS Machine Learning Blog:Amazon SageMaker AI real-time endpoints now support KV cache aware routing
Coverage timeline
Follow the coverage to see every side of the story.
18 Nov
- AWS Machine Learning BlogManaged NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Amazon SageMaker HyperPod Inference integrates NVIDIA Dynamo's distributed inference framework to address GPU memory shortages and low efficiency in processing long prompts in large language model inference
- AWS Machine Learning BlogAmazon SageMaker AI real-time endpoints now support KV cache aware routing
Amazon SageMaker AI has launched KV cache-aware routing, which sends requests to instances that have cached relevant prompts and have moderate load, based on the cache status reported by the model container. It is suitable for applications of large language models (LLMs) that handle multi-round conversations, long shared prompts, or high load.
Heat over time
- Now
- 5
- Peak
- 1018 Nov, 08:00
- 24h change
- –
02.557.510
Nov,Nov,Nov,
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.