Skip to content
Live
Trending storyDeveloping

Tideway enhances consumer device MoE inference efficiency

1 reports1 outletsyesterdayUpdate

The story so far

AI overview

Tideway enhances the inference efficiency of large expert models (MoE) on consumer devices through a unified memory pool and optimization techniques. Tideway treats host memory and GPU GPU memory as a unified memory pool, employing phase-aware memory layout and CUDA graph optimization under expert paging, significantly improving performance. On typical consumer hardware, Tideway accelerates end-to-end latency by 1.97x, first token generation time by 2.40x, and decoding throughput by 1.61x. Its source code is open-sourced.

AI-written from the reports · Updated 13 hours ago

Coverage timeline

Follow the coverage to see every side of the story.

19 Nov
  1. Hugging Face Daily Papers
    Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices

    Tideway significantly improves the inference efficiency of large expert models (MoE) on consumer devices by treating host memory and GPU GPU memory as a unified memory pool and adopting stage-aware memory layout and CUDA graph optimization under expert paging. On typical consumer hardware, Tideway accelerates end-to-end latency by 1.97x, first token generation time by 2.40x, and decoding throughput by 1.61x. Its source code is open-sourced.

Heat over time

Now
5
Peak
719 Nov, 19:00
24h change
–

Hover or tap the chart to see each hour; use the arrow keys on a keyboard.