Tideway enhances consumer device MoE inference efficiency
The story so far
Tideway enhances the inference efficiency of large expert models (MoE) on consumer devices through a unified memory pool and optimization techniques. Tideway treats host memory and GPU GPU memory as a unified memory pool, employing phase-aware memory layout and CUDA graph optimization under expert paging, significantly improving performance. On typical consumer hardware, Tideway accelerates end-to-end latency by 1.97x, first token generation time by 2.40x, and decoding throughput by 1.61x. Its source code is open-sourced.
AI-written from the reports · Updated 13 hours ago
Coverage timeline
Follow the coverage to see every side of the story.
- Hugging Face Daily PapersTideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices
Tideway significantly improves the inference efficiency of large expert models (MoE) on consumer devices by treating host memory and GPU GPU memory as a unified memory pool and adopting stage-aware memory layout and CUDA graph optimization under expert paging. On typical consumer hardware, Tideway accelerates end-to-end latency by 1.97x, first token generation time by 2.40x, and decoding throughput by 1.61x. Its source code is open-sourced.
Heat over time
- Now
- 5
- Peak
- 719 Nov, 19:00
- 24h change
- –
Hover or tap the chart to see each hour; use the arrow keys on a keyboard.