Skip to content
Live
vLLM releases· khluu·· 21 hours ago0

v0.34.0

v0.34.0

Brief overview

GitHub Copilot v0.34.0 released, providing code generation, review, security, and collaboration features, supporting custom integrations and advanced security features.

Article

v0.34.0

1,061 commits from 412 contributors (120 new). Thank you!

TL;DR

  • 10 new models (Mistral Large 4, MiniMax-M3.1-Flash, Nemotron-3.5-Transcribe, and more!)
  • Extensible KV cache is on by default: vLLM sizes the KV cache from memory actually free after warmup, so most models get a bigger cache and more throughput.
  • Model Runner V1 is gone. Model Runner V2 is now the only GPU runner.
  • Big speedups on Blackwell and MI355X: up to 1.95× decode (Step5 on GB200), 1.6× throughput (GLM-5.3-Flash), 1.24–1.40× faster long-prompt TTFT (GLM-5.3).
  • Upgrade check: default block size is now 64, ROCm 7.2 images are gone, and huggingface_hub >= 2.0.0 is required. See Before you upgrade.

Before you upgrade

Important

These changes can break existing setups.

Change What to do PR
Model Runner V1 removed Remove VLLM_USE_V2_MODEL_RUNNER=0; it now fails at startup. Port V1 logits processors to vllm.v1.worker.gpu.sample.logits_processor. #63373
Default --block-size is 64 (was 16) Pass --block-size 16 to keep the old behavior. ROCm paged attention still uses 16. #63231
Extensible KV cache on by default; gpu_memory_utilization defaults to 1.0 To keep the old sizing, pass --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache. #56492
ROCm 7.2 artifacts dropped Use the ROCm 10.0 wheels and images. -rocm72 is no longer published. #61939
huggingface_hub >= 2.0.0 required Also requires tokenizers >= 0.23.3 and timm >= 1.0.30. #64779
QuTLASS / fp_quant removed fp_quant checkpoints no longer load. #65230
Marlin W4A8 removed Use --quantization humming with VLLM_HUMMING_INPUT_QUANT_CONFIG instead of VLLM_MARLIN_INPUT_DTYPE. #65542
Keye-VL and Keye-VL-1.5 removed — #64287

All breaking changes, deprecations and new defaults are listed in the full changelog.

New models

Model Notes PR
Mistral Large 4 1T total / 52B active, FP8 and NVFP4. Runs on NVIDIA Blackwell/Hopper and AMD MI350. #64534
MiniMax-M3.1-Flash Blackwell NVFP4 and ROCm kernels, DSpark draft #64111
NVIDIA Nemotron-3.5-Transcribe 30B-A4B speech model: audio chat and /v1/audio/transcriptions #64282
Ricursive Ripple-65B Preview BF16 and NVFP4, ripple reasoning and tool parsers #65745
Zyphra ZAYA1 ZAYA1-8B, ZAYA1-74B-preview #65302
Baidu Unlimited-OCR-2 DASH sparse attention #64016
Liquid AI LFM3, LFM3-VL, LFM3-Audio #65826, #65827, #65828
NVIDIA Canary-1B-v2 ASR and X↔En translation #64215
Ming3-ASR #63585
NVIDIA Parakeet TDT Through the vllm-parakeet-plugin #65005

Highlights

Extensible KV cache, on by default

  • On CUDA and ROCm, the KV cache is sized from memory actually free after CUDA graph capture and warmup, not from an estimate (#56492).
  • Result: a larger cache (e.g. MiniMax-M3 on H200: 433k → 672k tokens), so throughput is higher and TTFT lower at high concurrency.
  • Trade-off: per-token decode latency can rise on some models (TPOT +6–18% on some H200 models at concurrency 128). Opt out with --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache.
  • Hardened with fixes for allocator fragmentation, startup OOMs and KV connectors (#67296, #67485, #66731, #66523, #66197, #65137, #66605, #66674).

Model Runner V2 only

  • Model Runner V1 is removed (#63373).
  • Dual-batch overlap now runs on MRV2, including with speculative decoding and DCP (#61464).

Speculative decoding

  • Suffix decoding runs on MRV2 with a GPU speculator; Arctic Inference is no longer needed (#60339).
  • Custom speculators: declarative configs plus a register_speculative_method plugin API (#63205).
  • CUDA graphs for MRV2 draft-model speculators (#60989).
  • Adaptive verification works with XPress (#65175).
  • Draft KV cache matches the target's dtype, so an FP8 target gets an FP8 draft cache (#65198).

Performance

Improvement Hardware Gain PR
Step5 NVFP4 MoE on TRTLLM-Gen 4×GB200 up to 1.95× decode #65152
GLM-5.3-Flash DeepGEMM FP8 MegaMoE NVIDIA 1.6× output throughput vs DEP4 #60895
Muse Spark Open specialized decode (opt-in) GB300 1.5× single-stream decode #64077
Token-sharded sparse-MLA prefill under TP (default) GB200 GLM-5.3 TTFT 1.24–1.40× faster at 32K–128K #63956
KV-balanced MLA decode for Kimi-K3 (opt-in) GB300 1.34–1.59× decode attention #63638
Qwen3.5 on breakable CUDA graphs (no torch.compile) NVIDIA 3.0–8.8% lower decode ITL #51012
FlyDSL decode monokernel, Muse Spark Open DSpark 8×MI355X +35–57% interactivity at conc. 1–8 #63440

AMD ROCm

  • Release images and wheels are ROCm 10.0 only (#61939).
  • MoRI EPv2 via --all2all-backend mori (#61361).
  • Kimi-K3 specialized decode on MI355X, opt-in (#61435).
  • TRITON_ATTN is the default attention backend on RDNA3 and RDNA4 (#60418).

Serving and operations

  • Checkpoint/restore (CRIU + cuda-checkpoint) through /checkpoint/prepare, /checkpoint/status and /checkpoint/restore, in both the Python and Rust frontends (#62570).
  • Fault-tolerant DP+EP scale-down: dead ranks are removed and their experts redistributed without a restart (#46370).
  • Best-effort NUMA binding on by default (#61258).
  • Opt-in cache-aware admission ordering (#54625).
  • Rust frontend: stateless Responses API (#63902) and n > 1 (#65625).

Install

Python wheels

Platform Install
CUDA 13.0 (PyPI) pip install vllm
CUDA 13.0 (uv) uv pip install vllm --torch-backend=auto
ROCm 10.0 pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.34.0/rocm100
XPU uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.34.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match

Docker images

Platform Image Short alias
CUDA 13.0 (default) vllm/vllm-openai:v0.34.0 vllm/vllm:v0.34.0
CUDA 13.0 + DeepEP over AWS EFA vllm/vllm-openai:v0.34.0-efa vllm/vllm:v0.34.0-efa
ROCm 10.0 vllm/vllm-openai-rocm:v0.34.0 vllm/vllm:v0.34.0-rocm
CPU vllm/vllm-openai-cpu:v0.34.0 vllm/vllm:v0.34.0-cpu
XPU vllm/vllm-openai-xpu:v0.34.0 vllm/vllm:v0.34.0-xpu

latest tags work the same way (e.g. vllm/vllm:latest-rocm). The default CUDA image now includes the AWS EFA userspace (libfabric, aws-ofi-nccl). The -efa image adds DeepEP built for EFA and routes NCCL over EFA by default (#65576).

The Assets section below has the source tarball and the CUDA 13.0 (x86_64, arm64), CPU (x86_64, arm64, macOS) and XPU (x86_64) wheels.

Full changelog

Model support

  • Mistral Large 4 (#64534) runs on MRV2 with:
    • sparse MLA plus the DSA indexer, and cross-layer KV sharing;
    • DSpark, DCP/PCP, NIXL P/D and HiSparse.
  • Other new models:
    • MiniMax-M3.1-Flash (#64111), plus P/D across different prefill and decode TP (#64475).
    • Nemotron-3.5-Transcribe with MTP and DFlash2 (#64282, #64944).
  • DeepSeek-V4 / V4.1
    • Prefill gathers compressed KV once per KV-source group: 38 → 4 gathers per step (#59131).
    • Decoder replay graphs share the model graph pool: prefill graph memory 15.52 → 9.88 GiB per rank on GB200 (#63155).
    • fused_inv_rope_fp8_quant 3.5× faster, prefill 6% faster (#63540).
    • DSpark/MTP memory savings of about 512 MiB per GPU (#63996).
    • Engram host tables shared across DP ranks (#62359).
    • DeepGEMM sparse MQA logits on the FP8 indexer cache (#62011).
    • Packed multi-image vision encoding, 1.6–2.6× (#65030).
    • LoRA on the language model, compressor and wo_a layers (#57245, #64880).
    • DSpark with PCP (#61994); PCP with PP or DP (#63098).
    • Fixes: sparse prefill in FULL CUDA graphs (#59986), prefill memory budgeting (#63047), MTP embedding under PP (#65816), DeepSeek-V3.2 PP (#59961).
  • GLM
    • GLM-5.3-Flash: DeepGEMM FP8 MegaMoE (#60895), adaptive verification (#60229), row-strided QKV into FlashKDA (#61614).
    • GLM-5.3 FP8 MoE sorted-A TMA on H200, 6.0–7.6% E2E throughput (#64760).
    • FP8 KV cache for DSpark and DFlash drafts (#63841, #64612, #65553).
    • FP32 indexer MQA logits restored for GLM-5/GLM-5.3/HY-V4, with new --attention-config.indexer_logits_dtype (#64580).
    • Fused KDA convs fix stale weights after RL refit (#60991).
  • Muse Spark
    • Opt-in Muse Spark Open specialized decode on GB300 (#64077).
    • One RoPE cache shared across layers: 11.5 GiB per GPU saved, 14.5% more KV cache on 8×B200 (#65206).
    • GEMM and communication+norm fusions (#65438, #64638); hybrid DeepGEMM MegaMoE / TRT-LLM for NVFP4 (#64525).
    • CUDA routing kernel (#64553), MegaMoE JIT warmup (#63920), DSpark draft loading fix (#65125).
  • MiniMax-M3 / M3.1
    • Sequence parallelism and GEMM-RS for TP + MegaMoE, 14–20% lower P50 TTFT (#64659), also with PP (#65537).
    • Fused norm + MXFP8 quant, 1.39–1.60× (#64852).
    • Triton query-tiled sparse prefill on Hopper (#57420); SM90 split-K indexer (#64037).
    • No host sync in prefill metadata (#65883); NVFP4 MegaMoE warmup (#66227); fmha_sm100 indexer updates (#66228).
    • Fixes: M3.1 prefill workspace (#64856), sequence parallelism (#65496), encoder CUDA graphs (#65226).
  • Kimi
    • Opt-in --attention-backend KIMI_K3_KV_BALANCED_MLA for agentic workloads on Blackwell (#63638).
    • Kimi-K3 MegaMoE symmetric buffer 10.73 → 2.50 GiB per GPU at TP8 EP (#63541).
    • FlashInfer fused FP8 chunked-context K/V pack (#61527).
    • Fixes: KDA prefill OOM (#63989), NaNs via FlashKDA/FlashMLA pins (#64759), sleep mode (#64583, #66273).
    • Kimi K3.1 groundwork landed (#66030, #66028), but K3.1 is not servable in this release.
  • Qwen
    • Qwen3.5: breakable CUDA graphs instead of torch.compile (#51012); fused norm/gate + quant (#64509) and AR + norm + quant (#64723); GDN in_proj overlap (#64425, #64999).
    • GDN prefill state gather/scatter fused for Qwen3-Next/Qwen3.5 and Kimi KDA, 1.16–1.50× (#66064).
    • Qwen3.8-Flash-Next: PLE disk offload via --engram-config '{"disk_offload": true}' (27 GiB less on DGX Spark, #59958), PP (#60234), shared QSA top-k buffer (#65619), loading fixes (#62318, #65812, #64820, #60490).
    • Fixes: long-context int32 overflow (#63761), multi-layer MTP (#62111).
  • Other models
    • Step5 MegaMoE with PP (#65504) and KV block sizing (#65153).
    • Gemma 4: multimodal encoder DP (#60314), PP (#65989), FA4 FP8 MMA on SM90 (#60372).
    • Moved to the Transformers modeling backend: OLMoE (#63384), Gemma v1 and StableLM (#66393), Aria (#59767; temporarily about 11% less KV cache and 25% lower image throughput).
    • Matryoshka dimensions for EmbeddingGemma 2 (#64885, #60978); RADIO C-RADIO v2/v3/v4 (#58468).
    • Qwen3-ASR on the upstream Transformers config; no --hf-overrides needed for the ForceAligner (#66097, #63274).
    • Fixes: GPT-OSS MXFP4 LoRA (#65825), PLaMo3 GPTQ (#62836), Ling3 (#65088), DomynEdge (#66819).
  • Multimodal
    • Encoder CUDA graphs for Pixtral/Mistral3 (#56277), shared capture buffers (#65084), one replay per batch up to encoder_cudagraph_max_token_budget (#65227).
    • On-device input normalization for Nemotron VL (#57924), Gemma 3 (#64487), Gemma 4 (#65577).
    • --mm-encoder-tp-mode data for MiMo-V2.6 (#64179).
    • Fixes: FlashInfer ViT attention on GLM-4.1V/GLM-OCR/GLM-5.3-Flash (#65778), Qwen3-Omni and Qwen2.5-Omni video budgets (#66017, #66190), dots_ocr with TP (#64543), embedding-only startup (#65689), media cache eviction (#65363).

Engine core

  • Extensible KV cache
    • On by default on CUDA and ROCm (#56492).
    • Headroom for measured allocator fragmentation, validated on Qwen3.5-397B-A17B-FP8 at TP8+EP on 8×H200 (#67296).
    • Fixes: sizing and startup OOMs (#67485, #66523, #66731, #66197, #65554), connectors that failed to start (#65225), up to 84% undersized caches on DGX Spark (#65413), --max-model-len auto (#65463).
    • Startup warning when an explicit gpu_memory_utilization withholds memory from it (#66704); docs on when to set it (#67071).
  • Model Runner V2
    • The only runner after MRV1's removal (#63373); DBO with spec decode and DCP (#61464).
    • Separate prefill (token) and decode (request) CUDA graph capture sizes (#63406).
    • Staged writes copied straight into UVA (#63633); idle sampler writes skipped, about 10–11% faster small-batch decode (#62748).
    • Sampler warmed up after graph capture to stop memory growth under load (#62560).
    • Lower peak memory for chunked prompt logprobs (#64841, #64914); Triton ReplaySSM (#65033).
    • NVIDIA Confidential Computing decode fixes (#59981).
  • Speculative decoding
    • Suffix decoding (#60339) and custom speculators (#63205) on MRV2; draft-model CUDA graphs (#60989).
    • XPress with adaptive verification (#65175); adaptive verification draft budgets on a Triton kernel (#63421).
    • Batch-sharded draft logits (#63554); DSpark sequential blocks (#63886).
    • DSpark sliding-window MLA draft layers (#63795) under DCP (#63796); fused Markov sampling, 42–50% less sampling time (#64476); draft_lm_head_quantization (#64368).
    • Draft KV cache dtype aligned with the target (#65198).
    • Trace replay with ngram/EAGLE3/MTP, 1.47–2.03× (#63275); 3× faster resample kernel (#65100).
    • Fixes: DFlash acceptance of 0 with FULL CUDA graphs (#64059), a resample race (#66048), acceptance-estimator cancellation (#65229), standalone draft prefill under FULL graphs (#65804), grammar-invalid drafts (#61557).
  • Scheduler and engine overhead
    • Prompt token ids kept as int32 buffers: 40× less per-request CPU on a 128K prompt (#62702).
    • Per-request id lists kept out of the cyclic GC (#65300); 45–50% less KV block allocation CPU (#65831).
    • New requests and aborts handled while waiting for model output (#65466), with a cheaper wait loop (#65773).
    • Opt-in cache-aware admission via --cache-aware-admission-window (#54625).
    • --max-num-inflight-kv-loads starts async KV loads while the batch is full (#65319), with cached reservations: 47.2 → 5.4 ms per schedule (#65285).
    • A multimodal item is no longer split by a prefix hit (#65271).
  • Startup and compile
    • No openai/fastapi imports in EngineCore and workers (#65423, #65589).
    • CUDA graph memory profile skipped when its estimate is disabled (#64622).
    • Device-neutral compile cache via VLLM_COMPILE_ON_ONE_RANK, on by default (#63247); all graph-affecting config fields hashed into compile cache keys (#65652).
    • --shuffle-safetensors-files for large EP startups (#42837).
    • BasevLLMParameter's Python __torch_function__ dropped: GLM-5.2-FP8 TTFT −6.1% on GB300 (#63785).
  • Lifecycle
    • Process checkpoint/restore over HTTP (#62570); opt-in multi-worker CUDA checkpoint sleep (#64186).
    • Best-effort NUMA binding by default; --no-numa-bind to disable (#61258).
    • Sleep mode with exact-size pinned backups, up to about 578 GiB less host memory per node on Mistral Large 4 TP8 (#64630); level 2 discards only parameters (#66010, #66165, #59994).
    • Fast Start weight-cache restarts for online FP8/MXFP8, ModelOpt CUTLASS FP8/NVFP4/block-FP8 and AWQ/GPTQ Marlin (#63672, #63754, #64119, #64182), with pluggable shared warmup-cache stores (#65168).
    • WatermarkConfig.enforce (#62966).
  • RL and weight loading
    • VLLM_RELOAD_MODE=direct writes checkpoint tensors straight into the live model (#64336, #64352, #65281).
    • Every load and update finishes through finalize_model_weights (#65914, #64357, #64692, #57678).
    • Load plans for weight-transfer engines (#65235); per-update NCCL source ranks for PP trainers (#62557).
    • Sparse weight patches validated in one pass, 40–48% shorter pause (#66021).
    • Offloaded KV tier survives pause/sleep (#57810); partial stacked weights rejected (#63752).
  • LoRA
    • MoE-LoRA shrink/expand kernels up to 64 tokens (#64653); embedding LoRA scaled by embed_scale (#62470).
    • Rejected: adapters on unwrapped modules (#63489), non-linear expert placement with EP (#62797).
    • Fixes: reloads with requests in flight (#64616), KV-sharing fast prefill (#61162), SEQ_CLS heads (#63286), fused MoE expand rounding (#63894), Mamba-1 (#64681).
  • Batch invariance: LFM2 ShortConv (#62367), FlashInfer FP8 bmm and W4A16 NVFP4 (#62192), GPTQ (#60479), FA sliding window (#65627), aten::mm out_dtype overloads (#60047).

Large-scale serving

  • Expert parallelism and fault tolerance
    • DP+EP scale-down via POST /v1/fault_tolerance/apply (#46370); MoRI EPv2 (#61361).
    • Opt-in symm_mem EPLB weight-migration backend (#62009).
    • --enable-ep-weight-filter with EPLB: DeepSeek-R1 NVFP4 DEP8 reads 105 GiB per rank instead of 385 GiB (#64187).
    • num_redundant_experts defaults to the minimum valid value (#30614).
    • Skipped combine staging copy in flashinfer_nvlink_one_sided (#65498); DeepEP V2 padded rows (#54109).
  • Context parallelism and sparse attention
    • Token-sharded sparse-MLA prefill under TP (#63956); DCP on the FlashMLA sparse bf16 KV path (#61609).
    • MLA decode up-projected before the A2A merge with --dcp-comm-backend a2a (#64994); faster DCP sparse-indexer top-k (#65178, #64129).
    • Cross-layer KV cache sharing for MLA layers (#64095).
    • Fixes: PP rank derivation under PCP (#65539), FlashInfer MLA DCP decode (#59392).
  • KV connectors
    • NIXL: push mode via transfer_mode (#64230); ucx_backend_params (rocm_ipc on MI355X: 3.4 → 483.7 Gbps, #62905); 38% fewer hybrid SSM transfer bytes (#62317); block pairing across different P/D block sizes (#59746), with mismatched push handshakes rejected (#59747); same-host NVLink with the extensible KV cache (#65137).
    • Mooncake: selective leases (#63420), automatic cache_prefix (#65160), per-GPU RDMA devices (#65076), heterogeneous-TP hybrid sharing (#54307), sleep mode (#59934), lease renewal (#64193).
    • Lookahead block hashes in NIXL, Mooncake and offloading connectors (#64096); UMBP standalone mode (#61795, #61793).
    • Out-of-tree and LMCache MP connectors with the extensible KV cache (#66605, #66674).
  • KV offload
  • HiSparse
    • Uses the extensible KV cache by default: 16% more GPU+host KV capacity (#65214).
    • Fused slot invalidation, up to 1.8% lower TPOT on GLM-5.3-NVFP4 (#60103); fewer hash probes (#61263); speculative-group prefetch (#63801).
    • Platform backend for hardware plugins (#62592); scheduler deadlock fix (#62979).
  • EPD and distributed
    • Metadata-only image and video inputs for GLM-5.3-Flash and DeepSeek-V4.1 (#64071, #62802).
    • Bitwise-identical 1-stage and 2-stage custom all-reduce (#57042); FlashInfer all-reduce gated on 16-byte rows (#63123).
    • Cross-node MNNVL groups kept off CUDA IPC all-reduce (#55835).

Hardware

  • NVIDIA
    • Step5 NVFP4 TRTLLM-Gen MoE (#65152).
    • DeepSeek-V4.1: latency_specialized decode extended, 5.9–17% faster decode step on GB300 (#63921); swap-AB CuTe DSL decode attention for TP2/TP4, 1.4–2.15× (#62633); opt-in fused DSpark Markov sampling, 1.6–3.9× (#64442).
    • Sparse-indexer top-k reads BF16 logits directly (#62172, #64138); single-launch fused prefill top-k (#60618).
    • GLM-5.3 TMA MoE projections on H200 and SM100 (#63820, #65746); SM90 native small-head sparse FlashMLA, +10% on GLM-5.3-Flash TP4 (#64510).
    • CuTe DSL BF16x3 router GEMM for fp32 routers on Hopper (#61903); fused GDN MTP decode for DSpark K=8, +18% on B300 (#65874).
    • GLM-5.3-Flash on 2× DGX Spark, 15% lower TPOT (#62956); faster weight loading on integrated GPUs (#58868).
    • Rubin locality-domain NVFP4 MoE (#61685); Jetson Thor (SM110) fixes (#62133, #65050, #62275).
  • AMD ROCm
    • ROCm 10.0 only (#61939); AITER v0.1.28.post1 (#63774, #65937); MoRI EPv2 (#61361).
    • MonoKernel decode for Muse Spark Open DSpark verify (#63440), DeepSeek-V4.1-Flash at TP8 (#64101) and the MiniMax-M3.1 DSpark drafter, +10% P90 interactivity (#65386).
    • Kimi-K3 specialized decode (#61435), AITER-tuned prefill GEMMs: mean TTFT 5.35 → 4.05 s (#63730), opt-in FlyDSL FP8 MLA prefill (#59124).
    • DeepSeek-V4-Pro P/D over MoRIIO (#48989); Step-5-Preview (#61837) and Reflection Beam 1.0 (#62057) on ROCm.
    • RDNA: TRITON_ATTN by default (#60418), AITER Triton kernels on RDNA3 (#52970), HIP GDN prefill backend for gfx1151 (#56307), opt-in W4A16 dense prefill cache: TTFT −22 to −26% on Strix Halo (#60710).
    • Fixes: Kimi-K2.6 int4 prefill regression on Triton 3.8 (#62565), MI355 extensible KV cache OOMs (#65418).
  • Intel XPU
    • PyTorch 2.15 (#63977) and vllm_xpu_kernels 0.1.17 with new MoE/MLA interfaces (#60609).
    • Per-token dynamic FP8 activations (#61362); faster batched Triton MoE (#51605); MRoPE kernel (#59938); GDN metadata reuse (#64479); DeepSeek-V4 adaptive verification (#62309).
  • CPU
    • MRV2 on CPU backends without Triton: Arm, IBM Z, PowerPC, RISC-V (#61995).
    • zentorch 2.14.0.2 (#65669) and Laguna MoE on Zen (#63711).
    • Faster GDN spec-decode updates (#64829); Gemma 4 MoE gelu_tanh (#61593); bidirectional encoder-only attention (#63738).
  • Other: out-of-tree layer overrides via vllm.model_executor.hw_agnostic.resolve (#60218); shared routed-expert row restorers (#64689); Ascend Triton fix (#64013).

Quantization

  • New --quantization lut_b_moe: about 3.14-bit MoE experts with MXFP8 activations, native on Rubin (#62759).
  • w4afp8 checkpoints on the W4A8 FP8 MoE path (#64410).
  • Block Hadamard input rotation with online mxfp4_hadamard32 and Quark online-rotation checkpoints (#62750).
  • CSF-compressed NVFP4 block scales in ModelOpt (#64312); activation quant overrides for online MXFP4 (#51808).
  • DeepSeek-V4.1 MXFP8 linears on cuBLASLt (#64815); 3× faster MXFP4 activation quant (#63101); faster FP8 quant on SM100+ (#61641).
  • QuIP NVFP4 on Humming (#65230).
  • Fixes: NVFP4 NaN logits (#63382, #61755), Humming FP8 (#60163, #56606), hadacore (#63514).

API and frontend

  • New endpoints and options
    • /checkpoint/prepare, /checkpoint/status, /checkpoint/restore (#62570).
    • thinking_token_budget on /v1/responses (#63896); /v1/completions in vllm run-batch (#64097).
    • Streaming derender on /inference/v1/generate (#63103).
    • Anthropic Messages API: min_tokens (#65459) and thinking.type=between_tools (#65954).
    • vllm:request_render_duration_seconds histogram (#60213).
  • Tool calling and reasoning
  • Behavior
    • Corrupt or oversized images and disabled local media return 400 instead of 500 (#63433, #60308, #64032).
    • Responses accepts incomplete assistant/reasoning history, e.g. from Codex (#35508).
    • Multi-template processors use the default template (#63606); an empty tools array is accepted like OpenAI (#65547).
    • The first streamed usage reports cached prompt tokens (#65036); plain completion stream frames render about 9× faster (#52875).
  • Rust frontend

Security, dependencies and build

  • Security
    • ReDoS in the validation-error sanitizer fixed (#52120).
    • FunAudioChat audio bounded to the profiled duration (#57411).
    • /v1/systemone honors --trust-request-chat-template (#64503).
    • Raised floors for filelock, OpenTelemetry, graphql-core and others (#63574).
  • Dependencies
    • PyTorch stays at 2.14.0. FlashInfer 0.7.3 (#65930).
    • huggingface_hub >= 2.0.0, tokenizers >= 0.23.3, timm >= 1.0.30 (#64779); transformers >= 5.16.1, < 5.23.0 (#64049, #64529); mistral-common >= 1.13.0 (#64169).
    • DeepEP 2.1 (#59844); DeepGEMM pin updates (#64411, #64580, #65494); FlashMLA and FlashKDA pins (#61971, #64759).
    • NIXL 1.5.2 (#64579, #64585); mooncake-transfer-engine 0.3.14 (#65576); Run:ai Model Streamer 0.17.1 (#63174); fastokens >= 0.3.5 for VLLM_USE_FASTOKENS (#65679).
    • ROCm: AITER v0.1.27 → v0.1.28.post1 (#63774, #65937). XPU: PyTorch 2.15 (#63977), triton-xpu 3.9.0 (#65047), vllm_xpu_kernels 0.1.17 (#60609). CPU: zentorch 2.14.0.2 (#65669).
  • Build
    • x86_64 CUDA wheel about 82 MB (about 25%) smaller, kernels unchanged (#65518).
    • Default CUDA image installs the AWS EFA userspace; the -efa image now only adds DeepEP for EFA (#65576); nightly EFA images as vllm/vllm-openai:efa-nightly (#64028).
    • One ROCm build and release pipeline, ROCm 10.0 only (#61939).
    • Source builds fetch triton_kernels from a tarball (#65162) and ship tuned kernel configs in the wheel (#64063).
    • CUDA x.yf arches no longer fall back to a lower minor (#66016).

All breaking changes, deprecations and new defaults

  • Model Runner V1 removed (#63373).
    • VLLM_USE_V2_MODEL_RUNNER and VllmConfig.use_v2_model_runner are gone; setting the env var to anything but 1 fails at startup.
    • Dropped with it: the V1 logits-processor API (vllm.v1.worker.gpu.sample.logits_processor is the only interface), the custom_class speculative method, use_heterogeneous_vocab, disable_padded_drafter_batch, parallel_drafting outside DFlash/DSpark, PP with external_launcher, NGramPerReqLogitsProcessor, and x86 CPUs without Triton.
    • Deprecated: parallel_drafting, capture_torch_profiler, enable_layerwise_nvtx_tracing.
  • Default --block-size is 64 instead of 16 (#63231). Backends with a preferred block size still override it (ROCm paged attention keeps 16), and an explicit --block-size still wins.
  • gpu_memory_utilization defaults to 1.0 when unset with the extensible KV cache on (#56492). An explicit value in (0.8, 1.0) that withholds 1 GiB or more logs a warning (#66704).
  • ROCm 7.2 wheels and -rocm72 images are no longer published (#61939).
  • Removed:
    • QuTLASS and the fp_quant method (#65230).
    • Marlin INT8/FP8-activation (W4A8) paths and VLLM_MARLIN_INPUT_DTYPE; use --quantization humming with VLLM_HUMMING_INPUT_QUANT_CONFIG (#65542).
    • Keye-VL and Keye-VL-1.5 (#64287); disable_mtp_completeness_check (#49459); the refresh_derived_weights hook (#65914).
  • Dependencies: huggingface_hub >= 2.0.0 (#64779); mistral-common >= 1.13.0 (#64169); XPU requires vllm_xpu_kernels 0.1.17 (#60609).
  • Model and plugin code: all process_weights_after_loading hooks share one signature and the loader runs every module's hook (#57678); config compute_hash() replaced by compile_factors() / hash_config() (#65652).
  • API behavior:
    • llama3_json no longer streams tool arguments incrementally (#63569).
    • Canary transcription without language returns 400 (#65445).
    • An empty tools array is accepted in both frontends (#65547, #65854).
    • /v1/systemone rejects request-supplied chat templates without --trust-request-chat-template (#64503).
    • LoRA adapters on modules vLLM does not wrap are refused (#63489); misconfigured NCCL weight transfer raises (#57022).
    • vllm-rs serve no longer reads --data-parallel-rank from VLLM_DP_RANK (#65369).
    • AITER fused shared experts with MiniMax M3 and DeepSeek V4 need VLLM_ROCM_USE_AITER=1 (#61457).
  • Still deprecated: --mamba-cache-mode has no effect; its removal was postponed (#64838).
  • New defaults:
    • Extensible KV cache (#56492); block size 64 (#63231); best-effort NUMA binding (#61258).
    • Token-sharded sparse-MLA prefill at 1024 or more prefill tokens (#63956); MLA decode up-projection before the A2A merge (#64994).
    • VLLM_COMPILE_ON_ONE_RANK (#63247); minimum valid EPLB num_redundant_experts (#30614).
    • HiSparse on the extensible KV cache (#65214); encoder CUDA graphs up to encoder_cudagraph_max_token_budget=5120, eager above (#65227).
    • TRITON_ATTN on RDNA3/RDNA4 (#60418); MRV2 on non-Triton CPUs (#61995); float32 Mamba SSM cache for Nemotron Omni (#63218); Zipf routing in vllm bench forward (#64167).

Contributors

120 new contributors: welcome!

All 412 contributors

@BugenZhao, @Thangnguyenvn98, @AndreasKaratzas, @njhill, @hmellor, @khluu, @aoshen02, @Juntian777, @mgoin, @NickLucche, @gau-nernst, @okorzh-amd, @yewentao256, @vadiklyutiy, @yma11, @zyongye, @Pb314314, @jeejeelee, @WoosukKwon, @fxmarty-amd, @waizuichougou, @stefankoncarevic, @xsxszab, @djramic, @ZJY0516, @vMaroon, @MatthewBonanni, @Fangzhou-Ai, @liranschour, @TheEpicDolphin, @chaojun-zhang, @gcanlin, @whx-sjtu, @Alex-Wengg, @roberteg16, @subhashpolisetti, @hshrivastava-droid, @hao-aaron, @mkhazraee, @divakar-amd, @Isotr0py, @seungrokj, @jperezdealgaba, @aarushjain29, @vllmellm, @bzsuni, @liusy58, @zhenwei-intel, @Etelis, @rasmith, @Ankur-singh, @reidliu41, @jzakrzew, @ZhenchengLin, @charxwu, @xinli-sw, @subin9, @namgyu-yun, @frida-andersson, @dilberx, @josh-schell-amd, @simon-veitner-redhat, @akii96, @cjackal, @chaunceyjiang, @wzhao18, @chunfangamd, @robertgshaw2-redhat, @cpersson-amd, @CJJ1008, @yzong-rh, @zhenwenqi2024, @johncalesp, @SherifWaly, @gc-fu, @Vegetog, @Manfredss, @sohom-cs, @albertoperdomo2, @Dao007forever, @Alex-ai-future, @taneem-ibrahim, @gty111, @bogdan-petkovic, @jiuyueQ, @noooop, @lucamotz, @RyanMa29, @paulpak58, @ilmarkov, @azaidy, @yifanFengg, @mispa-ms, @jh-nv, @xqoasis, @Prakash-sa, @juliendenize, @galletas1712, @mjkvaak-amd, @louie-tsai, @wtdcode, @nrenvo, @hyukjlee, @srkim-business, @hemuku90, @amenomori, @nsusanto, @zohaibm-amd, @grYe99, @LopezCastroRoberto, @janbernloehr, @RuibinCheung, @yaoyaoding, @DongjiGao, @luyixiao95, @ayush1399, @qli88, @qianlihuang, @roikoren755, @simondanielsson, @faaany, @inc-jeong, @bohnstingl, @fadara01, @spandantiwari, @BaoYunkai, @erezzarum, @mrn3088, @indivats, @LucasWilkinson, @Change72, @esmeetu, @franciscojavierarceo, @hickeyma, @zydo, @roborluo, @fuscof-ibm, @sfeng33, @gkswns0531, @anhtra3889, @lesj0610, @HollowMan6, @tomasruizt, @nien-hui, @DemisEom, @aijanai, @wxsIcey, @chao-xu-spec, @Woolgathererer, @VidyasagarDudekula, @sragrawal, @GCSimba, @ganeshr10, @Liamup777, @hj-mistral, @ivanium, @YaashuDave, @vgokhale, @shen-shanshan, @soodoshll, @Blaze-DSP, @git-jxj, @oonyshch, @zhou-yuhan, @shoutoutuoadi325, @brandonfzhang, @arpera, @zhewenl, @cleonard530, @dharak-cohere, @itej89, @Rockyeast, @he-yufeng, @xinyuangui2, @ischencheng, @NVShreyas, @GuSangmo, @neverhook, @JaredforReal, @nkuhn-vmw, @YukioZzz, @cagrikymk, @alec-flowers, @zylen-det, @xinnywinne, @wangxiyuan, @amd-xavierwang, @lucas-santos-amd, @hclsys, @stu-cao, @lijipeng787, @Sunt-ing, @Shr3yash, @giannis2two, @Hritik003, @yiminghub2024, @AlanFokCo, @lucifer1004, @bvolpato, @Yejing-Lai, @nbommi19, @paolo-perrone, @bantianyue, @AnvitDevadiga, @tarukumar, @Ronald1995, @suiyoubi, @mayuyuace, @drago1234, @mrodden, @nv-nikitaj, @jbelofamd, @hellozmz, @HwwwwwwwH, @SumanthRH, @alexm-redhat, @freyfwt, @rexlee8776, @qthequartermasterman, @coderfeli, @SIDDARTHAREDDY8, @shengliangxu, @jbottleson-nvidia, @yuanheng-zhao, @G-haoyu, @ericschreiber, @milesial, @jonahsamost, @jayzuccarelli, @sungsooha, @KaloyanNaumov, @javieratomic, @akshay-babbar, @saichowdary007, @tzielinski-habana, @CodersAcademy006, @glenmessenger, @abmfy, @orozery, @hongxiayang, @MIR-AMD, @zRzRzRzRzRzRzR, @peakcrosser7, @guan404ming, @afierka-intel, @kouroshHakha, @jikunshang, @luckyq, @chengy-sysu, @xaguilar-amd, @stecasta, @brandonpelfrey, @avininjamay8, @sarankumar1325, @hechengkai, @lizzzcai, @fangyuchu, @majiayu000, @seanbell, @karverma-amd, @wangyicong52, @ChengYM5685, @HangruiCao2000, @AMD-yanfeiwang, @TimRen01, @0z5a, @ActiveSky, @noa-neria, @Jackmeson1, @aniskumar-nv, @mindungil, @meenchen, @jthomson04, @anencore94, @SCcagg5, @heyemlee, @saifltr, @nvyutwu, @world-executed, @JulienDarve, @davidteather, @wguo7, @GodRishUniverse, @nilig, @benenzhu, @joa-stdn, @maithilijoshi20, @hyunnnchoi, @mickaelseznec, @suvigyav-cohere, @YangXu1990uiuc, @nishantrevur, @isztldav, @yeh-sudo, @chennyiiis, @ZhengGong-amd, @tevinshi2015, @Jaeyeong-CHOI, @andakai, @xu797, @zxd1997066, @Willian-Zhang, @kbenkhaled, @kliuae, @kazutoiris, @kushaldabbe, @liulanze, @UsamaKenway, @Anieca, @TQCB, @bandham-manikanta, @effi-ofer, @mirza-halilcevic, @umarkovi-amd, @nemanjaudovic, @yousafshah, @hlin99, @nliu365, @fanxingran, @afriedri, @NikitaLogvinenko, @rjrock, @0xDexe, @mosafariuk, @zixi-qi, @ltllt1, @djmmoss, @MayaJShetty, @chf003, @tlrmchlsmth, @JohnZhao01, @rudrakshkarpe, @Josephasafg, @obendidi, @Medhavin38, @Anchen798, @my-git9, @haosdent, @Than0316, @aleksandr-inherent, @tpopp, @SiluPanda, @okdshin, @fululi12, @eugr, @jamesETsmith, @askliar, @tahsintunan, @dudilester, @xzwgit, @jcotant-inferact, @Chinmay-Kulkarni-AMD, @JimmyToluene, @artem-spector, @Gin-Sin, @zexplorerhj, @jackyYang6, @qqaatw, @lHrHenry233, @zw-g, @gbyu-amd, @tryryry, @linitra24, @engint-cerebras, @katec846, @bnellnm, @MrGeva, @waynehacking8, @xutingl, @jiacao-amd, @hojuna, @MuonChaser, @bavinainuganti9, @zupengwang, @hungnnvidia, @akote123, @lance5890, @Neuro1729, @eicherseiji, @ziywang50, @lvvgang, @Hrushikesh-ramilla, @ilyas-mallah, @Mazukiri, @chenjr97, @micah-wil, @liuyumoye, @shustovTE, @emmericp, @Nuriel32, @sricursion, @Id545, @PerryZhang01, @ekintel, @GirasoleY, @guanzheh, @ShuoleiWang, @asm582, @pst2154, @mawong-amd, @majq92, @sheralskumar, @Hiro208, @vcdlk, @ZhouXiaoya25, @snadampal, @csahithi, @hnt2601, @ybgao-nvidia, @jiaqiang-dot-liu, @ketor, @ApostaC

Source: vLLM releases · github.com