跳到正文
实时
vLLM releases· khluu·· 21 小时前0

GitHub Copilot v0.34.0 发布

v0.34.0

要点速览

GitHub Copilot v0.34.0 发布,提供代码生成、审查、安全和协作功能,支持自定义集成和高级安全特性。

正文

v0.34.0

1,061 commits from 412 contributors (120 new). Thank you!

TL;DR

  • 10 new models (Mistral Large 4, MiniMax-M3.1-Flash, Nemotron-3.5-Transcribe, and more!)
  • Extensible KV cache is on by default: vLLM sizes the KV cache from memory actually free after warmup, so most models get a bigger cache and more throughput.
  • Model Runner V1 is gone. Model Runner V2 is now the only GPU runner.
  • Big speedups on Blackwell and MI355X: up to 1.95× decode (Step5 on GB200), 1.6× throughput (GLM-5.3-Flash), 1.24–1.40× faster long-prompt TTFT (GLM-5.3).
  • Upgrade check: default block size is now 64, ROCm 7.2 images are gone, and huggingface_hub >= 2.0.0 is required. See Before you upgrade.

Before you upgrade

Important

These changes can break existing setups.

Change What to do PR
Model Runner V1 removed Remove VLLM_USE_V2_MODEL_RUNNER=0; it now fails at startup. Port V1 logits processors to vllm.v1.worker.gpu.sample.logits_processor. #63373
Default --block-size is 64 (was 16) Pass --block-size 16 to keep the old behavior. ROCm paged attention still uses 16. #63231
Extensible KV cache on by default; gpu_memory_utilization defaults to 1.0 To keep the old sizing, pass --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache. #56492
ROCm 7.2 artifacts dropped Use the ROCm 10.0 wheels and images. -rocm72 is no longer published. #61939
huggingface_hub >= 2.0.0 required Also requires tokenizers >= 0.23.3 and timm >= 1.0.30. #64779
QuTLASS / fp_quant removed fp_quant checkpoints no longer load. #65230
Marlin W4A8 removed Use --quantization humming with VLLM_HUMMING_INPUT_QUANT_CONFIG instead of VLLM_MARLIN_INPUT_DTYPE. #65542
Keye-VL and Keye-VL-1.5 removed — #64287

All breaking changes, deprecations and new defaults are listed in the full changelog.

New models

Model Notes PR
Mistral Large 4 1T total / 52B active, FP8 and NVFP4. Runs on NVIDIA Blackwell/Hopper and AMD MI350. #64534
MiniMax-M3.1-Flash Blackwell NVFP4 and ROCm kernels, DSpark draft #64111
NVIDIA Nemotron-3.5-Transcribe 30B-A4B speech model: audio chat and /v1/audio/transcriptions #64282
Ricursive Ripple-65B Preview BF16 and NVFP4, ripple reasoning and tool parsers #65745
Zyphra ZAYA1 ZAYA1-8B, ZAYA1-74B-preview #65302
Baidu Unlimited-OCR-2 DASH sparse attention #64016
Liquid AI LFM3, LFM3-VL, LFM3-Audio #65826, #65827, #65828
NVIDIA Canary-1B-v2 ASR and X↔En translation #64215
Ming3-ASR #63585
NVIDIA Parakeet TDT Through the vllm-parakeet-plugin #65005

Highlights

Extensible KV cache, on by default

  • On CUDA and ROCm, the KV cache is sized from memory actually free after CUDA graph capture and warmup, not from an estimate (#56492).
  • Result: a larger cache (e.g. MiniMax-M3 on H200: 433k → 672k tokens), so throughput is higher and TTFT lower at high concurrency.
  • Trade-off: per-token decode latency can rise on some models (TPOT +6–18% on some H200 models at concurrency 128). Opt out with --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache.
  • Hardened with fixes for allocator fragmentation, startup OOMs and KV connectors (#67296, #67485, #66731, #66523, #66197, #65137, #66605, #66674).

Model Runner V2 only

  • Model Runner V1 is removed (#63373).
  • Dual-batch overlap now runs on MRV2, including with speculative decoding and DCP (#61464).

Speculative decoding

  • Suffix decoding runs on MRV2 with a GPU speculator; Arctic Inference is no longer needed (#60339).
  • Custom speculators: declarative configs plus a register_speculative_method plugin API (#63205).
  • CUDA graphs for MRV2 draft-model speculators (#60989).
  • Adaptive verification works with XPress (#65175).
  • Draft KV cache matches the target's dtype, so an FP8 target gets an FP8 draft cache (#65198).

Performance

Improvement Hardware Gain PR
Step5 NVFP4 MoE on TRTLLM-Gen 4×GB200 up to 1.95× decode #65152
GLM-5.3-Flash DeepGEMM FP8 MegaMoE NVIDIA 1.6× output throughput vs DEP4 #60895
Muse Spark Open specialized decode (opt-in) GB300 1.5× single-stream decode #64077
Token-sharded sparse-MLA prefill under TP (default) GB200 GLM-5.3 TTFT 1.24–1.40× faster at 32K–128K #63956
KV-balanced MLA decode for Kimi-K3 (opt-in) GB300 1.34–1.59× decode attention #63638
Qwen3.5 on breakable CUDA graphs (no torch.compile) NVIDIA 3.0–8.8% lower decode ITL #51012
FlyDSL decode monokernel, Muse Spark Open DSpark 8×MI355X +35–57% interactivity at conc. 1–8 #63440

AMD ROCm

  • Release images and wheels are ROCm 10.0 only (#61939).
  • MoRI EPv2 via --all2all-backend mori (#61361).
  • Kimi-K3 specialized decode on MI355X, opt-in (#61435).
  • TRITON_ATTN is the default attention backend on RDNA3 and RDNA4 (#60418).

Serving and operations

  • Checkpoint/restore (CRIU + cuda-checkpoint) through /checkpoint/prepare, /checkpoint/status and /checkpoint/restore, in both the Python and Rust frontends (#62570).
  • Fault-tolerant DP+EP scale-down: dead ranks are removed and their experts redistributed without a restart (#46370).
  • Best-effort NUMA binding on by default (#61258).
  • Opt-in cache-aware admission ordering (#54625).
  • Rust frontend: stateless Responses API (#63902) and n > 1 (#65625).

Install

Python wheels

Platform Install
CUDA 13.0 (PyPI) pip install vllm
CUDA 13.0 (uv) uv pip install vllm --torch-backend=auto
ROCm 10.0 pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.34.0/rocm100
XPU uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.34.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match

Docker images

Platform Image Short alias
CUDA 13.0 (default) vllm/vllm-openai:v0.34.0 vllm/vllm:v0.34.0
CUDA 13.0 + DeepEP over AWS EFA vllm/vllm-openai:v0.34.0-efa vllm/vllm:v0.34.0-efa
ROCm 10.0 vllm/vllm-openai-rocm:v0.34.0 vllm/vllm:v0.34.0-rocm
CPU vllm/vllm-openai-cpu:v0.34.0 vllm/vllm:v0.34.0-cpu
XPU vllm/vllm-openai-xpu:v0.34.0 vllm/vllm:v0.34.0-xpu

latest tags work the same way (e.g. vllm/vllm:latest-rocm). The default CUDA image now includes the AWS EFA userspace (libfabric, aws-ofi-nccl). The -efa image adds DeepEP built for EFA and routes NCCL over EFA by default (#65576).

The Assets section below has the source tarball and the CUDA 13.0 (x86_64, arm64), CPU (x86_64, arm64, macOS) and XPU (x86_64) wheels.

Full changelog

Model support

  • Mistral Large 4 (#64534) runs on MRV2 with:
    • sparse MLA plus the DSA indexer, and cross-layer KV sharing;
    • DSpark, DCP/PCP, NIXL P/D and HiSparse.
  • Other new models:
    • MiniMax-M3.1-Flash (#64111), plus P/D across different prefill and decode TP (#64475).
    • Nemotron-3.5-Transcribe with MTP and DFlash2 (#64282, #64944).
  • DeepSeek-V4 / V4.1
    • Prefill gathers compressed KV once per KV-source group: 38 → 4 gathers per step (#59131).
    • Decoder replay graphs share the model graph pool: prefill graph memory 15.52 → 9.88 GiB per rank on GB200 (#63155).
    • fused_inv_rope_fp8_quant 3.5× faster, prefill 6% faster (#63540).
    • DSpark/MTP memory savings of about 512 MiB per GPU (#63996).
    • Engram host tables shared across DP ranks (#62359).
    • DeepGEMM sparse MQA logits on the FP8 indexer cache (#62011).
    • Packed multi-image vision encoding, 1.6–2.6× (#65030).
    • LoRA on the language model, compressor and wo_a layers (#57245, #64880).
    • DSpark with PCP (#61994); PCP with PP or DP (#63098).
    • Fixes: sparse prefill in FULL CUDA graphs (#59986), prefill memory budgeting (#63047), MTP embedding under PP (#65816), DeepSeek-V3.2 PP (#59961).
  • GLM
    • GLM-5.3-Flash: DeepGEMM FP8 MegaMoE (#60895), adaptive verification (#60229), row-strided QKV into FlashKDA (#61614).
    • GLM-5.3 FP8 MoE sorted-A TMA on H200, 6.0–7.6% E2E throughput (#64760).
    • FP8 KV cache for DSpark and DFlash drafts (#63841, #64612, #65553).
    • FP32 indexer MQA logits restored for GLM-5/GLM-5.3/HY-V4, with new --attention-config.indexer_logits_dtype (#64580).
    • Fused KDA convs fix stale weights after RL refit (#60991).
  • Muse Spark
    • Opt-in Muse Spark Open specialized decode on GB300 (#64077).
    • One RoPE cache shared across layers: 11.5 GiB per GPU saved, 14.5% more KV cache on 8×B200 (#65206).
    • GEMM and communication+norm fusions (#65438, #64638); hybrid DeepGEMM MegaMoE / TRT-LLM for NVFP4 (#64525).
    • CUDA routing kernel (#64553), MegaMoE JIT warmup (#63920), DSpark draft loading fix (#65125).
  • MiniMax-M3 / M3.1
    • Sequence parallelism and GEMM-RS for TP + MegaMoE, 14–20% lower P50 TTFT (#64659), also with PP (#65537).
    • Fused norm + MXFP8 quant, 1.39–1.60× (#64852).
    • Triton query-tiled sparse prefill on Hopper (#57420); SM90 split-K indexer (#64037).
    • No host sync in prefill metadata (#65883); NVFP4 MegaMoE warmup (#66227); fmha_sm100 indexer updates (#66228).
    • Fixes: M3.1 prefill workspace (#64856), sequence parallelism (#65496), encoder CUDA graphs (#65226).
  • Kimi
    • Opt-in --attention-backend KIMI_K3_KV_BALANCED_MLA for agentic workloads on Blackwell (#63638).
    • Kimi-K3 MegaMoE symmetric buffer 10.73 → 2.50 GiB per GPU at TP8 EP (#63541).
    • FlashInfer fused FP8 chunked-context K/V pack (#61527).
    • Fixes: KDA prefill OOM (#63989), NaNs via FlashKDA/FlashMLA pins (#64759), sleep mode (#64583, #66273).
    • Kimi K3.1 groundwork landed (#66030, #66028), but K3.1 is not servable in this release.
  • Qwen
    • Qwen3.5: breakable CUDA graphs instead of torch.compile (#51012); fused norm/gate + quant (#64509) and AR + norm + quant (#64723); GDN in_proj overlap (#64425, #64999).
    • GDN prefill state gather/scatter fused for Qwen3-Next/Qwen3.5 and Kimi KDA, 1.16–1.50× (#66064).
    • Qwen3.8-Flash-Next: PLE disk offload via --engram-config '{"disk_offload": true}' (27 GiB less on DGX Spark, #59958), PP (#60234), shared QSA top-k buffer (#65619), loading fixes (#62318, #65812, #64820, #60490).
    • Fixes: long-context int32 overflow (#63761), multi-layer MTP (#62111).
  • Other models
    • Step5 MegaMoE with PP (#65504) and KV block sizing (#65153).
    • Gemma 4: multimodal encoder DP (#60314), PP (#65989), FA4 FP8 MMA on SM90 (#60372).
    • Moved to the Transformers modeling backend: OLMoE (#63384), Gemma v1 and StableLM (#66393), Aria (#59767; temporarily about 11% less KV cache and 25% lower image throughput).
    • Matryoshka dimensions for EmbeddingGemma 2 (#64885, #60978); RADIO C-RADIO v2/v3/v4 (#58468).
    • Qwen3-ASR on the upstream Transformers config; no --hf-overrides needed for the ForceAligner (#66097, #63274).
    • Fixes: GPT-OSS MXFP4 LoRA (#65825), PLaMo3 GPTQ (#62836), Ling3 (#65088), DomynEdge (#66819).
  • Multimodal
    • Encoder CUDA graphs for Pixtral/Mistral3 (#56277), shared capture buffers (#65084), one replay per batch up to encoder_cudagraph_max_token_budget (#65227).
    • On-device input normalization for Nemotron VL (#57924), Gemma 3 (#64487), Gemma 4 (#65577).
    • --mm-encoder-tp-mode data for MiMo-V2.6 (#64179).
    • Fixes: FlashInfer ViT attention on GLM-4.1V/GLM-OCR/GLM-5.3-Flash (#65778), Qwen3-Omni and Qwen2.5-Omni video budgets (#66017, #66190), dots_ocr with TP (#64543), embedding-only startup (#65689), media cache eviction (#65363).

Engine core

  • Extensible KV cache
    • On by default on CUDA and ROCm (#56492).
    • Headroom for measured allocator fragmentation, validated on Qwen3.5-397B-A17B-FP8 at TP8+EP on 8×H200 (#67296).
    • Fixes: sizing and startup OOMs (#67485, #66523, #66731, #66197, #65554), connectors that failed to start (#65225), up to 84% undersized caches on DGX Spark (#65413), --max-model-len auto (#65463).
    • Startup warning when an explicit gpu_memory_utilization withholds memory from it (#66704); docs on when to set it (#67071).
  • Model Runner V2
    • The only runner after MRV1's removal (#63373); DBO with spec decode and DCP (#61464).
    • Separate prefill (token) and decode (request) CUDA graph capture sizes (#63406).
    • Staged writes copied straight into UVA (#63633); idle sampler writes skipped, about 10–11% faster small-batch decode (#62748).
    • Sampler warmed up after graph capture to stop memory growth under load (#62560).
    • Lower peak memory for chunked prompt logprobs (#64841, #64914); Triton ReplaySSM (#65033).
    • NVIDIA Confidential Computing decode fixes (#59981).
  • Speculative decoding
    • Suffix decoding (#60339) and custom speculators (#63205) on MRV2; draft-model CUDA graphs (#60989).
    • XPress with adaptive verification (#65175); adaptive verification draft budgets on a Triton kernel (#63421).
    • Batch-sharded draft logits (#63554); DSpark sequential blocks (#63886).
    • DSpark sliding-window MLA draft layers (#63795) under DCP (#63796); fused Markov sampling, 42–50% less sampling time (#64476); draft_lm_head_quantization (#64368).
    • Draft KV cache dtype aligned with the target (#65198).
    • Trace replay with ngram/EAGLE3/MTP, 1.47–2.03× (#63275); 3× faster resample kernel (#65100).
    • Fixes: DFlash acceptance of 0 with FULL CUDA graphs (#64059), a resample race (#66048), acceptance-estimator cancellation (#65229), standalone draft prefill under FULL graphs (#65804), grammar-invalid drafts (#61557).
  • Scheduler and engine overhead
    • Prompt token ids kept as int32 buffers: 40× less per-request CPU on a 128K prompt (#62702).
    • Per-request id lists kept out of the cyclic GC (#65300); 45–50% less KV block allocation CPU (#65831).
    • New requests and aborts handled while waiting for model output (#65466), with a cheaper wait loop (#65773).
    • Opt-in cache-aware admission via --cache-aware-admission-window (#54625).
    • --max-num-inflight-kv-loads starts async KV loads while the batch is full (#65319), with cached reservations: 47.2 → 5.4 ms per schedule (#65285).
    • A multimodal item is no longer split by a prefix hit (#65271).
  • Startup and compile
    • No openai/fastapi imports in EngineCore and workers (#65423, #65589).
    • CUDA graph memory profile skipped when its estimate is disabled (#64622).
    • Device-neutral compile cache via VLLM_COMPILE_ON_ONE_RANK, on by default (#63247); all graph-affecting config fields hashed into compile cache keys (#65652).
    • --shuffle-safetensors-files for large EP startups (#42837).
    • BasevLLMParameter's Python __torch_function__ dropped: GLM-5.2-FP8 TTFT −6.1% on GB300 (#63785).
  • Lifecycle
    • Process checkpoint/restore over HTTP (#62570); opt-in multi-worker CUDA checkpoint sleep (#64186).
    • Best-effort NUMA binding by default; --no-numa-bind to disable (#61258).
    • Sleep mode with exact-size pinned backups, up to about 578 GiB less host memory per node on Mistral Large 4 TP8 (#64630); level 2 discards only parameters (#66010, #66165, #59994).
    • Fast Start weight-cache restarts for online FP8/MXFP8, ModelOpt CUTLASS FP8/NVFP4/block-FP8 and AWQ/GPTQ Marlin (#63672, #63754, #64119, #64182), with pluggable shared warmup-cache stores (#65168).
    • WatermarkConfig.enforce (#62966).
  • RL and weight loading
    • VLLM_RELOAD_MODE=direct writes checkpoint tensors straight into the live model (#64336, #64352, #65281).
    • Every load and update finishes through finalize_model_weights (#65914, #64357, #64692, #57678).
    • Load plans for weight-transfer engines (#65235); per-update NCCL source ranks for PP trainers (#62557).
    • Sparse weight patches validated in one pass, 40–48% shorter pause (#66021).
    • Offloaded KV tier survives pause/sleep (#57810); partial stacked weights rejected (#63752).
  • LoRA
    • MoE-LoRA shrink/expand kernels up to 64 tokens (#64653); embedding LoRA scaled by embed_scale (#62470).
    • Rejected: adapters on unwrapped modules (#63489), non-linear expert placement with EP (#62797).
    • Fixes: reloads with requests in flight (#64616), KV-sharing fast prefill (#61162), SEQ_CLS heads (#63286), fused MoE expand rounding (#63894), Mamba-1 (#64681).
  • Batch invariance: LFM2 ShortConv (#62367), FlashInfer FP8 bmm and W4A16 NVFP4 (#62192), GPTQ (#60479), FA sliding window (#65627), aten::mm out_dtype overloads (#60047).

Large-scale serving

  • Expert parallelism and fault tolerance
    • DP+EP scale-down via POST /v1/fault_tolerance/apply (#46370); MoRI EPv2 (#61361).
    • Opt-in symm_mem EPLB weight-migration backend (#62009).
    • --enable-ep-weight-filter with EPLB: DeepSeek-R1 NVFP4 DEP8 reads 105 GiB per rank instead of 385 GiB (#64187).
    • num_redundant_experts defaults to the minimum valid value (#30614).
    • Skipped combine staging copy in flashinfer_nvlink_one_sided (#65498); DeepEP V2 padded rows (#54109).
  • Context parallelism and sparse attention
    • Token-sharded sparse-MLA prefill under TP (#63956); DCP on the FlashMLA sparse bf16 KV path (#61609).
    • MLA decode up-projected before the A2A merge with --dcp-comm-backend a2a (#64994); faster DCP sparse-indexer top-k (#65178, #64129).
    • Cross-layer KV cache sharing for MLA layers (#64095).
    • Fixes: PP rank derivation under PCP (#65539), FlashInfer MLA DCP decode (#59392).
  • KV connectors
    • NIXL: push mode via transfer_mode (#64230); ucx_backend_params (rocm_ipc on MI355X: 3.4 → 483.7 Gbps, #62905); 38% fewer hybrid SSM transfer bytes (#62317); block pairing across different P/D block sizes (#59746), with mismatched push handshakes rejected (#59747); same-host NVLink with the extensible KV cache (#65137).
    • Mooncake: selective leases (#63420), automatic cache_prefix (#65160), per-GPU RDMA devices (#65076), heterogeneous-TP hybrid sharing (#54307), sleep mode (#59934), lease renewal (#64193).
    • Lookahead block hashes in NIXL, Mooncake and offloading connectors (#64096); UMBP standalone mode (#61795, #61793).
    • Out-of-tree and LMCache MP connectors with the extensible KV cache (#66605, #66674).
  • KV offload
  • HiSparse
    • Uses the extensible KV cache by default: 16% more GPU+host KV capacity (#65214).
    • Fused slot invalidation, up to 1.8% lower TPOT on GLM-5.3-NVFP4 (#60103); fewer hash probes (#61263); speculative-group prefetch (#63801).
    • Platform backend for hardware plugins (#62592); scheduler deadlock fix (#62979).
  • EPD and distributed
    • Metadata-only image and video inputs for GLM-5.3-Flash and DeepSeek-V4.1 (#64071, #62802).
    • Bitwise-identical 1-stage and 2-stage custom all-reduce (#57042); FlashInfer all-reduce gated on 16-byte rows (#63123).
    • Cross-node MNNVL groups kept off CUDA IPC all-reduce (#55835).

Hardware

  • NVIDIA
    • Step5 NVFP4 TRTLLM-Gen MoE (#65152).
    • DeepSeek-V4.1: latency_specialized decode extended, 5.9–17% faster decode step on GB300 (#63921); swap-AB CuTe DSL decode attention for TP2/TP4, 1.4–2.15× (#62633); opt-in fused DSpark Markov sampling, 1.6–3.9× (#64442).
    • Sparse-indexer top-k reads BF16 logits directly (#62172, #64138); single-launch fused prefill top-k (#60618).
    • GLM-5.3 TMA MoE projections on H200 and SM100 (#63820, #65746); SM90 native small-head sparse FlashMLA, +10% on GLM-5.3-Flash TP4 (#64510).
    • CuTe DSL BF16x3 router GEMM for fp32 routers on Hopper (#61903); fused GDN MTP decode for DSpark K=8, +18% on B300 (#65874).
    • GLM-5.3-Flash on 2× DGX Spark, 15% lower TPOT (#62956); faster weight loading on integrated GPUs (#58868).
    • Rubin locality-domain NVFP4 MoE (#61685); Jetson Thor (SM110) fixes (#62133, #65050, #62275).
  • AMD ROCm
    • ROCm 10.0 only (#61939); AITER v0.1.28.post1 (#63774, #65937); MoRI EPv2 (#61361).
    • MonoKernel decode for Muse Spark Open DSpark verify (#63440), DeepSeek-V4.1-Flash at TP8 (#64101) and the MiniMax-M3.1 DSpark drafter, +10% P90 interactivity (#65386).
    • Kimi-K3 specialized decode (#61435), AITER-tuned prefill GEMMs: mean TTFT 5.35 → 4.05 s (#63730), opt-in FlyDSL FP8 MLA prefill (#59124).
    • DeepSeek-V4-Pro P/D over MoRIIO (#48989); Step-5-Preview (#61837) and Reflection Beam 1.0 (#62057) on ROCm.
    • RDNA: TRITON_ATTN by default (#60418), AITER Triton kernels on RDNA3 (#52970), HIP GDN prefill backend for gfx1151 (#56307), opt-in W4A16 dense prefill cache: TTFT −22 to −26% on Strix Halo (#60710).
    • Fixes: Kimi-K2.6 int4 prefill regression on Triton 3.8 (#62565), MI355 extensible KV cache OOMs (#65418).
  • Intel XPU
    • PyTorch 2.15 (#63977) and vllm_xpu_kernels 0.1.17 with new MoE/MLA interfaces (#60609).
    • Per-token dynamic FP8 activations (#61362); faster batched Triton MoE (#51605); MRoPE kernel (#59938); GDN metadata reuse (#64479); DeepSeek-V4 adaptive verification (#62309).
  • CPU
    • MRV2 on CPU backends without Triton: Arm, IBM Z, PowerPC, RISC-V (#61995).
    • zentorch 2.14.0.2 (#65669) and Laguna MoE on Zen (#63711).
    • Faster GDN spec-decode updates (#64829); Gemma 4 MoE gelu_tanh (#61593); bidirectional encoder-only attention (#63738).
  • Other: out-of-tree layer overrides via vllm.model_executor.hw_agnostic.resolve (#60218); shared routed-expert row restorers (#64689); Ascend Triton fix (#64013).

Quantization

  • New --quantization lut_b_moe: about 3.14-bit MoE experts with MXFP8 activations, native on Rubin (#62759).
  • w4afp8 checkpoints on the W4A8 FP8 MoE path (#64410).
  • Block Hadamard input rotation with online mxfp4_hadamard32 and Quark online-rotation checkpoints (#62750).
  • CSF-compressed NVFP4 block scales in ModelOpt (#64312); activation quant overrides for online MXFP4 (#51808).
  • DeepSeek-V4.1 MXFP8 linears on cuBLASLt (#64815); 3× faster MXFP4 activation quant (#63101); faster FP8 quant on SM100+ (#61641).
  • QuIP NVFP4 on Humming (#65230).
  • Fixes: NVFP4 NaN logits (#63382, #61755), Humming FP8 (#60163, #56606), hadacore (#63514).

API and frontend

  • New endpoints and options
    • /checkpoint/prepare, /checkpoint/status, /checkpoint/restore (#62570).
    • thinking_token_budget on /v1/responses (#63896); /v1/completions in vllm run-batch (#64097).
    • Streaming derender on /inference/v1/generate (#63103).
    • Anthropic Messages API: min_tokens (#65459) and thinking.type=between_tools (#65954).
    • vllm:request_render_duration_seconds histogram (#60213).
  • Tool calling and reasoning
  • Behavior
    • Corrupt or oversized images and disabled local media return 400 instead of 500 (#63433, #60308, #64032).
    • Responses accepts incomplete assistant/reasoning history, e.g. from Codex (#35508).
    • Multi-template processors use the default template (#63606); an empty tools array is accepted like OpenAI (#65547).
    • The first streamed usage reports cached prompt tokens (#65036); plain completion stream frames render about 9× faster (#52875).
  • Rust frontend

Security, dependencies and build

  • Security
    • ReDoS in the validation-error sanitizer fixed (#52120).
    • FunAudioChat audio bounded to the profiled duration (#57411).
    • /v1/systemone honors --trust-request-chat-template (#64503).
    • Raised floors for filelock, OpenTelemetry, graphql-core and others (#63574).
  • Dependencies
    • PyTorch stays at 2.14.0. FlashInfer 0.7.3 (#65930).
    • huggingface_hub >= 2.0.0, tokenizers >= 0.23.3, timm >= 1.0.30 (#64779); transformers >= 5.16.1, < 5.23.0 (#64049, #64529); mistral-common >= 1.13.0 (#64169).
    • DeepEP 2.1 (#59844); DeepGEMM pin updates (#64411, #64580, #65494); FlashMLA and FlashKDA pins (#61971, #64759).
    • NIXL 1.5.2 (#64579, #64585); mooncake-transfer-engine 0.3.14 (#65576); Run:ai Model Streamer 0.17.1 (#63174); fastokens >= 0.3.5 for VLLM_USE_FASTOKENS (#65679).
    • ROCm: AITER v0.1.27 → v0.1.28.post1 (#63774, #65937). XPU: PyTorch 2.15 (#63977), triton-xpu 3.9.0 (#65047), vllm_xpu_kernels 0.1.17 (#60609). CPU: zentorch 2.14.0.2 (#65669).
  • Build
    • x86_64 CUDA wheel about 82 MB (about 25%) smaller, kernels unchanged (#65518).
    • Default CUDA image installs the AWS EFA userspace; the -efa image now only adds DeepEP for EFA (#65576); nightly EFA images as vllm/vllm-openai:efa-nightly (#64028).
    • One ROCm build and release pipeline, ROCm 10.0 only (#61939).
    • Source builds fetch triton_kernels from a tarball (#65162) and ship tuned kernel configs in the wheel (#64063).
    • CUDA x.yf arches no longer fall back to a lower minor (#66016).

All breaking changes, deprecations and new defaults

  • Model Runner V1 removed (#63373).
    • VLLM_USE_V2_MODEL_RUNNER and VllmConfig.use_v2_model_runner are gone; setting the env var to anything but 1 fails at startup.
    • Dropped with it: the V1 logits-processor API (vllm.v1.worker.gpu.sample.logits_processor is the only interface), the custom_class speculative method, use_heterogeneous_vocab, disable_padded_drafter_batch, parallel_drafting outside DFlash/DSpark, PP with external_launcher, NGramPerReqLogitsProcessor, and x86 CPUs without Triton.
    • Deprecated: parallel_drafting, capture_torch_profiler, enable_layerwise_nvtx_tracing.
  • Default --block-size is 64 instead of 16 (#63231). Backends with a preferred block size still override it (ROCm paged attention keeps 16), and an explicit --block-size still wins.
  • gpu_memory_utilization defaults to 1.0 when unset with the extensible KV cache on (#56492). An explicit value in (0.8, 1.0) that withholds 1 GiB or more logs a warning (#66704).
  • ROCm 7.2 wheels and -rocm72 images are no longer published (#61939).
  • Removed:
    • QuTLASS and the fp_quant method (#65230).
    • Marlin INT8/FP8-activation (W4A8) paths and VLLM_MARLIN_INPUT_DTYPE; use --quantization humming with VLLM_HUMMING_INPUT_QUANT_CONFIG (#65542).
    • Keye-VL and Keye-VL-1.5 (#64287); disable_mtp_completeness_check (#49459); the refresh_derived_weights hook (#65914).
  • Dependencies: huggingface_hub >= 2.0.0 (#64779); mistral-common >= 1.13.0 (#64169); XPU requires vllm_xpu_kernels 0.1.17 (#60609).
  • Model and plugin code: all process_weights_after_loading hooks share one signature and the loader runs every module's hook (#57678); config compute_hash() replaced by compile_factors() / hash_config() (#65652).
  • API behavior:
    • llama3_json no longer streams tool arguments incrementally (#63569).
    • Canary transcription without language returns 400 (#65445).
    • An empty tools array is accepted in both frontends (#65547, #65854).
    • /v1/systemone rejects request-supplied chat templates without --trust-request-chat-template (#64503).
    • LoRA adapters on modules vLLM does not wrap are refused (#63489); misconfigured NCCL weight transfer raises (#57022).
    • vllm-rs serve no longer reads --data-parallel-rank from VLLM_DP_RANK (#65369).
    • AITER fused shared experts with MiniMax M3 and DeepSeek V4 need VLLM_ROCM_USE_AITER=1 (#61457).
  • Still deprecated: --mamba-cache-mode has no effect; its removal was postponed (#64838).
  • New defaults:
    • Extensible KV cache (#56492); block size 64 (#63231); best-effort NUMA binding (#61258).
    • Token-sharded sparse-MLA prefill at 1024 or more prefill tokens (#63956); MLA decode up-projection before the A2A merge (#64994).
    • VLLM_COMPILE_ON_ONE_RANK (#63247); minimum valid EPLB num_redundant_experts (#30614).
    • HiSparse on the extensible KV cache (#65214); encoder CUDA graphs up to encoder_cudagraph_max_token_budget=5120, eager above (#65227).
    • TRITON_ATTN on RDNA3/RDNA4 (#60418); MRV2 on non-Triton CPUs (#61995); float32 Mamba SSM cache for Nemotron Omni (#63218); Zipf routing in vllm bench forward (#64167).

Contributors

120 new contributors: welcome!

All 412 contributors

@BugenZhao, @Thangnguyenvn98, @AndreasKaratzas, @njhill, @hmellor, @khluu, @aoshen02, @Juntian777, @mgoin, @NickLucche, @gau-nernst, @okorzh-amd, @yewentao256, @vadiklyutiy, @yma11, @zyongye, @Pb314314, @jeejeelee, @WoosukKwon, @fxmarty-amd, @waizuichougou, @stefankoncarevic, @xsxszab, @djramic, @ZJY0516, @vMaroon, @MatthewBonanni, @Fangzhou-Ai, @liranschour, @TheEpicDolphin, @chaojun-zhang, @gcanlin, @whx-sjtu, @Alex-Wengg, @roberteg16, @subhashpolisetti, @hshrivastava-droid, @hao-aaron, @mkhazraee, @divakar-amd, @Isotr0py, @seungrokj, @jperezdealgaba, @aarushjain29, @vllmellm, @bzsuni, @liusy58, @zhenwei-intel, @Etelis, @rasmith, @Ankur-singh, @reidliu41, @jzakrzew, @ZhenchengLin, @charxwu, @xinli-sw, @subin9, @namgyu-yun, @frida-andersson, @dilberx, @josh-schell-amd, @simon-veitner-redhat, @akii96, @cjackal, @chaunceyjiang, @wzhao18, @chunfangamd, @robertgshaw2-redhat, @cpersson-amd, @CJJ1008, @yzong-rh, @zhenwenqi2024, @johncalesp, @SherifWaly, @gc-fu, @Vegetog, @Manfredss, @sohom-cs, @albertoperdomo2, @Dao007forever, @Alex-ai-future, @taneem-ibrahim, @gty111, @bogdan-petkovic, @jiuyueQ, @noooop, @lucamotz, @RyanMa29, @paulpak58, @ilmarkov, @azaidy, @yifanFengg, @mispa-ms, @jh-nv, @xqoasis, @Prakash-sa, @juliendenize, @galletas1712, @mjkvaak-amd, @louie-tsai, @wtdcode, @nrenvo, @hyukjlee, @srkim-business, @hemuku90, @amenomori, @nsusanto, @zohaibm-amd, @grYe99, @LopezCastroRoberto, @janbernloehr, @RuibinCheung, @yaoyaoding, @DongjiGao, @luyixiao95, @ayush1399, @qli88, @qianlihuang, @roikoren755, @simondanielsson, @faaany, @inc-jeong, @bohnstingl, @fadara01, @spandantiwari, @BaoYunkai, @erezzarum, @mrn3088, @indivats, @LucasWilkinson, @Change72, @esmeetu, @franciscojavierarceo, @hickeyma, @zydo, @roborluo, @fuscof-ibm, @sfeng33, @gkswns0531, @anhtra3889, @lesj0610, @HollowMan6, @tomasruizt, @nien-hui, @DemisEom, @aijanai, @wxsIcey, @chao-xu-spec, @Woolgathererer, @VidyasagarDudekula, @sragrawal, @GCSimba, @ganeshr10, @Liamup777, @hj-mistral, @ivanium, @YaashuDave, @vgokhale, @shen-shanshan, @soodoshll, @Blaze-DSP, @git-jxj, @oonyshch, @zhou-yuhan, @shoutoutuoadi325, @brandonfzhang, @arpera, @zhewenl, @cleonard530, @dharak-cohere, @itej89, @Rockyeast, @he-yufeng, @xinyuangui2, @ischencheng, @NVShreyas, @GuSangmo, @neverhook, @JaredforReal, @nkuhn-vmw, @YukioZzz, @cagrikymk, @alec-flowers, @zylen-det, @xinnywinne, @wangxiyuan, @amd-xavierwang, @lucas-santos-amd, @hclsys, @stu-cao, @lijipeng787, @Sunt-ing, @Shr3yash, @giannis2two, @Hritik003, @yiminghub2024, @AlanFokCo, @lucifer1004, @bvolpato, @Yejing-Lai, @nbommi19, @paolo-perrone, @bantianyue, @AnvitDevadiga, @tarukumar, @Ronald1995, @suiyoubi, @mayuyuace, @drago1234, @mrodden, @nv-nikitaj, @jbelofamd, @hellozmz, @HwwwwwwwH, @SumanthRH, @alexm-redhat, @freyfwt, @rexlee8776, @qthequartermasterman, @coderfeli, @SIDDARTHAREDDY8, @shengliangxu, @jbottleson-nvidia, @yuanheng-zhao, @G-haoyu, @ericschreiber, @milesial, @jonahsamost, @jayzuccarelli, @sungsooha, @KaloyanNaumov, @javieratomic, @akshay-babbar, @saichowdary007, @tzielinski-habana, @CodersAcademy006, @glenmessenger, @abmfy, @orozery, @hongxiayang, @MIR-AMD, @zRzRzRzRzRzRzR, @peakcrosser7, @guan404ming, @afierka-intel, @kouroshHakha, @jikunshang, @luckyq, @chengy-sysu, @xaguilar-amd, @stecasta, @brandonpelfrey, @avininjamay8, @sarankumar1325, @hechengkai, @lizzzcai, @fangyuchu, @majiayu000, @seanbell, @karverma-amd, @wangyicong52, @ChengYM5685, @HangruiCao2000, @AMD-yanfeiwang, @TimRen01, @0z5a, @ActiveSky, @noa-neria, @Jackmeson1, @aniskumar-nv, @mindungil, @meenchen, @jthomson04, @anencore94, @SCcagg5, @heyemlee, @saifltr, @nvyutwu, @world-executed, @JulienDarve, @davidteather, @wguo7, @GodRishUniverse, @nilig, @benenzhu, @joa-stdn, @maithilijoshi20, @hyunnnchoi, @mickaelseznec, @suvigyav-cohere, @YangXu1990uiuc, @nishantrevur, @isztldav, @yeh-sudo, @chennyiiis, @ZhengGong-amd, @tevinshi2015, @Jaeyeong-CHOI, @andakai, @xu797, @zxd1997066, @Willian-Zhang, @kbenkhaled, @kliuae, @kazutoiris, @kushaldabbe, @liulanze, @UsamaKenway, @Anieca, @TQCB, @bandham-manikanta, @effi-ofer, @mirza-halilcevic, @umarkovi-amd, @nemanjaudovic, @yousafshah, @hlin99, @nliu365, @fanxingran, @afriedri, @NikitaLogvinenko, @rjrock, @0xDexe, @mosafariuk, @zixi-qi, @ltllt1, @djmmoss, @MayaJShetty, @chf003, @tlrmchlsmth, @JohnZhao01, @rudrakshkarpe, @Josephasafg, @obendidi, @Medhavin38, @Anchen798, @my-git9, @haosdent, @Than0316, @aleksandr-inherent, @tpopp, @SiluPanda, @okdshin, @fululi12, @eugr, @jamesETsmith, @askliar, @tahsintunan, @dudilester, @xzwgit, @jcotant-inferact, @Chinmay-Kulkarni-AMD, @JimmyToluene, @artem-spector, @Gin-Sin, @zexplorerhj, @jackyYang6, @qqaatw, @lHrHenry233, @zw-g, @gbyu-amd, @tryryry, @linitra24, @engint-cerebras, @katec846, @bnellnm, @MrGeva, @waynehacking8, @xutingl, @jiacao-amd, @hojuna, @MuonChaser, @bavinainuganti9, @zupengwang, @hungnnvidia, @akote123, @lance5890, @Neuro1729, @eicherseiji, @ziywang50, @lvvgang, @Hrushikesh-ramilla, @ilyas-mallah, @Mazukiri, @chenjr97, @micah-wil, @liuyumoye, @shustovTE, @emmericp, @Nuriel32, @sricursion, @Id545, @PerryZhang01, @ekintel, @GirasoleY, @guanzheh, @ShuoleiWang, @asm582, @pst2154, @mawong-amd, @majq92, @sheralskumar, @Hiro208, @vcdlk, @ZhouXiaoya25, @snadampal, @csahithi, @hnt2601, @ybgao-nvidia, @jiaqiang-dot-liu, @ketor, @ApostaC

来源: vLLM releases · github.com