跳到正文
实时
SGLang releases· Fridge003·· 16 天前0

GitHub Copilot v0.5.23 发布

v0.5.23

要点速览

GitHub Copilot v0.5.23 发布，提供代码生成、审查、安全和协作功能，支持自定义集成和高级安全特性。

正文

Highlights

1007 PRs from 257 contributors.

New models

Model Type Cookbook
Muse Spark Open LLM / VLM link
Beam-1.0 LLM / VLM link
MiniMax-M3.1-Flash LLM / VLM link
MiniCPM-V 4.7 LLM / VLM link
Kev LLM / VLM link
Cosmos3-Nano-Sim (Bimanual, Transfer, Auto) Diffusion link
FastH3 Omni Diffusion link
FLUX.3 Image Klein Diffusion link
HY-Nomas Diffusion link
MOSS-Image Diffusion link

What's new

  • DeepSeek-V4.1 on GB300 gets up to 1.5x more prefill throughput under load and about 28% lower first-token latency on 32K to 64K prompts (#44403, #45059, #45107).
  • SGLang low-latency mode: the opt-in NVIDIA megakernel and AMD monokernel plugins run DeepSeek-V4.1-Flash decode as fused persistent kernels, for up to 44% faster single-user decode on GB300 and 75% on MI350X (#44945, #45450, #44353, #44621).
  • MoE models such as DeepSeek-V4.1-Flash, GLM-5.3-Flash and gpt-oss reach the first token 30% to 58% sooner on 8K prompts on H200 (#43363).
  • Paged Experts (opt-in) runs MoE models larger than GPU memory on a single consumer GPU, up to 30x faster than CPU offload (#42557, #45787).
  • SGLang now runs on Google TPU7x through SGLang-TorchTPU, a PyTorch-native plugin for TPU.
  • You can now serve LoRA adapters on quantized MoE models (FP8 and NVFP4) with a new opt-in LoRA engine (#45892).
  • DeepSeek-style MLA models running decode context parallelism can now keep their prefix cache in HiCache file-based L3 storage (#41187).

To upgrade

uv pip install --prerelease=allow sglang==0.5.23
Platform Docker
NVIDIA (CUDA 13) docker pull lmsysorg/sglang:v0.5.23
AMD MI35x docker pull lmsysorg/sglang:v0.5.23-rocm10-mi35x
AMD MI30x docker pull lmsysorg/sglang:v0.5.23-rocm10-mi30x
Intel GPU docker pull lmsysorg/sglang:v0.5.23-xpu
Intel CPU docker pull lmsysorg/sglang:v0.5.23-xeon
Google TPU7x (TorchTPU) docker pull lmsysorg/sglang:v0.1.0-tpu

Full Release Notes

Speculative Decoding

  • [Feat] speculative: Seed rejection-sampling draft proposals for deterministic inference: #33395
  • [Fix] observability: Correct speculative FPM timing and decode lengths: #38711
  • [Perf] spec: Let the FlashAttention backend capture the draft-extend CUDA graph (about 1.5x decode throughput at batch size 1 with a single-layer MoE drafter): #40762
  • [Perf] spec-decoding: fuse simulated speculative acceptance tensor preparation: #41169
  • [Fix] dsa: Fix ragged seed indices before MTP draft decode: #42565
  • [Spec] Fix DFLASH verify mask policy for wrapped attention backends: #43054
  • [Feat] spec: Capture aux hidden states on the target without a speculative draft: #43449
  • [Fix] DSv4.1: Size the encoder-replay SWA pool by the DSpark draft's stages, and keep more prefix tails: #43502
  • [Fix] dsa-eagle-topk: Copy token keys and scales when relocating index cache: #44564
  • [Fix] deepseek-v4.1: Keep the DSpark draft SWA out of the radix tree: #44673
  • [Perf] speculative: Fuse draft write-window gathering and KV translation: #44972
  • [Perf] attention: Capture translated FA4 verify metadata for DSPARK: #44973
  • [Fix] deepseek-v4.1: Start the PD decode draft SWA window at the prefill draft valid-from: #45037
  • [Fix] spec: Remove --speculative-skip-dp-mlp-sync: #45043
  • [Feat] spec: Let out-of-tree platforms provide their own speculative worker: #45144
  • [Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners: #45145
  • [Fix] mamba: Skip zero-width conv pools in the MTP verify scatter: #45212
  • [Fix] deepseek-v4: Keep the draft MoE out of the target's expert placement: #45299
  • [Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path: #45450
  • [Feat] HiCache: Support fused DFLASH KV in unified hybrid SWA pools: #45700
  • [Perf] gemma4: Capture single-step frozen-KV drafts (Gemma 4 E2B median TPOT 33% lower at concurrency 1 on H200): #45771
  • [Perf] dsv41: Trim per-step DSpark decode overhead: #46176
  • [Feat] spec: Support CUDA attention DP for DFlash and DFlash2: #46330
  • [Perf] [NVIDIA] kda: Narrower V tiles for the KDA spec-verify recurrence on SM107: #46465
  • [Perf] [NVIDIA] kimi-k3: Route draft/dense bf16 GEMMs to TGV and narrow the MLA output gate vector: #46468

Breakable CUDA Graph

  • [Perf] prefill-cuda-graph: Make prefill shared-read completion explicit and default-on: #37958
  • [Perf] deepseek-v4.1: Replay the breakable prefill CUDA graph at any context length: #43733
  • [CP] Automatically enable BCG for supported prefill CP configurations: #44022
  • [Fix] DSv4.1: Fix BCG image handling and add multimodal bounded replay coverage: #44131
  • [Fix] Qwen3.8next: Stop retaining per-size PLE outputs in breakable prefill CUDA graphs: #44150
  • [Fix] cuda_graph: Wait for in-flight layer-wise cache loads before prefill graph replay: #44220
  • [DSv4.1] Support TRT-LLM decoder bounded replay with BCG: #44326
  • [Perf] deepseek-v4.1: Run encoder SWA bounded replay in the breakable prefill graph: #44930
  • [Perf] deepseek-v4.1: Run decoder replay tail layers from CUDA graphs on eager prefill steps (saturated 8K prefill throughput +33% on GB300 at TP4): #45059
  • [Perf] deepseek-v4.1: Run decoder replay tail graphs in prefill-graph steps and under DP attention (saturated prefill +37% at TP4 and +50% with DP4 attention): #45107
  • [Fix] glm-5.2: Record side-stream work right before its join to keep CUDA-graph replay on one stream: #45349
  • [Fix] [NVIDIA] deepseek-v4.1: Fix the breakable prefill CUDA graph in the late-layer tail on SM90: #45407
  • [Perf] cuda-graph: Capture FlashInfer KDA prefill with context parallelism: #45473
  • [Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192: #45565
  • [Perf] dsv4: Let the decoder tail leave the prefill graph under DP attention with MegaMoE: #45581
  • [Fix] prefill-graph: Keep every DP rank eager when the replay tail captures behind rank barriers: #45881
  • [Perf] dsv4: Replay the V4.1 prefill graph past 16K contexts on SM90: #45941
  • [Fix] dp-attention: Align prefill graph buckets and decode padding across DP ranks: #46138
  • [Perf] deepep_v2: Reduce masked GEMM buffer memory during CUDA Graph capture: #46309

Attention Backends

  • [Perf] fmha_v2: use non-interleaved paged KV input for trtllm sm90/120 GQA prefill: #32272
  • [Fix] attention: Honor selected FlashAttention implementation: #33007
  • fix: FA4 paged-KV zero-size page-table registers on SM90: #35757
  • Fix overflow in legacy and v1 DSA top-k selectors: #37625
  • [Fix] dsa: Make the fused top-k exact on an overflowing threshold bin: #37941
  • [Fix] DSA: skip-topk pipeline handoff for owed outputs: #43832
  • [Fix] attention: Opt FlashMLA scheduler into dynamic shared memory: #44580
  • [Chore] attention: Remove the tokenspeed_mla backend and dependency: #44803
  • [Feat] attention: Add the MSA-v1 sparse backend and NVFP4 KV pool: #46141
  • [Fix] dsa: Preserve token-first sparse attention outputs: #46424

MoE & Expert Parallelism

  • [Fix] moe: Fix flashinfer TRT-LLM BF16 expert weight reload on refit: #33743
  • [Fix] eplb: Fix Qwen3 MoE output corruption after EPLB rebalance: #39580
  • [Fix] moe: Preserve negative routing sentinel in EP mapping: #40498
  • [Fix] moe_sum_reduce: gate the bf16 vectorized path on hidden % 16: #42837
  • [Perf] moe: Extend FlashInfer CUTLASS MoE autotune coverage to prefill chunks (TTFT 30% to 58% lower for 8K prompts on seven MoE models on H200): #43363
  • [Feat] FlashInfer: Add NVFP4 W4A16 and SiTU support to MegaMoE: #44036
  • [Fix] moe: gated NVFP4 MoE intermediate alignment for FlashInfer TRTLLM: #44138
  • [Fix] moe: Map expert ids for layers whose runner needs local ids under EP: #44480
  • [Fix] megamoe: Respect request-pool limits in decode validation: #44816
  • [Fix] megamoe: Support EPLB rebalance on SM100: #44847
  • [Fix] moe: Count routed_scaling_factor once when top-k folds it: #44901
  • [Fix] moe: Renormalize per-rank shared-slot routing over routed experts only: #44904
  • [Fix] [NVIDIA] modelopt-nvfp4: Keep MoE global scales in gate/up order: #45289
  • [Perf] moe: Preserve Marlin workspace and route-count budgets: #45462
  • [Perf] moe: Add H200 block-FP8 Triton configurations: #45591
  • [Fix] gguf: Preserve MoE routing, weighting and allocation budgets: #45916
  • [Fix] moe: Preserve CUTLASS routing metadata and execution budgets: #45917

Quantization

  • [Fix] quantization: Preserve BF16 NaNs in per-token FP8 quantization: #42689
  • [Fix] compressed-tensors: Serve NVFP4 W4A4 linears via FP4 Marlin when --fp4-gemm-backend marlin is set on SM100: #44213
  • [Fix] nvfp4: Own streamed tensors while waiting for paired weights: #44500
  • [Fix] [NVIDIA] modelopt-nvfp4: Read w13 global scales in the TRT-LLM [up, gate] order: #44858
  • [Perf] mlp: Fuse dense SwiGLU with block-FP8 quantization on decode (single-request decode TPOT 1.9% lower on Qwen3-8B-FP8 on B200): #45219
  • [Fix] modelopt: Correct NVFP4 TRT-LLM clamps for untied gate/up scales: #45386
  • [Fix] [SM100] quantization: Honor deep_gemm for 32x32 block-FP8 linears: #45585
  • [Feat] quantization: Add the MiniMax-M3.1 NVFP4 MoE method: #46140

Parallelism & Disaggregation

  • [4/N] elastic-ep: Scale at DP-attention replica granularity: #33728
  • [Fix] hybrid-SSM: DP attention failures at low concurrency: #34535
  • [Feat] KVShard: PD transfer and e2e test: #38473
  • [Perf] pd: Stage decode-side KV preallocation lengths through pinned memory: #38955
  • [Qwen3.8 CP 4/4] Prefill context parallel E2E support of Qwen3.8: #39724
  • [Fix] disaggregation: Clean-up and optimize the KV checksum implementation: #39729
  • [Feat] KVShard: Support kv cache sharding for MTP: #40929
  • [Fix] epd: Accept tokens-in requests on scheduler-side transfer backends: #41376
  • [Fix] PD: retraction restores racing the in-flight decode forward: #42702
  • [Perf] [NVIDIA] custom-all-reduce: Add 2shot_push all-reduce and retune SM100 TP4/TP8: #42811
  • [Feat] KVShard: Integrate GLM DSA/MTP with PD and CPU HiCache: #42973
  • [Feat] cp: Enable GLM-5.3 Flash interleave BCG with batched prefill and DFlash2: #43182
  • [Fix] nccl: Disable graph buffer registration whenever a captured graph can hold a collective: #43216
  • [Feat] NIXL: Support per-GPU UCX device selection: #43482
  • [Fix] disaggregation: Reuse startup decode graphs across PD role switches: #43760
  • [Feat] glm5.3flash: Support padded CP BCG and prefix replay with PD/DFlash2: #43774
  • [Perf] CP: QSA Context Parallel Optimization (26.5% lower QSA prefill latency at 8K with CP4): #43792
  • [Feat] dsv4.1: Support encoder SWA bounded replay with CUDA DP attention: #43813
  • [Perf] KVShard: 0SM KV Cache Allgather for KV Cache Sharding: #44029
  • [PD] Send DSA indexer state only past decode_prefix_len: #44140
  • [Fix] pd: Allow runtime role switching with DP and DP attention: #44215
  • [Feat] KVShard: Add attention and indexer support for page-sharded DSA caches: #44280
  • [Fix] scheduler: Keep PD decode non-idle while deferred KV releases are held: #44386
  • [Fix] [NVIDIA] allreduce-fusion: Decline MNNVL fusion across NVLink cliques: #44387
  • [Fix] pp: Order output send/recv by rank parity to fix cross-node hang: #44453
  • [Perf] pd: Copy prebuilt decode batch inputs to the GPU without host syncs: #44598
  • [Feat] disaggregation: Let a KV transfer backend skip the checksum for an incomparable peer: #44605
  • [Perf] dp-attention: Exchange the scheduler's MLP sync rows with gather + broadcast on gloo: #44622
  • [Perf] distributed: Cache the eager launch of custom all-reduce v2 per size: #44642
  • [Perf] NIXL: prepared dlists for hybrid-model state transfer: #44740
  • [Feat] deepseek-v4.1: Run encoder SWA bounded replay on PD prefill workers: #44952
  • [Perf] deepseek-v4.1: Default decoder SWA bounded replay on for both PD roles on CUDA: #45035
  • [Perf] deepseek-v4.1: Default PD to MegaAttention, with the KV layout in the PD handshake: #45075
  • [Feat] mooncake: Enable the failed-session probe by default with backoff and idle pruning: #45338
  • [Fix] disaggregation: Keep a requeued optimistic prefill request's multimodal inputs: #45440
  • [Fix] disaggregation: Synchronize deferred decode KV release across TP ranks: #45453
  • [Perf] pd: Send prefill KV chunks without a host sync on the scheduler thread: #45492
  • [Fix] NIXL: Preserve explicit UCX network device selection: #45722
  • [Fix] disaggregation: Restart a lost prefill's heartbeat miss count: #45922
  • [Fix] disaggregation: Poison the deferred abort ack when a NIXL or Mori write fails: #45973
  • [Feat] layer-boundary: Fuse TP all-reduce, residual add and RMSNorm on the push plane: #46137
  • [Perf] dsa: Overlap CP KV gather with indexer computation: #46179
  • [Perf] pd: Apply the prefill WAR barrier after the previous result is processed: #46220
  • [Feat] dcp: Replicate a block-FP8 q_b_proj under --dcp-replicate-q-proj: #46359
  • [Fix] [NVIDIA] dist: Detect MNNVL fabric devices by NVML fabric state, not only by GB200/GB300 name (Kimi-K3 on two-node TP8/DCP8 decodes at 485 instead of 379 tok/s): #46463

Scheduler & Runtime

  • [Fix] Fix multi-tokenizer BatchEmbeddingOutput repack dropping fields: #31892
  • [Fix] detokenizer router: Route by rid without tokenizer worker ipcs and broadcast logging config: #33715
  • perf: fix overlap scheduling for NVIDIA Confidential Computing(CC) on Blackwell: #36810
  • [Fix] tracing: Fix OTel tracing context lost in multi-tokenizer router processes: #38211
  • [Perf] weight-sync: Allocate packed receive buffers from metadata: #39264
  • [Feat] bench: support --tokenize-prompt for generated-shared-prefix: #40900
  • [Fix] Keep rope_theta when --json-model-override-args overrides rope_scaling: #43067
  • [Feat] metrics: Expose multi-detokenizer router CPU usage: #43323
  • [Perf] Cache msgspec struct fields in the batch splitter: #43520
  • [Feat] runtime: Include FlashInfer’s default tactic in startup tuning.: #43927
  • [Feat] bench: report goodput and SLO attainment in bench_serving: #43967
  • [Feat] scheduler: Gate --prefill-decode-interval on running batch size: #44001
  • [Perf] Bind scheduler CPUs to the GPU's NUMA node when memory policy is not permitted: #44023
  • [Fix] offloader: Preserve cached weights during CPU offload: #44139
  • [Perf] detokenizer: Group return IPC sends by tokenizer worker (+2.1% output throughput on DeepSeek-V4-Pro): #44288
  • [Fix] scheduler: Drop finished requests from the running batch before every admission: #44351
  • [Fix] scheduler: Count a request's held req_to_token row once at admission: #44401
  • [Fix] imports: Bind FlashInfer lazily at four import sites so the server starts without it: #44430
  • [Perf] [NVIDIA] scheduler: Pageable host staging and inline hidden-state/DSpark readback under Confidential Computing: #44568
  • [Perf] request_logger: skip per-token watermark redaction walk: #44692
  • [Feat] profiler: Add the forward pass id to detailed step span labels: #44775
  • [Fix] srt: Apply --gc-threshold in every server process and freeze every tokenizer worker: #44785
  • [Fix] scheduler: Exclude accepted storage-tier hits from the waiting prefill load: #45158
  • [Perf] detokenizer: Add SGLANG_BATCH_TOKENIZER_OUTPUTS to opt out of grouped return sends: #45280
  • [Feat] platforms: Let out-of-tree platforms bind group devices and custom ops: #45897
  • [Fix] weight_checker: Make the checksum device-agnostic: #45918
  • [Feat] metrics: Export prealloc_{full,swa}_token_usage for decode requests with a pending KV transfer: #45970
  • [Feat] benchmark: Add --input-len-list to bench_one_batch_server for per-request input lengths: #45982
  • [Fix] scheduler: Keep long prefill steps in the busy counter and exclude engine pauses: #46127
  • [Fix] request_logger: Skip watermark redaction when watermarking is disabled: #46207

Sampling

  • [Perf] logits: Write lm_head logits straight into the fp32 logits buffer: #45631
  • [Fix] grammar: Give grammar accept failures an error status: #46049
  • [Perf] logits: Cast bf16 logits into the fp32 buffer with a Triton kernel: #46167

HiCache & Radix Cache

  • [Fix] unified-memory: Bound joint reclaim and preserve cached prefixes: #39294
  • [HiCache] Add the page_unified host KV layout and the unified L3 key: #39727
  • [Feat] hicache: Expose the Mooncake L3 object-key layout for external KV routers: #40467
  • [Fix] hicache: Fail fast on a null pointer from the Mooncake host allocator: #40565
  • [Feat] unified-cache: Support out-of-tree external linkers: #40595
  • [mem_cache][PD] Fix decode radix cache ownership and allow EAGLE/EAGLE3 with decode radix cache: #40857
  • [Feat] hicache: Support MLA DCP with HiCache L3(file/mooncake): #41187
  • [Feat] unified_cache: Support hybrid SWA + Mamba models in the Mooncake direct linker: #41343
  • [Fix] hicache: Key MLA Mamba state per TP rank in the file backend: #41907
  • [Fix] HiCache: stale host KV from write-through copies racing the in-flight forward: #42701
  • [Fix] unified-memory: enabling trtllm_mha with decode graphs and speculative decoding: #42930
  • [Fix] VLM: harden Kimi image cache identity across preprocessing and radix hashing: #43043
  • [Fix] swa: Retain a whole window behind insertion pages: #43168
  • [Perf] unified-memory: Free SWA segments without a host sync: #43170
  • [Feat] unified-memory: Support asymmetric KV rows: #43325
  • [Feat] unified-memory: Fuse drafts with sliding-window and recurrent-state layers: #43326
  • [Fix] hicache: Keep unified memory off the direct page_first_direct copies: #43333
  • [Perf] hicache: Unify PP prefetch hit queries and buffer allocation: #43344
  • [Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation: #43387
  • [Fix] hicache/pd: Promise prefill only the L3 span the decode prefetch takes: #43548
  • [Feat] hicache: Export host Mamba pool occupancy next to host KV: #43727
  • [Fix] hicache: Key MLA Mamba state per TP rank in the SeaweedFS backend: #43816
  • [Fix] hicache: Key MLA Mamba state per TP rank in the HF3FS backend: #43817
  • [HiCache] Buffer-only L3 for an auxiliary SWA window; load back only missing window rows: #43932
  • [Perf] hicache: Parallel THP-backed host pool allocation (opt-in) (host pool allocation about 4x faster, about 140 s saved per start): #43978
  • [Fix] PD: Skip the cached-prefix early send in batches that start a HiCache load-back: #44081
  • [Fix] unified-memory: Avoid gather scratch during Mamba and Conv compaction: #44097
  • [Feat] hicache: Add opt-in tail-only SWA backups in buffer mode: #44173
  • [Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id: #44592
  • [Fix] hicache: Bound the storage prefetch timeout from the first blocked admission poll: #44614
  • [Fix] cache: Support DeepSeek-V4.1 external linker pools: #44623
  • [Fix] hicache: Order write-back host copies after the forward that produced the KV: #44800
  • [Feat] server-args: Add --enable-hicache alias for --enable-hierarchical-cache: #44849
  • [Fix] cache: Accept packed draft layer mappings in the UMBP linker: #44929
  • [Perf] unified-memory: Reduce allocation and SWA retirement overhead: #44970
  • [Feat] unified_cache: Support FULL + MAMBA MLA models (Kimi-K3) in the direct external linker: #44977
  • [Fix] hicache: Count the EAGLE bigram boundary token in PD decode restores: #45076
  • [Fix] hisparse: Copy DeepSeek-V4 eager backups token by token: #45104
  • [Fix] hisparse: Keep PD destination host pages until the decode KV release: #45108
  • [Perf] unified-memory: Allocate free pages from each sub-pool's own end first: #45222
  • [Fix] mem_cache: Skip linker L3 lookup on PD decode: #45538
  • [Feat] disaggregation: Verify the KV checksum under a decode-side radix cache: #46423

LoRA

  • [KV events] Publish opaque LoRA block IDs and adapter names: #42817
  • [Fix] openai: distinguish s3 served-model-name from LoRA adapter syntax: #44450
  • [Fix] lora: Preserve MoE projection identity and routing lifetimes: #45681
  • [Feat] lora: Add plan-driven dense and MoE LoRA engines: #45892

Multimodal

  • [Fix] inkling: Return 400 instead of 500 for unfetchable/undecodable multimodal input: #40899
  • [Feat] runtime: Support tokenized multimodal generation: #42833
  • [Fix] glm4v: Clamp request video budgets to the deployment limit: #43871
  • [Perf] session: Cache multimodal prompt token counts across turns: #44112
  • [Fix] multimodal: Preserve caller hashes and offsets for precomputed inputs: #44160
  • [Fix] nemotron_vl: Preserve tokenized prompts during media expansion: #44184
  • [Fix] nemotron_vl: Pad visual spans using processor offsets: #44581
  • [Fix] multimodal: Clip pad spans to a truncated prompt: #46638

Model Support & Optimizations

  • [Fix] deepseek-v4: Fix DeepSeek-V4 non-EP TBO for attention TP > 1: #33250
  • [Fix] dLLM: Preserve prompt tails and literal mask tokens across prefix reuse: #39592
  • [Fix] dsv4: Skip DP-attention pad rows in the engram hasher and low-ratio sources: #40424
  • [Fix] dLLM: Size decode graphs to the default max running requests: #40941
  • [Fix] deepseek-v4.1: Match KV capacity to physical layouts: #41275
  • [Fix] deepseek-v4.1: Keep the full SWA window for short bounded-replay extends: #41303
  • [DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode.: #41561
  • [Fix] deepseek-v4: Refresh attn_sink after loading weights: #41630
  • [Feat] MiniCPM-V-4.7: add model support: #41993
  • [Fix] qwen3.8: Fix batch-invariant kernels for Qwen3.8 Flash-Next: #42087
  • [Perf] deepseek-v4.1: Skip wasted layer 20 work on earlier prompt tokens under decoder bounded replay: #42901
  • [Perf] dsv4: Vectorize paged KV cache dequantization: #42929
  • Fuse MiMo verification RoPE and KV cache writes (E2E throughput +4.6%): #43106
  • [DSv4.1] Fold encoder-SWA replay into the extend forward: #43166
  • [Perf] gdn: Select FlashInfer prefill for page-major pools: #43171
  • [Perf] dsv41: Integrate DeepGEMM MegaGate routing: #43262
  • [Fix] mamba: Release consumed lazy Mamba prefix locks before donation retry: #43559
  • [Fix] kimi-k3: Preserve pipeline graph bank inputs and bound chunk profiling: #43604
  • [Perf] Enable native MXFP4 routing and CuTeDSL fusion for MiMo (decode throughput +11.8% with DFlash): #43639
  • Support EPLB for Nemotron-H: #43685
  • [Perf] dsa: Route the k-pool indexer top-k through the top-k v2 kernel: #43775
  • [Perf] DeepSeek-V4.1: Run the small-batch decode fusions at TP2: #43838
  • [Perf] deepseek-v4.1: Integrate FlashMLA MegaAttention with NVFP4 KV: #43922
  • [DeepSeek-V4.1] DeepSelect cluster top-k for long rows and bf16 index logits: #44051
  • [Fix] glm5_next: Correct num_experts_per_tok default to 8: #44093
  • [GLM-5.3-Flash] Drop the NextN layer from the KDA state pool: #44098
  • [Perf] [NVIDIA] qsa: Keep sparse decode scratch in FP8 for FP8 KV pools: #44114
  • [Fix] DeepSeek-V4.1: Align paged indexer score rows for DeepSelect decode: #44210
  • [Perf] DSv4.1: Enable decoder bounded replay by default (about 28% lower first-token latency at 32K and 64K input): #44403
  • [Fix] nemotron: Preserve nested Puzzle configs: #44487
  • [Feat] nemotron: Port Nemotron 3.5 Super serving changes: #44518
  • [Perf] [NVIDIA] muse-spark-open: Fuse pre-expert RMSNorm with static-scale NVFP4 quantization: #44554
  • [DSv4.1] Fuse request-window KV copies (median E2E latency about 61% lower with encoder SWA bounded replay on a high-load replay on 4x GB300): #44555
  • [Perf] deepseek_v4: size the masked silu-mul-quant grid to the GPU, not the padded capacity: #44618
  • [Perf] DeepSeek-V4.1: Raise the SM100 swap-AB decode cap to 40 rows: #44632
  • [Feat] kernels: Add muse spark operators and collective GEMM (2/5): #44665
  • [Feat] srt: Add muse spark runtime, multimodal support and tests (3/5): #44666
  • [Fix] bailing: MoE gate dtype follows config.router_dtype: #44715
  • [Perf] DSv4.1: Skip discarded KV rows during window commit (median E2E latency 7% lower on 4x GB300): #44783
  • [Fix] engram: Avoid blocking history copies during PD admission: #44787
  • [Perf] deepseek_v4: fuse block-FP8 shared experts into MegaMoE for any 32-aligned block size: #44837
  • [Perf] deepseek_v4: fold the MegaAttention query layout into wq_b for block-FP8-as-MXFP8 checkpoints: #44838
  • [Perf] deepseek_v4: run FlashInfer autotune for block-FP8 models served as MXFP8: #44842
  • [Fix] deepseek-v4.1: Serve prompt-logprob requests under decoder SWA bounded replay: #44850
  • [Perf] mamba: Fuse strided state updates and replay index translation: #44971
  • [Fix] DSv4.1: Support CUDA image requests with encoder bounded replay: #45048
  • [Feat] dsv4.1: Return prompt logprobs under encoder SWA bounded replay: #45050
  • [Fix] nemotron: Correct weight-export failures and GEMV dispatch: #45082
  • [Fix] kimi-k3: Avoid vision attention child-name collisions: #45110
  • [Perf] mamba: Copy tracked decode states at HBM bandwidth in one persistent launch: #45213
  • [Feat] kernels: Beam model support: #45254
  • [Fix] inkling: Keep BF16 LM-head operands with FP32 output: #45335
  • [Feat] Inkling: render and parse chat via tml-renderers: #45340
  • [Fix] muse_spark: support unified KV memory with TRT-LLM: #45356
  • [Feat] systemone: serve Kev pointer head decision models: #45413
  • [Fix] mamba: Keep prefill rows' checkpoint tracking through mixed chunked prefill: #45435
  • [Perf] scheduler: Plan hybrid Mamba extend batches without host-device syncs: #45472
  • [Perf] [KDA] muse_spark: Run the one-token router GEMV on CuTe DSL: #45508
  • [Perf] dsv4: Restore sparse prefill for request-window rows (+9.6% throughput per GPU at 32K tokens per rank on GB300): #45573
  • [Perf] dsv41: Use tuned MXFP8 tactics at decode sizes: #45630
  • [Perf] dsv41: Mega-mHC and the wo_a einsum write MXFP8 for wqkv_a and wo_b: #45632
  • [Perf] dsv41: Fuse the q_norm MXFP8 quantize for wq_b at every row count: #45633
  • [Perf] [KDA] muse_spark: use BLOCK_M 240 for BF16 MegaMoE prefill shards: #45637
  • [Perf] QSA: Optimize the prefill indexer gather: #45752
  • [Perf] QSA: Optimize prefill indexer in long cached prefixes: #45791
  • [Perf] [NVIDIA] deepseek-v4.1: Align BF16 prefill indexer score width: #45794
  • [Perf] dsv4: Score the V4.1 index logits in bf16 on SM100: #45880
  • [Perf] dsv41: Serve token-interleaved V4.1 KV caches with sgl-flash-mla: #45893
  • [Perf] dsv4: Use the fused MXFP8 producers in eager prefill: #45903
  • [Perf] dsv4: Optionally stage V4.1 engram rows from the shared host table on the scheduler: #45921
  • [Feat] minimax: Support MiniMax-M3.1-Flash: #45925
  • [Perf] moe: Run the Marlin GPT-OSS SwiGLU through the fused Triton kernel: #46134
  • [Perf] minimax-sparse: Speed up MiniMax-M3 sparse attention on Hopper: #46135
  • [Perf] dsv41: Let Mega-mHC write the MegaMoE input under MegaGate: #46163
  • [Perf] dsv41: Read Engram rows from peer device shards: #46180
  • [Perf] [NVIDIA] deepseek-v4.1: Default aggregated servers to MegaAttention only when each rank holds 64 query heads (P90 interactivity up to +27% at TP4): #46263
  • [Perf] [NVIDIA] kimi-k3: FP32 warp split-K GEMM for the MoE front projection on SM107: #46464

Kernel Library

  • [Fix] attn_residual: int64 program ids in the score/combine kernels (int32 overflow at 37,451 tokens): #43438
  • [Perf] async-probe: Fuse asynchronous NaN validation into one kernel: #43720
  • [Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins: #44945
  • [Perf] qknorm: run head_dim-128 rows two per warp with 16B lanes, bit-identical: #45214
  • [Fix] [NVIDIA] jit: Build sm_100f on CC 10.7 until the bundled CUTLASS supports SM107: #45911

OpenAI-Compatible API

  • [Feat] parser: Enrich chat_parsing streaming events: #40478
  • [Feat] parser: Add checkpoint-driven response-template adapters: #40479
  • [Fix] glm47: Disable tool grammar for tool_choice="none": #41233
  • [Fix] parser: Accept whole-number floats for integer tool arguments: #43941
  • [Fix] chat-template: Pass return_dict=False to apply_chat_template(tokenize=True) for transformers 5: #44020
  • [Fix] tokenizer_manager: re-report only emitted customized_info keys on cumulative streams: #44537
  • [Feat] http_server: Make the multi-worker ASGI app path configurable: #44583
  • [Fix] metrics: record HTTP request metrics in tokenizer workers: #44663
  • [Fix] muse: Support Responses namespace tools and defer_loading: #44685
  • [Fix] Inkling: frame constrained output in native message tokens: #44716
  • [Fix] dsv41: Preserve tool-result image order and current-turn thinking: #44751
  • [Perf] function_call: stop rescanning unclosed GLM tool calls per chunk: #45143
  • [Feat] glm-5.3: Render GLM-5.3 and GLM-5.3-Flash chat prompts with a Python chat template (prompt rendering 4.7x to 17x faster): #45154
  • [Feat] entrypoints: add a chat encoding adapter registry: #45300
  • [Feat] openai: Render DeepSeek-V4/V4.1 chat prompts with deepseek-recipe: #45315
  • [Fix] http_server: Make SageMaker /ping reflect server readiness: #45445
  • [Feat] srt: Balance uvicorn tokenizer workers with per-worker SO_REUSEPORT listeners: #45482
  • [Fix] http_server: Apply SGLANG_TIMEOUT_KEEP_ALIVE to Granian HTTP/1.1 connections: #45760
  • [Feat] openai: parse DeepSeek output through deepseek-recipe: #45907
  • [Fix] minimax-m3: Type nested tool-call arguments and count self-opened reasoning: #46133
  • [Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe: #46219

Rust Server

  • [Feat] rust/grpc: Add ListModels, Detokenize, and Abort to api.v1: #42118
  • [Fix] sglang-server: Keep the matched stop token in output_ids: #42637
  • [Feat] sgl-router: Serve native gRPC ChatComplete: #42674
  • [Feat] sgl-router: Serve the remaining native gRPC inference RPCs: #42681
  • [Feat] sgl-router: Separate LoRA block IDs from prompt lookup keys: #42818
  • [Feat] sgl-router: Add --dp-rank-policy round_robin for --dp-aware: #43511
  • [Feat] sgl-router: Render Kimi-K3 through sglang-processor: #43608
  • [sgl-router] Render GLM-5.3 and GLM-5.3-Flash through sglang-processor: #43625
  • [Feat] sgl-router: Render Qwen3.8 (Max, 27B, Flash-Next) through sglang-processor: #43689
  • [Rust] Serve /weights_checker from the Rust frontend: #43926
  • [Rust] Serve scheduler metrics on the Rust frontend's /metrics: #43942
  • [Fix] sgl-router: Send disagg_prefill_dp_rank to decode under --dp-aware: #44072
  • [Fix] sgl-router: Resend on a closed pooled connection; add --disable-circuit-breaker: #44075
  • [Fix] sglang-server: Honour extra_key on the Rust frontend's /generate: #44207
  • [Feat] sgl-router: skip engines at an admission cap in power-of-two; forward x-sglext headers: #44218
  • [Fix] sglang-processor: Follow Python's OpenAI layer through #43027, #37577, #43006, #43026, #43005: #44243
  • [Feat] sgl-router: Fix cache-aware routing for MTP/EAGLE on hybrid (Mamba/KDA) models: #44252
  • [Fix] sgl-router: Receive kv-indexer-bridge event batches outside tokio's coop budget: #44305
  • [Feat] rust-server: Apply the x-override-* request headers: #44312
  • [Feat] rust-server: Attach configured metadata headers to a healthy /health_generate: #44315
  • [Feat] sgl-router: Add --decode-dispatch-limit to pick decode DP ranks near KV arrival: #44402
  • [Feat] rust-server: Report cached_tokens and reasoning_tokens in meta_info: #44531
  • [Feat] sglang-server: serve /v1/loads from the Rust frontend: #44532
  • [Fix] rust-server: Honour cache_salt and require_reasoning on /generate: #44533
  • [Feat] rust-server: Add HiCache storage management endpoints: #44726
  • [Feat] rust-server: Report Python's launch-time /server_info fields from the Rust frontend: #44764
  • [Feat] sglang-server: Serve /flush_cache from the Rust frontend: #44766
  • [Feat] rust-server: Serve /v1/tokenize and /v1/detokenize from the Rust frontend: #44767
  • [Fix] sgl-router: Keep the kv-indexer connection through bursts of cancelled prefix queries: #45100
  • [Feat] sgl-router: Add --pd-stream-after-prefill: #45102
  • [Feat] sgl-router: Route prefill buckets by uncached tokens: #45103
  • [Fix] sgl-router: Refuse prefill workers that cannot compute before bootstrap under --decode-dispatch-limit: #45120
  • [Feat] rust-server: Let the launcher set the default max_new_tokens for /generate: #45431
  • [Fix] rust-server: Drop the fixed 64-item multimodal cap in favor of --limit-mm-data-per-request: #45432
  • [Feat] rust-server: Render, tokenize and serve chat through sglang-processor: #45449
  • [Fix] rust-server: Answer health probes a busy scheduler skipped on the response ring: #45475
  • [Fix] rust-server: Match Python's chat layer on templates, streaming and lowered-chat errors: #45659
  • [Fix] sgl-router: Do not abort forwarded requests on reqwest's default 30 s TCP_USER_TIMEOUT: #45696
  • [Feat] rust-server: Serve /abort_request from the Rust frontend: #45888
  • [Fix] rust-server: Deliver scheduler-side aborts to Rust frontend clients: #45899
  • [Feat] rust-server: Let a multimodal processor finalize item layout after caller hashes: #45914
  • [Fix] rust-server: Keep a transfer backend's own bootstrap port on prefill: #46066
  • [Fix] rust-server: Start a transfer backend's own KV bootstrap server on prefill: #46074
  • [Feat] rust-server: Report weight versions and media token counts in generate meta_info: #46205
  • [Feat] rust-server: Let a multimodal processor check and fetch media during admission: #46206
  • [Feat] rust: Render DeepSeek-V4/V4.1 prompts with deepseek-recipe in sglang-processor and sgl-router: #46243
  • [Feat] sgl-router: Confirmed-first pending prefixes for cold prompts, with a follower cap: #46505
  • [Feat] rust_server: Run --sidecar once per node: #46635
  • [Feat] grpc: Advertise media inputs in GetServerInfo: #46882

Simulator

  • [Feat] simulator: Add baseline collection and ML latency predictor training tools: #44299

SGLang-Diffusion

  • [Perf] diffusion: fuse Wan VAE DupUp3D shortcut add: #32860
  • [Feat] diffusion: selectable sequence-parallel strategies for LingBot causal attention: #34385
  • [Perf] diffusion: Accumulate chunked LoRA merges with addmm (1.7x faster chunked LoRA merges (geometric mean)): #37620
  • [Diffusion] Fuse the Wan VAE conv bias epilogue and tile the RMSNorm+SiLU kernel: #38650
  • [Feat] diffusion/model: support Cosmos Dreams and Cosmos Multiview: #39475
  • [Fix] diffusion/lora: revalidate FLUX.2 FP8/NVFP4 fast paths after LoRA attach: #40171
  • [Diffusion] Qwen-Image 2.1 VAE decoder fast paths: bit-exact norm/bias fusions by default, channels_last decode under quality=extra-high: #41340
  • [Perf] diffusion: fused QKV norm + RoPE, cached RoPE tables for FLUX 3 video VAE: #41894
  • [Perf] flux2: Build the single-block output projection without a cat: #41943
  • [Perf] diffusion: fuse FLUX3 SwiGLU with rowwise FP8 quantization: #42758
  • [Diffusion] Bound frame_interpolation_exp to RIFE's supported range: #43014
  • [Diffusion] comfyui: consolidate 6 unrelated plugin fixes: #43493
  • [Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin: #43536
  • [Feat] comfyui: video-to-video input and prompt enhancement for server-mode nodes: #43673
  • [Feat] comfyui: Apply ControlNet residuals in Flux integrated mode: #43698
  • [Diffusion] cache-dit: respect CFG parallel for dual-transformer models: #43704
  • [Fix] diffusion: keep ZMQ ingress sockets on their configured port: #43732
  • [Fix] wan: fix opt-in Wan2.2 progressive mode and one-step transitions: #43768
  • [Feat] VSA-H3: support sm80 (A100) with a raw-pointer Triton load path (FastH3 on A100 in 59.6 s instead of 85.0 s): #43823
  • [Diffusion] comfyui: grouped SGLDOptions, native model loader and model-aware preflight: #44058
  • [Diffusion] Z-Image: replace padded rows without a host sync: #44069
  • [Feat] wan-animate-2: Support the distilled model: #44085
  • [Fix] wan-animate-2: Size the in-context slot from both input grids: #44090
  • [Perf] diffusion: convrot_int8 comfy_kitchen issues one GEMM per layer by default: #44179
  • [Perf] MiniMax-H3: fused indexed modulation for FP32 AdaLN tables: #44180
  • [Perf] diffusion: fused gate/up + SwiGLU MLP on comfy_kitchen INT8 for Qwen-Image 2.1: #44196
  • [Fix] diffusion: Invalidate merged LoRA cache after adapter rewrites: #44204
  • [Fix] flux: use true_cfg_scale as the FLUX CFG weight: #44260
  • [Fix] diffusion: Align Qwen-Image 2.1 resolution and handle RGBA JPEG output: #44275
  • [Fix] diffusion: Fix VMoBA chunk selection bugs: #44286
  • [Feat] diffusion: Add support for the MOSS-Image family: #44325
  • [Feat] diffusion: support FLUX 3 Image Klein (peak memory 54.3 to 29.5 GiB with a 2048x2048 reference on GB300): #44418
  • [Fix] diffusion: stabilize FLUX.3 serving and reuse static conditioning: #44493
  • [Feat] diffusion: Add opt-in pi0.5 completion-boundary action batching: #44634
  • [Perf] diffusion: Cache the fused linear-GELU verdict per linear: #44643
  • [Feat] diffusion: Per-task prompt enhancer config and the FLUX.3 Klein upsampler protocol: #44662
  • [Perf] [NVIDIA] diffusion/FLUX.3: Run dynamic LoRA through PrimsTS GEMMs: #44761
  • [Fix] diffusion: Unmerge LoRA layers the next adapter does not cover: #44770
  • [Feat] diffusion: Add worker-side ComfyUI compatibility patches: #44841
  • [Feat] ComfyUI: Add online INT8 quantization for BF16 Qwen-Image-2.1 single files (about 1.4x faster than BF16 on L40S with half the DiT weight memory): #44865
  • [Fix] diffusion: runtime utils bug fixes: #44887
  • [Fix] diffusion: Prevent int32 overflow in Qwen-Image-2.1-Turbo VAE RMSNorm: #45118
  • [Perf] diffusion/minimax-h3: Batch the video VAE decoder tiles and fuse the output chain: #45179
  • [Feat] diffusion/disaggregation: Split FLUX.3 prompt and image encoders: #45309
  • [Perf] cosmos3: optimize v2v preprocess performance: #45398
  • [Feat] diffusion: Support FastH3 Omni (Ref2VA 8-step PDD): #45530
  • [Perf] diffusion/preprocess: Extend exact GPU input processing across pipelines: #45733
  • [Feat] diffusion/openai: Report per-step progress for async video jobs: #45919
  • [Fix] diffusion/ltx: Keep tiled VAE decode off the sharded convs under parallel decode (E2E 4.7% lower and 1.7 GB less memory per rank at 768x512): #45942
  • [Fix] diffusion: Keep dynamic LoRA deltas on fused linear-GELU sites: #45943
  • [Perf] diffusion: Shard LTX-2 stage 2 over the CFG ranks as Ulysses: #45965
  • [Perf] diffusion: Cut LTX-2 denoising-step host overhead: #45966
  • [Perf] diffusion: Remove per-step and per-request host syncs in Cosmos3 and Qwen-Image-2.1: #45967
  • [Perf] diffusion: Turn on breakable CUDA graphs by default for LTX-2/LTX-2.3 two-stage serving: #45977
  • [Perf] diffusion: Keep every component resident in auto mode when memory allows: #45978
  • [Perf] diffusion: Serve image responses from in-memory encoded bytes: #45987
  • [Fix] diffusion: Step MOVA latents outside the serial-CFG branch: #45993
  • [Fix] diffusion: Give every SP rank the same Wan TI2V timestep: #45994
  • [Fix] diffusion: Offset LTX-2.3 BCG video RoPE by the SP shard start: #45995
  • [Fix] diffusion: Fail closed after local worker loss or multi-rank OOM: #46025
  • [Fix] diffusion/zimage: Stop serving a previous request's RoPE tables: #46044
  • [Perf] diffusion: Keep shared denoising paths off the host and add host-sync test guards: #46045
  • [Perf] diffusion: Remove per-step host syncs and uploads in image models: #46046
  • [Perf] diffusion: Remove per-step host syncs and uploads in video models: #46047
  • [Fix] diffusion/server: Clean up rejected and cancelled video requests: #46057
  • [Fix] diffusion/api: Publish image inputs atomically: #46060
  • [Fix] diffusion/storage: Join uploads before request cleanup: #46068
  • [Fix] diffusion/mesh: Guard asynchronous job publication and defaults: #46081
  • [Fix] diffusion/loader: Preserve checkpoints on read failures: #46095
  • [Perf] diffusion: Write uncompressed PNG responses without Pillow's row filters: #46099
  • [Perf] diffusion/qwen-image-2.1: Keep an image prefix's K/V in the packed buffer across steps: #46101
  • [Perf] diffusion: Serve FA4 head sizes on Blackwell instead of falling back to sm80 SDPA: #46102
  • [Fix] diffusion/disaggregation: Keep decoded images in tensor transport: #46104
  • [Perf] diffusion: Project LTX-2 prompt K/V once per denoising loop: #46105
  • [Perf] diffusion: Drop LTX-2's all-valid prompt mask under lossless quality: #46106
  • [Perf] diffusion: Route dense attention with a sub-tile side to FA4 on Blackwell: #46107
  • [Perf] diffusion: Run LTX-2 stage 2 on a premerged resident DiT: #46108
  • [Perf] diffusion: Build one LTX-2.3 AdaLN row per sample when every token shares the sigma: #46109
  • [Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate): #46110
  • [Perf] diffusion: Vectorize the LTX-2 Q/K RMSNorm + split-RoPE kernel: #46111
  • [Fix] diffusion/disaggregation: Isolate malformed request and result messages: #46120
  • [Feat] diffusion: add Hy-Nomas offline and realtime support: #46124
  • [Fix] diffusion/server: Clean up workers on startup failure: #46142
  • [Fix] diffusion/disagg: Recompute automatic topology per role: #46144
  • [Perf] diffusion: Prefer FA4 over cuDNN SDPA for long self-attention on B300: #46146
  • [Perf] diffusion: Speed up Wan's spatial-parallel VAE decode: #46149
  • [Fix] diffusion/openai: Bound job lists and complete cursor pagination: #46156
  • [Fix] diffusion/scheduler: Isolate invalid IPC requests: #46157
  • [Fix] diffusion/api: Do not publish failed generation outputs: #46162
  • [Fix] diffusion/http: Allow bounded graceful shutdown: #46169
  • [Perf] diffusion: Encode Cosmos3 video while it decodes: #46172
  • [Perf] diffusion: Run Cosmos3's multi-GPU VAE decode on the lossless fast path: #46173
  • [Perf] diffusion: Build spatial-parallel causal conv inputs in one pass: #46174
  • [Perf] diffusion: Walk several row chunks per warp in the CuTe-DSL norm kernels: #46227
  • [Perf] diffusion: Fuse Wan's q/k RMSNorm with its interleaved RoPE: #46228
  • [Perf] diffusion: Split LTX tiled VAE decode across the decode-parallel ranks: #46237
  • [Perf] diffusion: Validate breakable CUDA graphs for LTX-2.5 two-stage serving: #46239
  • [Perf] diffusion/worker: Move cyclic GC out of request execution: #46493
  • [Fix] diffusion: Resolve cached named weights without the Hub API: #46679

Local & Desktop AI

  • [Feat] paged-experts: Serve MoE models whose weights do not fit VRAM (12x to 30x faster single-stream decode than --cpu-offload-gb on a 16 GB card): #42557
  • [Perf] [NVIDIA] fp8_fa_sm120: Use the block-scaled MMA encoding (1.47x faster at the kernel level): #43390
  • [Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM: #43556
  • [Feat] paged-experts: AWQ, compressed-tensors, MXFP4, NVFP4, W8A8 and block-INT8 experts: #45787
  • [Fix] mlx: Accept draft_parallel_layout in the stub's alloc_memory_pool: #46441

AMD / ROCm

  • [Perf] [AMD] attention: Enable diffusion BF16 attention and MXFP4 compilation: #27702
  • [Feat] [AMD] disagg: Support DSV4 decode radix cache on ROCm unified KV (bf16/fp8, DSpark) (long-context KV transfer 68% to 72% smaller): #30929
  • [Fix] [AMD] custom-allreduce: Stop registering oversized inputs in the deterministic all-reduce: #34543
  • [Fix] [AMD] dsa: Fix AITER prefill with an FP8 KV cache: #39083
  • [Perf] [AMD] quark: Add opt-in AITER ASM FP4 GEMM for Quark dense linear: #39598
  • [Perf] [AMD] dsv4: fp8 target lm_head GEMM for DSPARK verify logits: #39622
  • [Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader: #40776
  • [Perf] [AMD] attention: Spread uneven Triton decode batches across XCDs: #41229
  • [Fix] [AMD] mamba: Keep a chunked prefill cut on the mamba chunk grid so DCP can donate checkpoints: #41701
  • [Perf] [AMD] quark/Qwen3.8: Quantize the BF16 shared expert of Quark MXFP4 checkpoints to MXFP4 while loading for shared-expert fusion (+4.5% output throughput): #41899
  • [Perf] [AMD] dsv4: reuse the q_lora fp8 quant for the C4 indexer wq_b (4/8): #41903
  • [AMD][Diffusion] Route unquantized linears through aiter tgemm on gfx1250 (opt-in with SGLANG_USE_ROCM_AITER_TGEMM=1; Wan2.2 denoising 3.55x faster on MI450): #41988
  • [Perf] [AMD] qwen3.5: Fuse activation quant into small-M FP8 block-scale GEMM on gfx950 (Qwen3.5 interactivity 5% to 10% higher at TP8): #42229
  • [Fix] [AMD] hicache: Serve encoder-replay prefix hits with a paged DSpark draft SWA: #42716
  • [Perf] speculative: Parallelize EAGLE spec-decode KV index building over token blocks (flashinfer draft decode; verify/draft-extend for flashinfer, flashinfer_mla, aiter): #42803
  • [Fix] [AMD] aiter: Fix garbage decode output for non-sliding-window models on the vectorized_5d KV layout: #43335
  • [AMD] Use the preallocated EAGLE topk=1 draft chain on HIP: #43640
  • [Perf] [AMD] dsa: Run DSA prefill under breakable CUDA graphs on HIP: #43642
  • [Perf] [AMD] dsa: Enable the gfx950 fused DSA indexer for block-FP8 wq_b/wk (GLM-5.3 MXFP4 P50 interactivity +6.8%): #43643
  • [Fix] [AMD] diffusion: Compare ReduceOp by identity so the compiled TP all-reduce stays one graph: #43770
  • [Fix] [AMD] deterministic: Refuse RCCL all-reduce in deterministic mode above TP2: #43809
  • [Fix] [AMD] diffusion: Prevent ComponentResidencyError after compile warmup: #43952
  • [Fix] [AMD] kda: Match the fused verify Q/K norm to its reference kernel: #43991
  • [Perf] [AMD] dsv4: fp8 verify writes the SWA ring in the fused norm/rope kernel (2/8): #44003
  • [Perf] [AMD] moe: small-batch sort with histogram + bitonic sort for 32 < P <= 256 (3/8) (decode step about 4% lower): #44004
  • [Perf] [AMD] dsv4: FP4 indexer on aiter's row-group kernel for decode and target-verify (5/8): #44006
  • [Perf] [AMD] dsv4: optional e8m0 mxscale GEMM for dense 128x128 fp8 linears (6/8): #44007
  • [Perf] [AMD] dsv4: fp8 unified_kv decode/verify on FlyDSL v4 nm decode with a latency-model split plan (1/8) (TP8 decode step 3.1% to 3.9% lower): #44016
  • [Fix] [AMD] activation: Run ReLU2 natively on ROCm: #44198
  • [Fix] [AMD] qwen3: Fix KV cache write in the fused QK-norm + mRoPE decode path (Qwen3-VL GSM8K 0.005 to 0.942): #44266
  • [Perf] [AMD] qwen3: Fuse QK-norm + RoPE + KV cache write for dense Qwen3 decode (TPOT up to 10.5% lower): #44267
  • [Feat] [AMD] monokernels: one runtime switch for vendor monokernel plugins (DeepSeek-V4.1-Flash single-request decode +75% with DSpark): #44353
  • [Perf] [AMD] dspark: No per-step host sync in decode (1/4) (DeepSeek-V4.1-Flash DSpark TPOT 6.8% lower at concurrency 1): #44405
  • [Fix] [AMD] mori: Look up elided DSA state entries by the paired dst index: #44412
  • [Perf] [AMD] monokernels: Let a plugin skip per-step attention metadata: #44601
  • [Fix] monokernels: Honor resolved CUDA graph configuration: #44630
  • [Perf] [AMD] dsv4: Add a separate MegaMoE decode instance: #44657
  • [Fix] [AMD] diffusion: Keep gfx1250 FlyDSL BF16 attention out of torch.compile: #44671
  • [Fix] [AMD] diffusion: Accept residual and skip_bias in the ROCm Conv3d-to-Conv2d VAE patch: #44672
  • [Perf] [AMD] deepseek-v4: Skip top-k refinement for proven positive-zero ties (P90 interactivity +8.2%): #44927
  • [Fix] [AMD] deepseek-v4.1: Reject attention DP with encoder SWA bounded replay on ROCm: #44934
  • [Perf] [AMD] deterministic: Faster extend attention launch on gfx950 (deterministic-mode prefill 21% to 130% faster on long prompts): #45098
  • [Perf] [AMD] attention: Restore the gfx950 extend tile for head_dim <= 128 (+28% throughput at 32K input with the Triton backend): #45099
  • [Perf] [AMD] dsv41: run gfx950 MXFP8 dense GEMMs on aiter's preshuffled FlyDSL kernel (DeepSeek-V4.1-Flash decode step up to 4.8% lower on MI355X): #45123
  • [Perf] [AMD] dsv41: faster mHC boundary Triton tile for 96+ rows, bitwise unchanged: #45124
  • [Fix] [AMD] grok: accept allow_patterns_overrides in presharded _prepare_weights patch: #45217
  • [Fix] [AMD] diffusion: Enable --quantization fp8 on gfx1250 (torch.compile lowering + per-channel weight scales): #45304
  • [Perf] [AMD] dsv41: Skip unneeded layer 20 computation under decoder-bounded-replay: #45339
  • [Fix] [AMD] gpt-oss: support EP for gfx1250 W4A8 MXFP4 MoE: #45365
  • [Fix] [AMD] dsv41: Set moe act to fp8 by default: #45480
  • [Fix] [AMD] attention: Pass the layer softcap to aiter unified-attention decode: #45531
  • [Perf] [AMD] kda: Use a HIP chunked-prefill kernel for KDA on gfx950 (Kimi-K3 single-request TTFT 11.9% lower at 1K input): #45710
  • [Perf] [AMD] kda: Use a HIP fused decode kernel by default for KDA (Kimi-K3) on gfx950: #45713
  • [Perf] [AMD] Qwen3-Next: fused QK Gemma RMSNorm + partial RoPE on gfx950: #45739
  • [Perf] [AMD] GPT-OSS: Pick the gfx1250 W4A8 MoE backend, in-kernel GEMM 1 gather (opt-in with SGLANG_GFX1250_W4A8_KERNEL_GATHER=1; gpt-oss-120b W4A8 output throughput +30%, TTFT 40% lower): #45758
  • [Fix] [AMD] diffusion: Fix the first-step denoise regression from compiling ENABLE_CK=0 BF16 attention: #45838
  • [Fix] [AMD] rotary: Pass a query-dtype cos_sin_cache to the HIP fallback: #46161
  • [Fix] [AMD] MiniMax-M3: default SGLANG_MINIMAX_M3_INDEX_TOPK_FREQ to 1: #46340
  • [Perf] [AMD] diffusion: Route large BF16 linears to the FlyDSL a16w16 GEMM on gfx1250 (FLUX.2-dev 1.21x to 1.28x faster end to end on MI450): #46366
  • [Fix] [AMD] aiter: Fix garbage output with an fp8 KV cache and calibrated KV scales: #46528

NPU / Ascend

  • [NPU] [Fix] Fix rope sin cos pre-calculation for Qwen3.5 during Pipeline Parallelism: #29753
  • [Fix] [NPU] qwen3.5: Fix dense serving on Ascend 950: #32745
  • [Feat] [NPU] llguidance: add bitmask path for Ascend: #36074
  • [Perf] [NPU] attn: add not needs_cpu_seq_lens for dsa modules: #38461
  • [Fix] [NPU] scope: fix dtype error in sampling backend ascend: #39609
  • [Fix] [NPU] attention: Use FIA TND for MLA prefix attention and add Gemma norm fallback: #40131
  • [Fix] [NPU] glm5.2: support pcp v2 with interleave & zigzag: #40165
  • [Fix] [NPU] dspark: Fix attn_tp > 1 draft wo_b all_reduce group: #40399
  • [Fix] [NPU] requantization: gate arch35 block-FP8 -> MXFP8 requantization: #40435
  • [Fix] [NPU] qwen: Fix shared-expert routing with Ascend FuseEP: #40516
  • [Perf] [NPU] deepep: enable MXFP8 low-latency dispatch for Fp8MoEMethod FP4 experts (MiMo-V2.5-Pro with DFlash mean TTFT 21.8% lower): #40519
  • [Perf] [NPU] mem_cache: Add optional ScatterPa KV cache writes: #40535
  • [Fix] [NPU] qwen: adapt pcp-v2: #41564
  • [Fix] [NPU] glm5.2-mtp: L2 cache adaptation: #41642
  • [Fix] [NPU] mla-pool: DCP allocator page and index-K bound, short-pool warning: #42976
  • [Fix] [NPU] pre_warm_nccl: init npu backend before the parallel runtime touches a device: #43039
  • [Perf] [NPU] glm5.2: Support dynamic INT8 MLAProlog for attention (decode latency 8.4% lower): #43469
  • [Fix] [NPU] disaggregation: Fix host-pool backup for decode retraction: #43814
  • [Fix] [NPU] kimi-k3: skip fused gmm1 for situ activation in Ascend runner: #43837
  • [NPU][Fix] use CANN libdevice for DSpark Markov sampling: #43849
  • [Perf] [NPU] mem_cache: Increase GLM-5.2-W4A8C8 KV cache capacity with compact DSA indexer allocation: #44436
  • [Perf] [NPU] scheduler: Remove blocking seq_lens D2H from graph replay: #45170
  • [Perf] [NPU] AscendAttnBackend: D2H-free block-table gather for mirrorless DSA graph replay: #45949

CPU / Intel / XPU

  • [Feat] [Intel] hicache: Enable L1<->L2 support on XPU: #32503
  • DSPARK enable for XPU: #33739
  • [CPU]: Add group-aware CPU SHM collective kernels: #35435
  • [Feat] [CPU] model: Support LTX and MOVA models on CPU: #36632
  • [Perf] [XPU] usp: Enable packed Ulysses QKV all-to-all: #39989
  • [Perf] [XPU] fp8: Optimize FP8 scale layouts: #40052
  • [Fix] [CPU] memory: Honor cgroup memory.max when sizing CPU memory: #40066
  • [Feat] [Intel] disaggregation: Support XPU for staging-buffer KV transfer: #40861
  • [Fix] [XPU] runtime: Fix XPU rotary embeddings and tokenizer padding: #41213
  • [Perf] [XPU] diffusion: Use the AOT SYCL fused residual-norm-scale-shift kernel: #41401
  • [Perf] [XPU] flux2: Fuse bit-exact LayerNorm and gated residual norm: #41942
  • [Perf] [XPU] diffusion: Avoid QKV copies in flash attention: #41944
  • [XPU] Use the shared fused decode-metadata kernel in the intel_xpu ba…: #42122
  • [Feat] [CPU] Grok-model: enable grok-2 model on xeon cpu: #43341
  • [Perf] [XPU] server_args: Enable fused QK-norm + RoPE by default: #43345
  • [Feat] platform: Add initial T-HEAD SAIL platform support [2/N]: #43826
  • [Fix] [XPU] gemma3: Enable bidirectional image attention: #44225
  • [Feat] [CPU] model: Support Muse models on CPU: #44624
  • [Feat] [CPU] model: Support Cosmos3 series models on CPU: #45478
  • [Fix] [CPU] memory: budget overestimation caused by filtering empty NUMA nodes: #45516

Security

  • [Fix] model_loader: Avoid unsafe pickle fallback when loading PyTorch checkpoints: #42760
  • [Fix] diffusion: OpenAI API hardening: #43585
  • [Fix] utils: Allowlist torch.bcomplex32 in SafeUnpickler: #44440
  • [Fix] sgl-router: Bump rustls to 0.23.46 for RUSTSEC-2026-0346..0349: #44509
  • [Fix] srt: Admit only allowlisted peers on TCP-bound ZMQ IPC sockets: #45250
  • [Fix] srt: Pin the source address of TCP ZMQ connects for the peer allowlist: #45437
  • [Fix] sgl-router: Bump rustls-webpki and webpki-roots for emSign advisories: #45526
  • [Fix] sgl-router: Bump tokio for RUSTSEC-2026-0386: #46260

Cookbook Updates

  • DeepSeek-V4 and DeepSeek-V4.1: a DeepSeek-V4.1-Flash 1017 variant, PD with DSpark, refreshed AMD recipes for V4 and V4.1 (including the PD image and c1 capacity fix), and Ascend A3 / 950 single-node recipes: #44937, #44848, #44368, #44454, #44903, #45070, #45297, #45693, #45602
  • SemiAnalysis InferenceX AgentX: launches for Muse Spark Open, GLM-5.3 and MiniMax-M3.1-Flash, and refreshed points for GLM-5.2, Qwen3.5, Qwen3.8-Flash-Next and DeepSeek-V4.1-Flash on B300, GB200 and GB300: #44065, #44228, #44654, #44712, #45039, #45111, #45311, #45495, #45513, #45749, #46005, #46506, #46547
  • Muse Spark Open: a new page with GB300 Dynamo deployments, an MI355X AgentX recipe and tuned GB300 BF16 recipes: #44527, #45629, #45689, #45422, #45429
  • MiniMax-M3.1: a new page with GB200 recipes and a B200 low-latency monokernel recipe: #45924, #46546
  • Beam 1.0: a new page with Python and Docker commands: #44694, #45991
  • MiniCPM-V 4.7: DSpark benchmarks refreshed at the final model PR: #44310
  • FLUX.3 Image Klein: an expanded model description and a prompt upsampler section: #44515, #44512
  • Cosmos3 and diffusion guides: prompt enhancer tasks, Cosmos recipe links, reference resizing, LoRA auto mode and image profiling: #45705, #44377
  • GLM-5.2 and GLM-5.3: Ascend deployment recipes for GLM-5.2, GLM-5.3 and GLM-5.3-Flash, and a TPU7x recipe for GLM-5.3-Flash: #43372, #45597, #45642, #45326
  • Qwen3.5, Qwen3.6 and Qwen3.8: Qwen3.5 MI355X FP8 agentic tuning with HiCache write-through, an RTX Spark Qwen3.6 NVFP4 recipe, Ascend A3 scenarios for Qwen3.6, FP8 KV and the QSA indexer by default for Qwen3.8-Flash-Next, and the released Qwen3.8-Max W4A8 checkpoint on Ascend: #46017, #46032, #44707, #38678, #45245, #45574
  • Jais 3: a new page (runs on the K2 Horizon architecture): #42766
  • Ornith-1.5: a new page (runs on the Qwen3.5 architectures): #45455
  • New pages for models that run on preview images in this release: Ling-3.1-flash (#44301, #44307), Nemotron 3.5 Super VL (#44510, #44587), Step 5 Preview (#44357) and Recursive Flint: #45625
  • TPU: the README and TPU guide introduce SGLang-TorchTPU next to SGL-JAX: #45246
  • Ascend NPU docs move to CANN 9.1.0: #44177, #44432, #44695

Dependencies

  • FlashInfer 0.7.1 to 0.7.2: #44591
  • sglang-kernel 0.4.9 to 0.4.10 (#43922), sgl-deep-gemm 0.2.2 to 0.2.3 (#44666) and then 0.2.3.post2 (#45933), and the new sgl-flash-mla 1.0.0: #45893
  • New dependencies: deepseek-recipe 0.2.0 for DeepSeek-V4 / V4.1 chat (#45315) and tml-renderers 0.1.0 for Inkling chat: #45340
  • tokenspeed_mla is no longer installed (#44803) and cuda-tile is dropped: #44393
  • OpenAI SDK 2.6.1 to 2.8.0 (#45291), uvicorn >= 0.51 (#45482), timm 1.0.16 to 1.0.25 (#46620), python-dateutil < 3 to keep botocore resolvable (#45857), and sgl-eval 0.1.3: #44131
  • Docker: CUDA images ship the sgl-nvidia-mega megakernel plugin, inert unless --enable-nvidia-megakernel is set (#44945, #46030); AWS EFA image variants (#41006) and numactl in runtime images for SGLANG_NUMA_BIND_V2: #43883
  • ROCm: a ROCm 10.1 image for gfx1250 alongside ROCm 10.0 (#42699), the sgl-amd-mono monokernel plugin in the ROCm image (#44621, #45389), AITER pins 2bb549102 (#44379) and 285579b62 for gfx942 / gfx950 (#45498), and Mooncake v0.3.14-rc1 for cross-node KV transfer: #44404
  • XPU: oneAPI 2026.1.3 and sglang-kernel-xpu 0.4.0 to 0.5.0: #43989
  • MLX: mlx >= 0.32.4: #44367
  • T-HEAD SAIL: a new srt_sail extra (in pyproject_other.toml) for the initial platform support: #43826

Breaking Changes & Upgrade Notes

  • DeepSeek-V4 and V4.1 chat now follows deepseek-recipe. Prompts, outputs and the native /separate_reasoning and /parse_function_call endpoints go through DeepSeek's official encoder, and SGLang's own encoding_dsv4.py / encoding_dsv41.py are gone. Clients may notice: streamed tool-call arguments arrive in pieces (the first chunk carries arguments: ""), so concatenate them as the OpenAI SDK does; tool-call ids use the official call_00_... format; V4 reasoning effort follows DeepSeek's official profile and the dsv4_reasoning_effort_profile override is removed; a numeric reasoning_effort on V4 returns 400; developer messages render as system messages; /parse_function_call also returns calls to tools outside tools with tool_index -1; requests that combine tools with continue_final_message return 400. Output parsing goes through deepseek-recipe when the DeepSeek reasoning or tool-call parser is set; sgl-router and sglang-server keep their Rust parsers for now: #45315, #45907, #46219
  • DeepSeek-V4.1 decoder SWA bounded replay is on by default for supported CUDA configurations, including both PD roles. It is not numerically identical to full prefill. Pass --no-enable-decoder-swa-bounded-replay to keep full prefill, for example for RL rollouts whose logprobs must match the trainer: #44403, #45035
  • DeepSeek-V4.1 defaults to MegaAttention under PD. On SM100 / SM103 both PD roles now resolve flashmla_mega and the PD handshake checks the KV layout, so prefill and decode must run the same attention backend (#45075). Upgrade prefill and decode together, since an older decode cannot read the new handshake field; pass --dsv4-attn-backend flashmla on both roles to keep FlashMLA. Aggregated servers keep FlashMLA when attention is tensor-parallel across ranks: #46263
  • The tokenspeed_mla attention backend is removed. tokenspeed_mla still starts as a deprecated alias for cutedsl_mla with a warning. To switch, pass --attention-backend cutedsl_mla --kv-cache-dtype fp8_e4m3, plus --speculative-attention-mode decode with speculative decoding: #44803
  • --speculative-skip-dp-mlp-sync is removed. Launch commands that pass it now fail with an unrecognized-argument error; drop the flag: #45043
  • The kitchen_int8 diffusion quantization name is removed. Use convrot_int8, deprecated in favor of it since v0.5.22: #45706
  • SGLANG_ENABLE_UNIFIED_RADIX_TREE is now ignored without a warning. It already had no effect because the unified radix tree is the default; drop it from launch scripts: #45090
  • Legacy tar-format .pt checkpoints no longer load. The unsafe weights_only=False pickle fallback is removed; convert such checkpoints to safetensors or zip-format .pt offline: #42760
  • Diffusion HTTP endpoints validate more strictly. URL values for upscaling_model_path and frame_interpolation_model_path return 400, an unsupported response_format is rejected before generation, and a failed cloud upload returns 502 instead of 400: #43585
  • The first startup of a FlashInfer CUTLASS MoE server takes longer. Startup autotune now also covers prefill sizes, so a server with no saved tuning results takes longer to become ready (on DeepSeek-V4.1-Flash on 8x H200, autotune goes from about 41 s to 177 s). Later startups reuse the saved results, and --disable-flashinfer-autotune skips tuning: #43363
  • Diffusion PNG responses are larger. PNG output now defaults to zlib level 0: the same pixels, about 3x the bytes, several times faster to encode. Set output_compression to a value other than 75 to pick a level: #45987
  • sgl-router moved to the top-level sgl-router/ directory. Builds and scripts that point at experimental/sgl-router/ must update the path (#45414); --max-in-flight is now --max-inflight-requests, and the old name still works as an alias: #44950

New Contributors

Full Changelog: v0.5.22...v0.5.23

来源: SGLang releases · github.com

开源生态产品更新#产品更新#开源/仓库