Qwen3.8-Flash-Next: Day-0 Support in SGLang

Introduction

Today, the Qwen team open-sourced Qwen3.8-Flash-Next, a multimodal MoE model and an early preview of the Qwen4 architecture. It plays the same role for Qwen4 that Qwen3-Next played for Qwen3.5. The Gated DeltaNet + Gated Attention hybrid design has been used from Qwen3.5 through Qwen3.8. In collaboration with the Qwen, NVIDIA, and AMD teams, SGLang provides day-0 support for the model.

Qwen3.8-Flash-Next upgrades the architecture in several areas:

• GDN + QSA hybrid attention: Gated DeltaNet (GDN) efficiently compresses the history, while Qwen Sparse Attention (QSA) uses a lightweight indexer to select important context at micro-block granularity, keeping long-sequence attention costs low.
• Gated Residual (GR): widens the residual stream into 4 branches and controls reads and writes with a dynamic gate, strengthening cross-layer information flow.
• N-gram Embedding: performs lookups based on the local context, providing additional representations for common phrases and local patterns, expanding model capacity with very little extra computation.

Highlights:

• Hybrid architecture: a 125B-parameter main model, supplemented by an additional 51B N-gram Embedding, with 6B parameters activated per token. 48 layers in total: 36 GDN linear attention layers and 12 QSA sparse attention layers. MoE layers use 512 experts with top-10 routing.
• An NVFP4 checkpoint we quantized: RadixArk/Qwen3.8-Flash-Next-NVFP4, released day-0.
• N-Gram Embedding: offloading the N-gram embedding to host memory greatly reduces GPU memory usage, and asynchronous prefetching overlaps it with model computation at almost no extra cost.
• Gated Residual, built with NVIDIA and shipped through FlashInfer: high-performance Mix/Combine HyperConnection operators via a low-latency single-GEMM path (2.05× kernel-level speedup).
• GDN+QSA: KV cache memory management for the GDN+QSA hybrid architecture, compatible with Radix Cache.
• Speculative Decoding: an index-reuse feature for the MTP draft model, cutting the draft model's indexer time at long context lengths. At TP4 on B200, the NVFP4 checkpoint decodes at 540 tok/s for batch size 1 with MTP, at an accept length of 3.3 (which includes the bonus token).

Launch commands and per-workload configuration guidance live in the SGLang Cookbook.

Model Architecture

• GDN+QSA Hybrid architecture: Following the architectural design introduced in Qwen3.5, Qwen3.8-Flash-Next adopts a GDN + Attention hybrid architecture: out of every 4 layers, 3 GDN layers compress history into a fixed-size state, while the remaining layer performs precise retrieval over the full context. For the global Attention layers, Qwen3.8-Flash-Next further introduces Qwen Sparse Attention (QSA) to address the fact that both computation and KV cache memory-access cost grow substantially as the context lengthens. Sparse attention reduces long-sequence computation by attending only to important context. QSA goes one step further: it aggregates the sequence into micro-blocks, estimates importance at the block level, and then selects the most relevant regions, reducing both the indexing overhead and the attention cost at the same time.
• Gated Residual (GR): combines two ideas: following Hyper-Connection, it widens the residual stream into multiple branches; and it brings GatedNorm-style element-wise dynamic gating into the residual read. The original single residual stream is expanded into 4 parallel branches, allowing the model to decide dynamically, based on the current content, how much information to read from each branch and how much to write back.
• N-gram Embedding: Lookups are performed using the local context formed by "the current token plus several preceding tokens", providing additional representations for common phrases and local patterns while adding almost no per-token compute overhead. The N-gram Embedding can reside entirely in host memory to save GPU memory: lookup positions are computed in advance and asynchronously prefetched, so it never permanently occupies GPU memory. In the end, the model uses only a single N-gram Embedding layer near the start of the network, adding a large-scale "local-pattern memory" at relatively low cost.
• IndexShare MTP: the QSA top-k selection computed by the draft-extend pass over the tokens the target just accepted is held for the whole MTP iteration, so every draft decode step skips the indexer and reads that frozen selection plus the positions drafted since. At long context lengths, this substantially speeds up the MTP draft steps.

Qwen Sparse Attention: Retrieve Coarsely, Attend Precisely

Qwen3.8-Flash-Next uses compressed QSA with a compression ratio of 4, or c4. Each QSA layer has two paths: a lightweight indexer decides where to look, while sparse GQA reads the selected entries from the original attention K/V cache.

The indexer projects four 128-dimensional query heads and one shared key head. Every four raw index keys are averaged in FP32, normalized, and rotated with the first token's MRoPE position to form one compressed key. A query scores the visible compressed blocks with

$$s_{t,b} = \frac{1}{\sqrt{128}} \sum_{h=1}^{4} \mathrm{ReLU} \left(\left\langle q^I_{t,h}, \bar{k}^I_b \right\rangle\right).$$

QSA keeps the best 512 blocks, expands them back to 2048 logical token positions, and appends the zero-to-three tokens in the current incomplete block. The final sparse attention therefore sees at most 2051 positions. Importantly, the compressed keys are only an index: the final softmax and value aggregation use the original, uncompressed K/V.

This means QSA trades a small amount of cache capacity for much lower long-context compute and memory traffic. The indexer scans roughly $L/4$ small keys, then sparse attention reads about 2K full K/V entries instead of all $L$. The model-level KV saving comes from the hybrid layout: only 12 of 48 layers store growing attention K/V, while the other 36 GDN layers use fixed-size state, not from discarding K/V inside a QSA layer.

SGLang attaches the indexer only to the full-attention layers and reuses their MRoPE implementation. The original K/V stays in the normal paged pool. QSA adds one BF16 compressed index key per four tokens; the raw keys for the unfinished block live in a four-slot per-request ring. This avoids retaining raw index keys for the full context and reduces QSA's index-cache overhead by 80%. Page-aligned full_slot / 4 addressing lets the compressed cache follow Radix Cache ownership without a separate lifecycle.

For prefill, a custom GPU kernel computes the index scores, a fast top-k selects the blocks, and Triton expands the indices and runs sparse GQA. Decode uses a paged version of the same scorer, compacts the selected original K/V, and dispatches to TRTLLM-Gen on Blackwell or packed FlashAttention otherwise. The indexer can overlap the main Q/K/V projection on a second CUDA stream, and the metadata paths are CUDA-graph compatible.

IndexShare MTP: Reusing the QSA Selection Across Draft Steps

A QSA layer runs an indexer that picks which tokens to attend, then a sparse attention over exactly those tokens. The second stage has a fixed token budget; the first scores its query against all ⌈L/4⌉ compressed blocks, so beyond a few thousand tokens the indexer, not the attention it feeds, is what sets the cost of the layer. Speculative decoding multiplies it: with --speculative-num-steps N, one MTP iteration spends N indexer invocations (N - 1 draft decode forwards plus one draft-extend) to advance the draft by at most N positions.

So the draft decode steps stop running the indexer altogether. Every MTP iteration opens with a draft-extend over the tokens the target just accepted, and that pass runs the indexer anyway; each request's last accepted row is captured there and reused by the whole draft loop, with N + 1 extra columns filled at lookup with the positions drafted since the capture, so the draft still sees its own in-flight tokens. The selection is a list of logical token indices and a request only ever grows, so it can never go out of range; and because the query has moved by at most N positions out of L, the reused ranking is essentially the one the indexer would have recomputed; accept length is unchanged. The draft's indexer work per MTP iteration drops from N invocations to one. The small metadata kernels that exist only to feed it, including the compressed decode view and the pending-ring and group-ring layouts, are removed from the draft decode step as well.

HyperConnection Kernel Optimizations

HyperConnection (HC) maintains four parallel residual streams, while Attention and MoE operate on a single hidden state. Each block therefore uses Mix to read from the four streams and Combine to write its output back. Here, M is the number of tokens processed by one call: it is small during decode and speculative verification, but can reach thousands during prefill. We dispatch to different kernels according to M.

Mix

Mix uses a low-rank projection to generate element-wise gates and reduce the four residual streams into one hidden state. For M ≤ 16, we use the low-latency split-K CuTe GEMM from FlashInfer PR #4266. Split-K partitions the K dimension so multiple CTAs can process the same output region in parallel, compensating for the limited M-dimension parallelism. SiLU, Sigmoid, gating, and the final reduction are fused into the two GEMM epilogues, avoiding intermediate writes to global memory. The up-projection weights are reordered offline so the four gate values for each output can be reduced locally inside a tile. For larger M, the implementation uses cuBLAS, which is more efficient at these shapes.

On NVIDIA B300 at M = 4, the fused path reduces Mix latency from 12.36 to 6.03 µs, a 2.05× kernel-level speedup. In an end-to-end speculative-decode benchmark against the previous Triton path, throughput improves by 7.6%.

Combine

Combine computes four injection coefficients and applies a residual update to the four streams. For large M, one fused kernel processes each token row in a single pass. At small M, this mapping exposes too few CTAs, so the M ≤ 32 path splits each row along the hidden dimension. The resulting two-kernel implementation provides enough parallelism while preserving the reference FP32 accumulation order and bitwise-identical outputs.

At M = 4, the split path reduces Combine latency from 4.17 to 2.13 µs, a 1.96× kernel-level speedup. In a separate end-to-end benchmark against the original one-CTA-per-row kernel, throughput improves by 5.49%. For large M, the fused kernel is up to 2.54× faster than the cuBLAS-based baseline and reaches 6144 GB/s of effective bandwidth.

Shape-aware dispatch lets HC use the appropriate execution path for both low-latency decode and large-scale prefill.

Per-Layer Embeddings (PLE)

Architecture

This model places PLE, a hash-addressed learned N-gram embedding memory, at the second decoder block (configured layer ID 2, corresponding to zero-based index 1). Its 51.2 billion embedding parameters, about 95.4 GiB in BF16, are fixed model weights rather than KV cache or mutable attention memory.

For token x_t, eight 2-gram hash heads use (x_{t-1}, x_t) and eight 3-gram hash heads use (x_{t-2}, x_{t-1}, x_t), producing 16 embedding row IDs. Each row contributes 160 values, which are concatenated into E_t with shape [2560].

PLE at the second decoder block. Sparse N-gram retrieval is gated into four HC branches before HC Mix. SGLang moves the vocabulary-parallel table shard to pinned host memory and gathers only the 16 selected rows per token.

$$E_t \in \mathbb{R}^{2560} \longrightarrow K_t \in \mathbb{R}^{4 \times 2560}, \qquad V_t \in \mathbb{R}^{2560}$$ $$R_t \in \mathbb{R}^{4 \times 2560} \longrightarrow Q_t \in \mathbb{R}^{4 \times 2560}$$ $$g_t = \mathrm{Gate}(\mathrm{Norm}(Q_t), \mathrm{Norm}(K_t)) \in \mathbb{R}^{4 \times 1}, \qquad U_t = g_t \odot V_t$$ $$\Delta_t = U_t + \mathrm{SiLU}(\mathrm{DWConv}(\mathrm{RMSNorm}(U_t)))$$ $$\widetilde{R}_t = R_t + \Delta_t, \qquad \widetilde{R}_t \xrightarrow{\mathrm{HC\ Mix}} h_t \in \mathbb{R}^{2560}$$

The fourth line forms the PLE delta by adding the gated value to its short-conv output; the fifth injects that delta into the HC state. PLE keeps two request-local states: the two recent token IDs used for hashing and a short-conv history of shape [10240, 9]. The target model retains PLE during prefill, decode, and target verification; only the one-layer MTP draft model disables it.

Sparse Pinned-Host Offload

Because each token touches only 16 rows, SGLang keeps each rank's vocabulary-parallel table shard in pinned host memory and gathers the selected rows into a small BF16 GPU buffer with a Triton UVA kernel. A dedicated CUDA stream overlaps the gather with the first decoder block. The existing TP reduction and DP gather/scatter paths are preserved: offload changes storage location, not table ownership or PLE math. This CUDA path is enabled by default when the effective model dtype is BF16 and remains separate from KV-cache or generic layer offload.

On H200 with TP4 and MTP-213 (2 draft steps, top-k 1, and 3 draft tokens per target verification), offload reduced target-model weights from 83.91 to 60.45 GiB per GPU (-23.46 GiB) and increased allocated KV capacity from 1.84M to 3.28M tokens (+78.54%) at the same memory fraction. With 1, 2, and 4 concurrent requests, matched throughput was effectively unchanged (-0.07% geometric mean). Four fixed prompts with 128 generated tokens each matched exactly in output IDs; the recorded chosen-token logprob trace for the first case also matched exactly.

Acknowledgments

This work was a collaboration among the SGLang team at RadixArk, Qwen, NVIDIA, and AMD.

SGLang Community: Qiaolin Yu, Yuhao Yang, Cheng Wan, Xinyuan Tong, Zijie Xia, Ke Bao, Mingyi Lu, Haoguang Cai, Banghua Zhu, Ying Sheng

Qwen: Yi Zhang, Yizhong Cao, Guangda Liu

AMD: Andy Luo, Haichen Zhang

NVIDIA: NVIDIA and SGLang jointly optimized Qwen3.8-Flash-Next performance on Blackwell and Hopper.