SGLang and Miles Add Day-0 Support for DeepSeek-V4.1

1. Architecture overview

DeepSeek-V4.1 introduces several architecture choices that shape the serving stack.

Low-ratio compression and sliding-window attention (SWA). Each layer maintains an fp8 sliding-window cache for its most recent 128 positions, while selected source layers produce an fp4 compressed representation for long-range attention. Beyond the first two layers, each query attends jointly to its local window and up to 512 compressed positions selected by the indexer. Layers 2-19 use pairwise compression, so the KV source layers must retain incomplete pairs across decode steps.

Compressed KV and retrieval selections are shared across layers, as described in section 2.

Manifold hyper-connections (mHC). Each sublayer reads from and writes to four parallel residual streams through token-dependent mixing coefficients. A sublayer consumes the coefficients produced by its predecessor, allowing the next coefficient projection to overlap with the current attention or FFN computation.

Engram memory. Layers 1 and 14 retrieve rows keyed by hashed token n-grams from two large fp8 tables and gate the selected rows into the residual stream. Each step reads only a small number of rows, making table placement and lookup overhead more important than dense compute.

2. Cross-layer sharing and sparse retrieval

Shared KV and indexer keys. Four KV source layers produce compressed KV and indexer keys. Consumer layers read these from the most recent KV source, avoiding separate storage and generation at each consumer. Window KV remains specific to each layer.

Shared candidates for two-level selection. Layer 20 selects up to 2,048 blocks of eight positions per query, always retaining the block containing the newest position. It publishes these candidates for later index source layers, while selecting its own top-512 from all reachable positions. Later index source layers restrict their top-512 to the shared candidates. When all reachable positions fit within the 16,384-position budget, the candidate filter excludes none of them.

Cross-layer selection reuse. Eight index source layers score indexer keys with their own queries and select up to 512 compressed positions per query. The remaining compressed-attention layers reuse the most recent selection without running an indexer.

Figure 1. Layer roles and cross-layer sharing.

3. Engram optimization

Two fp8 tables are 189 GiB of weight that every decode step touches for a handful of rows. Row-sharding them across tensor-parallel ranks costs an all-reduce per lookup and pins a quarter of the tables in each GPU's HBM at TP4 to hold data that is read at kilobytes per token.

3.1 Host-memory placement

Host offload moves the Engram tables out of GPU memory, freeing capacity for KV cache. Dequantization, gating, and value projection remain on the GPU. SGLang supports two host layouts with different communication costs.

In the shared layout, every TP rank accesses one complete host-resident copy of the tables and gathers the rows it needs. This removes row sharding and the lookup all-reduce. In the private layout, each rank holds a host-resident shard and the lookup all-reduce is retained.

Random access to large tables can be limited by address translation. Huge-page backing reduces this overhead. In the evaluated GB300 container, private anonymous mappings supported huge pages while shared mappings did not. The automatic layout selection therefore chose private shards, trading the retained all-reduce for faster host lookups.

3.2 Performance evaluation

In paired tests on 4x GB300 (TP4/EP4), host offload increased KV cache capacity by 36%, with comparable decode throughput and TTFT. These runs used private host shards with huge-page backing and retained the lookup all-reduce. All 28 greedy probe completions matched the baseline.

Host placement is opt-in (SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1).

Figure 2. Engram placement and lookup communication at TP4.

4. SWA bounded replay

Sliding-window KV is specific to each layer. Retaining it for prefix reuse consumes cache capacity, while full-sequence prefill computes window states that subsequent decoding no longer accesses directly. The model's deployment note proposes bounded replay to reduce these storage and computation costs.

4.1 Encoder-side bounded replay

Standard prefix reuse requires cached compressed KV, indexer keys, and a valid sliding-window checkpoint. Encoder-side bounded replay removes the checkpoint requirement. The prefix cache retains compressed KV and indexer keys, while each active request maintains a 128-position window.

On a cache hit, SGLang recomputes the cached prefix's final 128 tokens to reconstruct window KV. Cached compressed KV and indexer keys remain unchanged. This exchanges bounded recomputation for lower cache storage and enables prefix reuse without a window checkpoint at the matched position.

4.2 Decoder-side tail-only computation

Layer 20 is the final compressed-KV source. Layers 21-39 reuse its compressed KV and indexer keys while computing their own window KV. For each prefill chunk, SGLang executes layers 0-20 over all tokens and layers 21-39 over at most the final 128 tokens of each request.

Layers 0-20 retain full-context computation. In the late layers, local attention is restricted to the retained tail because window KV preceding that boundary is not computed.

Figure 3. Encoder-side reconstruction and decoder-side tail-only prefill.

4.3 Correctness boundaries and results

Both modes truncate local attention at the reconstruction boundary. Recomputed hidden states can therefore differ from full prefill, even when cached compressed KV is unchanged. Bounded replay is an approximation whose quality must be evaluated empirically.

In paired tests with batches of eight 8K-token prompts, decoder-side replay raised prefill throughput by 1.56x on 8x H200 and 1.37x on 4x GB300. On 4x GB300, the paired AIME 2026 evaluation measured the same pass@1 with replay off and on (453/480 correct samples each).

Both modes are opt-in (--enable-encoder-swa-bounded-replay, --enable-decoder-swa-bounded-replay) and can be combined. Encoder replay excludes speculative decoding. Decoder tail-only computation does not support input logprobs or full prompt hidden-state capture.

5. Kernel and execution optimizations

mHC execution and numerical consistency. Predecessor pre-mix allows the mixing coefficients for the next sublayer to be computed alongside the current attention or FFN. SGLang overlaps this work and fuses mixing-statistic reductions with Sinkhorn iterations. The reductions use a fixed order independent of batch size, keeping each token's mixing coefficients consistent across batch compositions. For small batches, the HC=4 post-mix is tiled across hidden dimensions to expose more parallel work.

FP4 indexing and TP layout. Indexer scoring reads directly from the FP4 cache. Indexer heads are replicated across TP ranks because sharding this MQA-shaped operation would not reduce key bandwidth. It would instead require an all-reduce of scores whose size grows with context length.

Fusion with preserved quantization semantics. The indexer combines RoPE, FP4 quantization, and packing or cache writes in fused kernels. Fusion must retain the intermediate rounding and scaling of the original computation: removing an intermediate memory write does not permit removing its numerical effect. This reduces launches and intermediate memory traffic while preserving the quantization sequence.

Low-ratio compression. Compressor projections use BF16 checkpoint weights with FP32 accumulation and output. Ratio-2 decode pooling is fused into one kernel that combines each completed pair of positions. Compression and indexing also overlap with attention preparation once their inputs become available, joining before attention consumes their outputs.

Single-token projection and Engram fusion. The grouped output projection uses a specialized BF16 matrix-vector kernel for single-token inputs, where the general matrix-multiplication path has low GPU utilization. Fused Engram gating reduces FP32 temporary storage, while n-gram hashing is computed in one kernel. These optimizations target the small projections and sparse memory operations that recur at every decode step.

6. Reinforcement learning in Miles

Miles provides a Megatron-Core plugin for DeepSeek-V4.1 and uses SGLang for rollouts. The training backend implements the model's shared attention state, mHC, and Engram memory. A central objective is to minimize the difference between trainer and rollout log-probabilities for the same responses.

Parallelism and shared state. The backend supports DP, TP, SP, EP, PP, and CP. TP and SP partition attention projections and compressor groups. Pipeline boundaries carry all four mHC residual streams, predecessor mixing coefficients, and attention state still needed by downstream consumers. This state also lets recomputed layers recover their inputs locally. Context parallelism keeps queries local while gathering window KV, compressed KV, and indexer keys across ranks for global sparse retrieval.

Quantization-aware training. The training forward reproduces the serving engine's FP4 rounding for compressed latents and indexer queries and keys, and FP8 rounding for the window cache. Straight-through gradients allow optimization through these discrete operations. The window cache uses the engine's paged-cache kernels for the forward values, while a differentiable emulation carries the gradient. Sparse attention and indexing reuse the DeepSeek-V4 plugin's TileLang kernels; RoPE and fake quantization follow the serving formulas.

Routing replay. Rollout Routing Replay feeds the sampled expert assignments into the trainer's MoE layers, preventing routing ties from selecting a different expert path. Indexer top-k is recomputed rather than stored and replayed: once its quantized inputs matched, replay added no measured parity benefit. This avoids retaining the per-layer selections for every rollout token.

Numerical consistency. Compressor gates, normalization statistics, mHC mixing, Engram gating, and attention-sink accumulation use FP32, with casts at operation boundaries. The compressed-KV projection's gradient all-reduce also uses FP32. Deterministic reductions and matrix-multiplication settings make repeated forwards reproducible, helping separate execution variability from persistent trainer–rollout differences.

Colocated training and rollout. Training and rollout alternate on 16 GPUs. Optimizer moments stream to node-local NVMe, and trainer state is offloaded before the rollout engines resume. Updated BF16 weights are transferred in buckets after each training step. The frozen FP8 Engram tables use host-memory backing and are excluded from weight synchronization. The backend loads the Hugging Face checkpoint directly through its model bridge.

6.1 The validated run

Figure 4 shows steps 0–80 of a DAPO run on DAPO-Math-17K with a 2K-token response cap on 16 GB300 GPUs (TP4, EP16, 128 samples per step). The first and last five-step mean rewards are 0.51 and 0.78. Over the plotted interval, per-token KL between trainer and rollout ranges from 0.0012 to 0.0017, and the mean absolute log-probability gap ranges from 0.017 to 0.025 nats.

The measured discrepancy remains small without sustained growth during this run. These measurements do not isolate its cause or establish numerical equivalence. The run completed 120+ steps without a failure.

Figure 4. DAPO reward and trainer–rollout discrepancy over steps 0–80.

7. Acknowledgments

We thank the DeepSeek team for DeepSeek-V4.1, and the SGLang and Miles contributors and reviewers for the model integration, optimization, and validation work described in this post.