Unified Radix Cache: One Tree for Hybrid Model Prefix Caching

Introduction

Prefix caching reuses KV when requests share the same token prefix. Under full attention, once the KV for a shared prefix is computed, it remains valid as more tokens are appended. A later request with the same prefix can reuse those KV entries instead of recomputing them during prefill. SGLang tracks this mapping with a radix tree keyed by token sequences. Before prefill, the tree finds the longest reusable prefix and returns its KV locations to the scheduler.

Hybrid models break this single reuse rule. A request may combine full attention KV, sliding window attention KV, and recurrent states, each with a different notion of reuse. Full attention KV remains reusable across the entire matched prefix. Sliding window attention KV covers only the trailing window. A recurrent state is valid only at an exact prefix checkpoint. They share the same token prefix, but not the same reusable boundary. Forcing a single boundary across all of them either discards valid reuse or permits invalid reuse.

These reuse rules appear in different combinations across model families. Encoding each combination as a specialized cache class creates a combinatorial class matrix, especially once orthogonal capabilities such as HiCache are added. Earlier implementations followed this pattern, duplicating matching, insertion, locking, and eviction logic across cache variants.

Unified Radix Cache addresses this combinatorial design by separating shared prefix identity from component-specific reuse validity. A single token-keyed radix topology provides a canonical coordinate for each prefix, while full attention KV, sliding window attention KV, and Mamba checkpoints attach as components. HiCache is native to the same component lifecycle, extending component identity across GPU L1, Host L2, and an external L3 tier. Auxiliary pools can follow components as sidecars without defining new reuse boundaries. New model families can compose these capabilities without introducing another cache tree.

Highlights

• One tree replaces the cache class matrix. Full attention KV, sliding window attention KV, and Mamba checkpoints share one radix topology while components enforce distinct reuse semantics.
• Hooks keep the tree core generic. Components control matching, splitting, insertion, locking, and eviction, so new hybrid combinations do not require new tree implementations.
• HiCache is native to the component lifecycle. Components and sidecars retain the same prefix identity as they move across GPU L1, Host L2, and an external L3 tier. In the multi-turn benchmarks, L3 kept hit rates in later rounds near 98% on DeepSeek-V4-Flash and 96.8% on Inkling-Small.
• Session activity guides eviction. Session-aware eviction favors cache entries from active sessions without pinning them. In the SWE-bench runs, the session-aware Unified Radix Cache configuration recorded 2.9% to 16.6% lower TTFT than ordinary HiRadixCache with LRU.
• The experimental Rust tree core reduces long prefix overhead. In the sliding window benchmark, the prototype recorded up to 42% lower TTFT over turns 176 to 200 than the Python tree.

One Tree, Composable Components

Hybrid models combine cacheable values that share the same token prefix but follow different reuse rules. Unified Radix Cache maps the shared prefix to one radix topology and each reuse rule to a TreeComponent. UnifiedTreeCore runs the common matching, splitting, insertion, locking, and eviction mechanics. UnifiedRadixCache coordinates pool operations, while each component defines only the semantics that vary.

The FULL component is always present. SGLang adds the SWA component for hybrid sliding window attention and the MAMBA component for hybrid recurrent layers. For example, DeepSeek-V4 composes FULL and SWA, Kimi-K3 composes FULL and MAMBA for its KDA recurrent state, and Inkling composes all three components on the same tree. A new model family can reuse an existing component composition. If it introduces a reuse rule that the current set cannot express, SGLang can add a new TreeComponent without creating another tree implementation.

FULL provides path reuse. It keeps KV for every token in the matched prefix and protects the corresponding ancestor path. SWA provides window reuse. It requires a contiguous trailing window, while older SWA slots may be empty tombstones whose radix nodes remain in the shared topology. MAMBA provides checkpoint reuse. It requires one recurrent checkpoint at the reusable frontier and copies shared state into a private request slot before mutation. These components apply different rules to the same candidate boundary.

Finding a Safe Reuse Boundary

During prefix matching, UnifiedTreeCore follows the canonical FULL path and treats each visited node as a candidate boundary. A FULL match alone is not sufficient. Every active component creates a validator, and the reusable boundary advances only when all validators accept the candidate. A rejection does not stop traversal because a component may accept a later node. In Figure 2, n1 and n2 pass every validator, while n3 and n4 fail at least one component check. The walk reaches n4 but retains n2 as the deepest safe result.

After traversal, the core builds a MatchResult. Component finalizers then prepare the selected values for reuse, including the copy needed when a shared MAMBA checkpoint becomes private to one request.

Component Hooks Across the Tree Lifecycle

The same component contract covers the rest of the tree lifecycle:

This contract keeps the tree core generic while allowing components to preserve different correctness rules. Removing one component payload does not always remove the radix node. The remaining topology can still anchor other components, and the empty component slot can remain as a tombstone until that component is restored or the node becomes unnecessary. Because component semantics remain attached to one prefix identity, HiCache can extend component payloads across memory tiers without introducing another tree.

Native HiCache Across Memory Tiers

Components determine what can be reused. HiCache determines where the reusable payload resides. Unified Radix Cache carries the same component identity across GPU L1, Host L2, and an external L3 tier, so moving data between tiers does not change its prefix identity or reuse rule. Components describe the required transfers, and HybridCacheController executes the physical I/O.

Components, Anchors, and Sidecars

Not every physical pool needs its own component. An anchor determines reuse semantics or supplies the page indices that other pools follow. A sidecar stores a separate payload but reuses the indices of its declared source pool. It moves with that source without voting on the reusable boundary or adding another slot to the radix topology.

DeepSeek-V4 makes this distinction concrete. FULL covers the logical prefix, while SWA covers only its trailing window, so both are components. They also use independent device index spaces. In the normalized six-page example in Figure 3, the allocator maps the FULL tail slots F4, F5 to the SWA slots S0, S1 at runtime. The C4 and C128 compressed KV pools, indexer buffers, and compressor states do not define new reuse boundaries. They register as sidecars, with three pools following FULL and two following SWA.

HiCache Multi-Turn Benchmark Results

Multi-turn workloads grow a reusable conversational prefix on every round. If lower tiers preserve that prefix after GPU capacity is exhausted, cache hit rate should remain high and TTFT should grow more slowly.

We compare three cache configurations on two hybrid models: GPU L1 only, GPU L1 with Host L2, and GPU L1 with Host L2 plus a 500 GiB Mooncake Store distributed memory tier as L3. DeepSeek-V4-Flash uses FULL and SWA on four H200 GPUs with TP4, 48 clients, 60 rounds, and 4,096 input plus 16 output tokens per turn. Inkling-Small uses FULL, SWA, and MAMBA on eight H200 GPUs with TP8, 64 clients, 30 rounds, and 1,216 input plus 64 output tokens per turn.

The command outlines below contain placeholders for model paths and the Mooncake client configuration. They record the runtime and workload flags used here, but not a complete reproducible environment.

export MODEL=/path/to/DeepSeek-V4-Flash-FP8
export SGLANG_ENABLE_UNIFIED_RADIX_TREE=1
COMMON="--trust-remote-code --model-path $MODEL --tp 4 --mem-fraction-static 0.9 \
  --context-length 262144 --page-size 64 --max-running-requests 16 \
  --host 0.0.0.0 --enable-cache-report --enable-metrics \
  --enable-metrics-for-all-schedulers"
HICACHE="--enable-hierarchical-cache --hicache-ratio 2 --hicache-size 0 \
  --hicache-mem-layout page_first --hicache-io-backend kernel \
  --hicache-write-policy write_through \
  --hicache-storage-prefetch-policy wait_complete"

# L1-only
sglang serve $COMMON --port 30001

# L2 HiCache
sglang serve $COMMON $HICACHE --port 30000

# Start the 500 GiB external Mooncake Store tier first, then add:
sglang serve $COMMON $HICACHE --port 30000 \
  --hicache-storage-backend mooncake \
  --hicache-storage-backend-extra-config "$MOONCAKE_CLIENT_JSON"

# Benchmark: use port 30001 for L1, or port 30000 for L2/L3.
PORT=30000
python3 benchmark/hicache/bench_multiturn.py \
  --host 127.0.0.1 --port "$PORT" --model-path "$MODEL" \
  --num-clients 48 --num-rounds 60 --request-length 4096 --output-length 16 \
  --max-parallel 16 --request-rate 64 --disable-auto-run \
  --disable-random-sample --enable-round-barrier --ready-queue-policy fifo \
  --seed 20260626

export MODEL=/path/to/inkling
export SGLANG_ENABLE_UNIFIED_RADIX_TREE=1
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

COMMON="--trust-remote-code --model-path $MODEL --tp 8 \
  --mem-fraction-static 0.85 --context-length 262144 --page-size 64 \
  --max-total-tokens 750080 --max-running-requests 16 --host 0.0.0.0 \
  --enable-cache-report --enable-metrics --enable-metrics-for-all-schedulers \
  --mamba-radix-cache-strategy extra_buffer --swa-full-tokens-ratio 0.1 \
  --mamba-full-memory-ratio 0.1 --disable-prefill-cuda-graph"
HICACHE="--enable-hierarchical-cache --hicache-ratio 2 --hicache-size 0 \
  --hicache-mem-layout page_first --hicache-io-backend kernel \
  --hicache-write-policy write_through \
  --hicache-storage-prefetch-policy wait_complete"

# L1-only
sglang serve $COMMON --port 30001

# L2 HiCache
sglang serve $COMMON $HICACHE --port 30000

# Start the 500 GiB external Mooncake Store tier first, then add:
sglang serve $COMMON $HICACHE --port 30000 \
  --hicache-storage-backend mooncake \
  --hicache-storage-backend-extra-config "$MOONCAKE_CLIENT_JSON"

# Benchmark: use port 30001 for L1, or port 30000 for L2/L3.
PORT=30000
python3 benchmark/hicache/bench_multiturn.py \
  --host 127.0.0.1 --port "$PORT" --model-path "$MODEL" \
  --num-clients 64 --num-rounds 30 --request-length 1216 --output-length 64 \
  --max-parallel 16 --request-rate 64 --disable-auto-run \
  --disable-random-sample --enable-round-barrier --ready-queue-policy fifo \
  --seed 20260626 --log-file "inkling-${PORT}.jsonl" --tag inkling

Figure 4 reports average TTFT and prompt-token cache hit rate by round. For each round, the hit rate is the sum of cached prefix tokens across requests divided by the sum of their complete prompt lengths. In both workloads, L1 loses reusable prefixes first, L2 delays the capacity limit, and L3 remains high after warmup and finishes above 96%.

On DeepSeek-V4-Flash, L3 keeps the hit rate near 98%, holds average TTFT below 9 seconds, and reaches 145.5K effective input tokens/s, compared with 9.4K for L1 and 14.3K for L1 plus L2. On Inkling-Small, L3 finishes at a 96.8% hit rate and 1.23-second TTFT while reaching 67.1K effective input tokens/s, compared with 15.5K for L1 and 21.1K for L1 plus L2.

Effective input-token throughput follows bench_multiturn.py: the sum of complete prompt lengths divided by wall-clock duration. It credits cache-hit prefix tokens, so it measures serving progress under prefix reuse rather than raw prefill compute throughput. In these runs, the L3 gain comes primarily from keeping reusable prefixes available after the smaller tiers reach capacity.

Session-Aware Eviction on the Shared Tree

Session-aware eviction is implemented directly in UnifiedRadixCache. The mechanism supplies a reuse signal that ordinary LRU does not have. LRU records which cache entries were accessed recently, but not which prefixes belong to active sessions and are likely to be reused on their next turns. Under memory pressure, it can evict an active session's GPU KV while retaining unrelated entries.

Applications attach a stable session_id to every request. After a request finishes successfully, Unified Radix Cache registers the reusable region for that session. FULL tracks its prefix path, SWA tracks its trailing window, and MAMBA tracks its reusable frontier. All sessions still share one radix topology, and every turn still supplies its complete prompt.

These references change eviction order rather than pinning memory. FULL orders candidates by whether they are referenced, their session reference count, and the configured base eviction priority. SWA and MAMBA first scan unreferenced entries in their own reusable regions, then fall back to referenced entries when more space is required. The current policy covers GPU L1 and Host L2, not external L3 tiers.

When an application calls /close_session, Unified Radix Cache removes that session's references without immediately deleting its cache entries. Session generations and bounded closed-session tombstones prevent stale requests that finish after a close or reopen from restoring released references.

Session-Aware HiCache on SWE-bench Workloads

We evaluate DeepSeek-V4-Pro and Qwen3.5-397B-A17B with TP8 and HiCache on SWE-bench agent trajectories. The baseline uses ordinary HiRadixCache with LRU. The comparison enables Unified Radix Cache and --enable-session-radix-cache. Because this changes both the cache implementation and the eviction policy, the observed differences should not be interpreted as an isolated ablation of session awareness. The benchmark record provides the server flags and sandbox configuration.

The top row of Figure 6 stacks device and host cache hit ratios. At batch size 128, DeepSeek-V4-Pro's device hit ratio increases from about 42% to 51%. At batch size 32, Qwen3.5-397B-A17B increases from about 5% to 34%. At batch size 64, Qwen's total device plus host hit ratio increases from about 58% to 67%.

The bottom row reports the corresponding TTFT. Relative to the ordinary HiRadixCache baseline, the session-aware Unified Radix Cache configuration observes 11.0% and 2.9% lower TTFT for DeepSeek-V4-Pro at batch sizes 128 and 256. Qwen3.5-397B-A17B observes 13.5% and 16.6% lower TTFT at batch sizes 32 and 64.

Toward a Rust Tree Core

As a shared prefix grows, tree traversal, lock bookkeeping, LRU updates, and eviction scans add work to the scheduler's critical path. UnifiedRadixCache separates this tree state machine from cache orchestration, which makes the tree core a natural target for a native implementation.

The experimental Rust Unified Radix Cache is an opt-in, L1-only prototype. Rust owns the radix topology, per-component lock accounting, intrusive LRU lists, and eviction walks. Python remains the single owner of request-to-token mappings and physical KV allocation. After mutating the tree, Rust returns deferred actions for Python to apply to the pools. The prototype supports FULL, SWA, and MAMBA, but not HiCache.

We compare this prototype with the Python UnifiedRadixCache over a 200-turn synthetic conversation. Every turn adds 100 input tokens and generates 100 output tokens. Both backends use the same model and server flags, run sequentially on the same GPUs, and execute six trials. The workloads cover full attention with Qwen3-32B at TP2, SWA with gpt-oss-20b at TP2, and hybrid SSM with Qwen3-Next-80B-A3B at TP4. The reproduction scripts require a release build of the Rust extension.

The Rust prototype records the largest reduction on the SWA workload. TTFT is 38% lower across all 200 turns and 42% lower over turns 176 to 200. Full attention records 10% lower TTFT overall and 18% lower over the final 25 turns. The hybrid SSM workload records 5% lower TTFT overall and 7% lower over the final 25 turns.

The bottom row of Figure 7 subtracts the CUDA-event-timed GPU prefill interval from total TTFT. This residual includes tree bookkeeping, scheduling, synchronization, sampling, detokenization, transport, and other uninstrumented work. It is not a direct CPU timer, and the complete Rust and Python difference cannot be attributed only to radix tree operations. The hybrid SSM result illustrates this boundary. Its residual falls substantially, but the larger GPU forward limits the change visible in total TTFT.

These measurements apply only to the experimental prototype above. The follow-up Rust UnifiedTreeCoreInterface RFC #32710 defines the target ownership boundary. Orchestration and pool management remain in Python behind a replaceable tree core. The RFC currently supports FULL only and does not publish performance results.

Future Work

Unified Radix Cache establishes a shared prefix identity, but three parts of the system still need to converge around it:

• Complete the replaceable Rust tree core. The roadmap #20415 tracks this migration. The remaining work is to extend the Rust core to SWA, MAMBA, and HiCache while keeping pool allocation and orchestration in Python.
• Connect GPU L1 directly to external L3 tiers. Direct L3 mode would make Host L2 an optional staging tier and expose distributed memory as a larger shared cache. This requires coordinated admission, prefetch, transfer, and eviction rather than only another storage connector.
• Coordinate agentic KV caching across the serving stack. Agentic workloads already benefit from prefix caching within each engine. The roadmap #21846 extends the same cache identity across routers, prefill and decode workers, and HiCache so these layers can coordinate prefetch, demotion, and retention for sessions, subagents, and tool calls.

Conclusion

Hybrid models do not have one universal reusable boundary, but they do not require a separate radix tree for every combination of attention and recurrent state. Unified Radix Cache keeps one canonical token topology while FULL, SWA, and MAMBA components enforce their own reuse, locking, and eviction semantics.

That shared identity is useful beyond prefix matching. HiCache preserves it across memory tiers, session references turn it into a retention signal, and the experimental Rust core shows how tree bookkeeping can evolve without moving pool ownership out of Python. The longer-term direction is to make reusable cache state visible to the whole serving system, not only to a local allocator.

Acknowledgements

We thank the Alibaba Cloud TairKVCache team for co-leading the integration of Unified Radix Cache with HiCache across hybrid models and validating large-scale production deployments. We thank the Thinking Machines Lab team for validating Unified Radix Cache under heavy load and fixing correctness issues across component combinations. We also thank the Clank.world team for validating Gemma 4 SWA HiCache in a high-concurrency production deployment.

We also thank Mingjun Zhang for the session-aware eviction work and its validation. We thank Tingwei Huang from Ant Group SCT Inference team for helping integrate Hybrid HiCache with Mooncake. We are grateful to Lianmin Zheng, Ishan Dhanani, Zhiqiang Xie, Chao Shi, Yanbo Yang, Shangming Cai, Hongjia Zhang, and the SGLang community for architecture reviews, systems integration, benchmarking, and feedback. We also thank all contributors to the related roadmaps #20415 and #21846.