Ollama × Mooncake — Prefill/Decode KV Cache Disaggregation

A single-machine, no-RDMA demo of the Ollama connector from PR #2800: a prefill worker publishes a prompt's KV pages into a real Mooncake Store; a second request that shares the prompt's prefix reloads those pages over TCP and skips prefill for the shared tokens. All figures below are measured on this run.

transport: TCP single machine no RDMA hardware real Mooncake Store real ollama/prefix keying round-trip verified
1,792
tokens of prefill skipped
of 2,048 in the request
87.5%
KV page cache-hit rate
7/8 pages served from store
23 ms
prefix reload (p50)
98 MB of KV over TCP
4.2 GB/s
KV reuse throughput
vs 332 MB/s publish write
verified
byte-exact round-trip
loaded KV == published KV

Data flow

Prefill worker computes KV for the prompt StoreComputedPrefix() Decode worker reuses cached prefix, skips prefill LoadCachedPrefix() Mooncake Store · KV blocks (TCP transport, no RDMA) page 0: shared prefix → cache HIT 0 page 1: shared prefix → cache HIT 1 page 2: shared prefix → cache HIT 2 page 3: shared prefix → cache HIT 3 page 4: shared prefix → cache HIT 4 page 5: shared prefix → cache HIT 5 page 6: shared prefix → cache HIT 6 page 7: prompt-specific → recomputed 7 ■ 7 shared pages reused (87.5%) ■ 1 prompt-specific publish reuse decode reloads 1792 shared tokens (98 MB) in ~22 ms — prefill skipped for them

What the numbers say

Prefill work avoided
reused from store recomputed
with Mooncake reuse 7 reused pages 7 reused 1 pages 1 without reuse (baseline) 8 recomputed pages 8 recomputed
Second request: 7 of 8 KV pages come from the store; only the divergent page is recomputed. Without the connector every page is recomputed.
KV transfer throughput
reuse read publish write
reuse read (p50) reuse read (p50): 4293 MB/s 4293 MB/s publish write publish write: 332 MB/s 332 MB/s
Reads (the hot decode path) run far faster than the one-time publish write, which includes segment allocation and first-touch.
Prefix reload latency · 2,048-token prompt, 20 iterations
min min: 21.2 ms 21.2 ms p50 p50: 22.8 ms 22.8 ms mean mean: 23.3 ms 23.3 ms max max: 28.0 ms 28.0 ms

How it works

  1. Page keying (real connector code). Token ids are split into fixed pages; each page key is a SHA-256 hash chain seeded with the model name (ollama/prefix). A shared token prefix yields identical keys, so pages are reusable across requests; any earlier divergence changes every later key.
  2. Prefill → Store. StoreComputedPrefix checks batch_is_exist and publishes only missing pages with batch_put_from (put-if-absent, so a peer's pages are never clobbered).
  3. Store → Decode. LoadCachedPrefix takes the longest contiguous cached run and pulls it with batch_get_into, leaving only the divergent tail for the runtime to compute.
  4. Verified. Loaded bytes are compared against what prefill published — on both a 2,048-token realistic layout and the tiny layout from the PR's own example (3/4 pages shared).

Honest boundaries

KV layout per page: 28 layers × 4 KV heads × 128 head-dim, 256 tokens/page, fp16 → 14 MB/page. Publish 112 MB in 338 ms. Store: Mooncake Store (mooncake-transfer-engine wheel, real master+client). Reproduce with bash demo/run_demo.sh. Generated 2026-07-09T17:59:10+0800 on LAPTOP-JHMCRIFR from demo/bench-results.json.