kvio
GPU-free KV-cache-offload storage-IO toolkit

01What it is

Storage and systems engineers need to evaluate the disk I/O of LLM KV-cache offload — command sizes, counts, volume, latency — but that I/O normally only exists behind a GPU running a model through vLLM + LMCache. kvio removes the GPU from that loop.

The key observation: storage I/O geometry is content-independent. How many NVMe commands a KV store/load produces, and how big each one is, depends only on the KV block size and the device's transfer limit — not on the actual tensor values. So real model dimensions + fake bytes reproduce the real offload I/O pattern. kvio computes the block size from real model geometry, issues that store/load workload through LMCache's real raw_block NVMe-passthrough engine, and confirms the result against the actual device commands captured by an eBPF tracer.

02How it works

kvio is a two-phase model layered on top of LMCache's real storage engine.

A recorded workload is captured as a compact kvio_record.json manifest — per-object payload sizes + device geometry, a few KB — that can be replayed later to reissue the identical command stream on any device.

03Mapping a KV object to its NVMe commands

A single KV store or load does not map to a single disk operation. LMCache hands its raw_block engine one KV object; the engine splits it into many NVMe commands (each capped at the device's maximum transfer size). Down at the device, the eBPF tracer sees only those raw commands — an offset, a length, a direction — with no idea which object produced them. This layer's whole job is to reconnect them: take one object and find the exact commands it caused, and go the other way too.

Our original implementation — a tag threaded through the stack

kvio gives every KV-object operation a unique integer we call its trace_id — literally "this store/load, of this object." The trick is getting that label all the way down to the device tracer. io_uring already hands each submission a 64-bit scratch field, user_data, which the kernel returns untouched on completion; programs normally use it as a completion counter to match a finished I/O back to the request that started it. That counter was already there. Our original change simply packed the trace_id into the unused high half of that same field:

user_data = (trace_id << 32) | counter

■ trace_id — the object's id, the part kvio added (high 32 bits).   ■ counter — io_uring's own completion counter, already there (low 32 bits, untouched).

Because the low half is unchanged, io_uring's completion matching still works exactly as before. The eBPF tracer reads user_data off each NVMe command and recovers the object as trace_id = user_data >> 32 — one logical object joined to its N wire commands. The catch: this needs a change to LMCache's engine. (Every io_uring request carries a user_data field — not just passthrough — but it never travels on the NVMe wire, and ordinary POSIX I/O has no request cookie at all, so this join only reaches as far as the passthrough path our monitor can tie together.)

The current implementation — no LMCache change needed

The tag works, but needing an engine change — and a user_data field that only passthrough has — is a real limitation. The current kvio removes it entirely: it attributes each device command to a KV object using only two things the trace and the semantic record already carry — the command's byte offset and its timestamp. No engine change — it works across any NVMe path the Linux driver tracepoint can see (POSIX, io_uring on a file, local cuFile/GDS).

Worked examples — one command, mapped to its object

Illustrative records, trimmed to the fields that do the join.

1 · io_uring_cmd passthrough — the object's tag rides in the command:

semantic  {"op":"store", "object_id":42, "trace_id":42, "bytes":262144}
command   {"op":"write", "slba":57344, "bytes":131072, "user_data":"0x0000002a_00000005"}
join      user_data >> 32  =  0x2a  =  42   →   object 42

2 · NVMe Key-Value command set — the 16-byte object key is inside the command, so no cookie is needed at all:

semantic  {"op":"store", "key_hex":"a7f3c1…9b", "bytes":262144}
command   {"op":"kv_store", "key_hex":"a7f3c1…9b", "bytes":262144}
join      key_hex matches   →   same object      (unmodified client, zero engine change)

3 · POSIX O_DIRECT on a file — offset + time, using FIEMAP to line the offsets up:

fiemap    file 0x0700000..0x0740000  →  device LBA 57344     (once, via FS_IOC_FIEMAP)
semantic  {"object_id":7, "slot_dev_off":29360128, "bytes":262144,
           "ts_start":1828074395351013, "ts":1828074395894120}
command   {"op":"write", "slba":57344, "bytes":131072, "ts":1828074395502771}
join      offset  57344 × 512 = 29360128  ∈  object 7's slot [29360128, 29622272)
          time    1828074395502771       ∈  [ts_start, ts]      →  the store
                                                                   →  object 7

The 256 KiB object splits into two 128 KiB writes; the second (slba 57600) lands in the same slot and joins to the same object. cuFile / GDS uses this identical offset+time path — no engine change — pending a GPU-direct-capable box.

Validation status

04Components

05Dependencies

06How to run

1 · Generate real-geometry offload I/O (GPU-free)

  LMCache examples/kv_cache_offload_io
# real model geometry -> real io_uring_cmd passthrough I/O for fake KV blocks
python run_kv_offload_io.py --model meta-llama/Llama-3.1-8B-Instruct \
    --dtype bfloat16 --chunk-tokens 256 --num-chunks 8 \
    --device /dev/ng0n1 --engine uring_cmd \
    --record /tmp/kvio_record.json --trace /tmp/sem.jsonl

2 · Capture the wire trace alongside it

  ebpf-syscall
sudo ./nvme_uring_cmd_monitor --dur 90 --lba-size 512 --jsonl /tmp/nvme.jsonl &
# ... run step 1 with LMCACHE_KVIO_TRACE=/tmp/sem.jsonl ...

3 · Validate projection vs. real device commands

  ebpf-syscall/examples/lmcache
python kvio_validate.py --tracer /tmp/nvme.jsonl --semantic /tmp/sem.jsonl \
    --lba-bytes 4096 --mdts-bytes 131072
# exact-match 8/8, WAPE 0.0000%, per-cmd size 8/8, size-dist TV 0.0000

4 · Replay a recorded workload elsewhere

  ebpf-syscall/examples/lmcache
python kvio_replay.py --record /tmp/kvio_record.json --device /dev/ng0n1 --iters 5
# reissues the exact store-all-then-load-all command stream

07kvio bench: sustained storage pressure

kvio bench asks how much KV-like I/O a storage tier can sustain, what its tail latency is under load, and how large transfers affect another workload on the same drive. It runs fio without a GPU or model server and saves the resolved workload, fio output, result rows, hardware details, and kernel settings for each run.

Run and compare

  ebpf-syscall
make kvio-test
./kvio bench --list-profiles

# WARNING: preconditions and overwrites the selected raw namespace
sudo ./kvio bench /dev/nvmeXnY \
  --yes-really-use-device --size 8GiB \
  --runtime 20 --ramp-time 3 --reps 3 \
  --output-dir results/baseline

./kvio bench-compare results/baseline/results.jsonl \
                     results/candidate/results.jsonl

The comparison reports median bandwidth, IOPS, whole-I/O p50 and p99 latency, fio system CPU, and target-controller interrupts per GiB. Keep the device, region, runtime, and repetition count identical across an A/B test.

Know which profiles are measured

Only restore-calibrated is derived from a recorded KV-cache setup. David traced LMCache 0.5.3 with vLLM 0.27.1, Qwen2.5-1.5B-Instruct, TP1, bf16, and 256-token chunks. One whole object is:

2 (K and V) × 28 layers × 2 KV heads × 128 values per head
× 2 bytes (bf16) × 256 tokens = 7,340,032 bytes = 7 MiB

The run used four synchronous disk workers and reported 108 stores and 200 restores. This source profile is one calibration point, not a universal object size. Use --calibrated-block-size for another model, tensor-parallel rank, dtype, or chunk size.

Add a sustained profile from a capture

--profile FILE adds a workload without changing Python. Each JSON profile must label its evidence measured or synthetic, identify the source, and define each fio job's operation, block size, queue depth, and worker count. The resolved profile is copied into run.json.

{
  "schema_version": 1,
  "name": "service-a-restore",
  "description": "Sustained restore shape from service A capture 17",
  "evidence": {
    "kind": "measured",
    "source": "/srv/kvio/captures/service-a-17.jsonl"
  },
  "jobs": [{
    "name": "restore", "rw": "randread", "bs": "32MiB",
    "iodepth": 1, "numjobs": 4
  }]
}

Lineage

Davidlohr Bueso wrote the standalone kvspill prototype. Its workload shapes, preconditioning, fio result parsing, and median A/B comparison became the starting point for kvio bench. They were merged with his permission and credit under this repository's Apache-2.0 license, then integrated with kvio's CLI, evidence labels, external profile format, result artifacts, and raw-device safety checks. There is no separate kvspill command in this tree; the old hostname records that history.

08Fidelity metrics

Three complementary scores, joined per object by trace_id. AUC is deliberately not used — there is no class label; the geometry is deterministic.

09Validated results

On an 8× H100 server with real Samsung Gen5 NVMe (io_uring_cmd passthrough on /dev/ng), the projection was validated against a real GPU-driven vLLM + LMCache offload — the previously hardware-gated step is now closed.

Scale & parity campaign — 7 models, 1B → 70B, TP 1/2/4

Each cell: real GPU capture → wire-validate → replay the recorded manifest → regenerate GPU-free from the calculator. Every real leg was exact.