kvio
GPU-free KV-cache-offload storage-IO projector & replayer

01What it is

Storage and systems engineers need to evaluate the disk I/O of LLM KV-cache offload — command sizes, counts, volume, latency — but that I/O normally only exists behind a GPU running a model through vLLM + LMCache. kvio removes the GPU from that loop.

The key observation: storage I/O geometry is content-independent. How many NVMe commands a KV store/load produces, and how big each one is, depends only on the KV block size and the device's transfer limit — not on the actual tensor values. So real model dimensions + fake bytes reproduce the real offload I/O pattern. kvio computes the block size from real model geometry, issues that store/load workload through LMCache's real raw_block NVMe-passthrough engine, and confirms the result against the actual device commands captured by an eBPF tracer.

02How it works

kvio is a two-phase model layered on top of LMCache's real storage engine.

A recorded workload is captured as a compact kvio_record.json manifest — per-object payload sizes + device geometry, a few KB — that can be replayed later to reissue the identical command stream on any device.

03Mapping a KV object to its NVMe commands

A single KV store or load does not map to a single disk operation. LMCache hands its raw_block engine one KV object; the engine splits it into many NVMe commands (each capped at the device's maximum transfer size). Down at the device, the eBPF tracer sees only those raw commands — an offset, a length, a direction — with no idea which object produced them. This layer's whole job is to reconnect them: take one object and find the exact commands it caused, and go the other way too.

Our original implementation — a tag threaded through the stack

kvio gives every KV-object operation a unique integer we call its trace_id — literally "this store/load, of this object." The trick is getting that label all the way down to the device tracer. io_uring already hands each submission a 64-bit scratch field, user_data, which the kernel returns untouched on completion; programs normally use it as a completion counter to match a finished I/O back to the request that started it. That counter was already there. Our original change simply packed the trace_id into the unused high half of that same field:

user_data = (trace_id << 32) | counter

■ trace_id — the object's id, the part kvio added (high 32 bits).   ■ counter — io_uring's own completion counter, already there (low 32 bits, untouched).

Because the low half is unchanged, io_uring's completion matching still works exactly as before. The eBPF tracer reads user_data off each NVMe command and recovers the object as trace_id = user_data >> 32 — one logical object joined to its N wire commands. The catch: this needs a change to LMCache's engine. (Every io_uring request carries a user_data field — not just passthrough — but it never travels on the NVMe wire, and ordinary POSIX I/O has no request cookie at all, so this join only reaches as far as the passthrough path our monitor can tie together.)

The current implementation — no LMCache change needed

The tag works, but needing an engine change — and a user_data field that only passthrough has — is a real limitation. The current kvio removes it entirely: it attributes each device command to a KV object using only two things the trace and the semantic record already carry — the command's byte offset and its timestamp. No engine change — it works across any NVMe path the Linux driver tracepoint can see (POSIX, io_uring on a file, local cuFile/GDS).

Worked examples — one command, mapped to its object

Illustrative records, trimmed to the fields that do the join.

1 · io_uring_cmd passthrough — the object's tag rides in the command:

semantic  {"op":"store", "object_id":42, "trace_id":42, "bytes":262144}
command   {"op":"write", "slba":57344, "bytes":131072, "user_data":"0x0000002a_00000005"}
join      user_data >> 32  =  0x2a  =  42   →   object 42

2 · NVMe Key-Value command set — the 16-byte object key is inside the command, so no cookie is needed at all:

semantic  {"op":"store", "key_hex":"a7f3c1…9b", "bytes":262144}
command   {"op":"kv_store", "key_hex":"a7f3c1…9b", "bytes":262144}
join      key_hex matches   →   same object      (unmodified client, zero engine change)

3 · POSIX O_DIRECT on a file — offset + time, using FIEMAP to line the offsets up:

fiemap    file 0x0700000..0x0740000  →  device LBA 57344     (once, via FS_IOC_FIEMAP)
semantic  {"object_id":7, "slot_dev_off":29360128, "bytes":262144,
           "ts_start":1828074395351013, "ts":1828074395894120}
command   {"op":"write", "slba":57344, "bytes":131072, "ts":1828074395502771}
join      offset  57344 × 512 = 29360128  ∈  object 7's slot [29360128, 29622272)
          time    1828074395502771       ∈  [ts_start, ts]      →  the store
                                                                   →  object 7

The 256 KiB object splits into two 128 KiB writes; the second (slba 57600) lands in the same slot and joins to the same object. cuFile / GDS uses this identical offset+time path — no engine change — pending a GPU-direct-capable box.

Validation status

04Components

05Dependencies

06How to run

1 · Generate real-geometry offload I/O (GPU-free)

  LMCache examples/kv_cache_offload_io
# real model geometry -> real io_uring_cmd passthrough I/O for fake KV blocks
python run_kv_offload_io.py --model meta-llama/Llama-3.1-8B-Instruct \
    --dtype bfloat16 --chunk-tokens 256 --num-chunks 8 \
    --device /dev/ng0n1 --engine uring_cmd \
    --record /tmp/kvio_record.json --trace /tmp/sem.jsonl

2 · Capture the wire trace alongside it

  ebpf-syscall
sudo ./nvme_uring_cmd_monitor --dur 90 --lba-size 512 --jsonl /tmp/nvme.jsonl &
# ... run step 1 with LMCACHE_KVIO_TRACE=/tmp/sem.jsonl ...

3 · Validate projection vs. real device commands

  ebpf-syscall/examples/lmcache
python kvio_validate.py --tracer /tmp/nvme.jsonl --semantic /tmp/sem.jsonl \
    --lba-bytes 4096 --mdts-bytes 131072
# exact-match 8/8, WAPE 0.0000%, per-cmd size 8/8, size-dist TV 0.0000

4 · Replay a recorded workload elsewhere

  ebpf-syscall/examples/lmcache
python kvio_replay.py --record /tmp/kvio_record.json --device /dev/ng0n1 --iters 5
# reissues the exact store-all-then-load-all command stream

07Fidelity metrics

Three complementary scores, joined per object by trace_id. AUC is deliberately not used — there is no class label; the geometry is deterministic.

08Validated results

On an 8× H100 server with real Samsung Gen5 NVMe (io_uring_cmd passthrough on /dev/ng), the projection was validated against a real GPU-driven vLLM + LMCache offload — the previously hardware-gated step is now closed.

Scale & parity campaign — 7 models, 1B → 70B, TP 1/2/4

Each cell: real GPU capture → wire-validate → replay the recorded manifest → regenerate GPU-free from the calculator. Every real leg was exact.