Two RTX 5090 hosts sharing a KV cache over Ethernet

KV-cache sharing lets a language-model session move to another GPU host without making the new host process the entire conversation again. The tested system saves a replica of a session’s cache after a sufficiently long turn, sends only the changed portion to the host the session would move to next, and imports it when needed. The experiment compared two RTX 5090 hosts connected over switched 1 GbE, 10 GbE through Tailscale, and direct 10 GbE. Coding agents built a todo app during 90-minute runs with 12 or 16 sessions; the clearest comparison came from the busier 16-session runs.

For a typical 247 MiB cache update, sending took a median 3.24 seconds over 1 GbE, 0.81 seconds over 10 GbE with Tailscale, and 0.55 seconds over direct 10 GbE. Faster networking improved the share of host switches that reused at least 90% of the prompt, from 93.8% at 1 GbE to 96.8% with Tailscale; direct 10 GbE reached 96.2%, a difference too small to distinguish from run-to-run variation. Yet importing a replica still took about 2.8 seconds at every link speed, and some switches remained cold because software incorrectly assumed the engine still held an earlier import. That bug was fixed in viiwork v2.11.1. The results suggest 1 GbE was workable for this setup, while engine import speed and reliable cache tracking mattered more than additional bandwidth. The comparison used single runs, not repeated trials.