Ranvier
An open-source router for self-hosted LLM fleets. It sends each request to the GPU that already holds its prompt prefix in KV cache, rather than to whichever backend is free, so the prefill is not computed twice. Routing is an adaptive radix tree over token prefixes, with O(L) lookups under 50µs independent of how many prefixes are cached, in a shared-nothing, thread-per-core design on Seastar with no locks on the hot path. Circuit-breaking health checks, clustering over encrypted gossip, and Kubernetes discovery with a Helm chart.
- 3–5×KV-cache hit rate
- 27–29%lower median TTFT
- 8× A100measured on