Show HN: Llmbridge, a C++ LLM gateway with sub-millisecond overhead

llmbridge is a C++ OpenAI-compatible LLM gateway designed for minimal translation overhead and high concurrency. It provides microsecond-scale translation, dependency-free default builds (TLS via OpenSSL optional), OpenAI↔Anthropic translation (streaming and non-streaming), OpenAI↔Google Gemini and OpenAI↔Cohere non-streaming, passthrough for OpenAI-compatible providers, tool-calling support (including streaming tool results), prompt caching with forwarded breakpoints, per-request timing headers, credential passthrough to upstreams, and byte-forwards for Anthropic clients. The project is open-core: the repo exposes the fast gateway core (single-upstream translation and proxy), while multi-provider routing, observability, SSO, pricing/latency book and managed cloud are commercial add-ons from Kottos AI.

Benchmarks show extremely low gateway overhead: single-thread non-streaming added latency p99 around tens of microseconds (p99 = 3.0 reported elsewhere in the repo), and a single-worker throughput ceiling near 84k requests/sec (best run 87k). Direct comparisons vs LiteLLM show llmbridge adding ~80 µs p99 where LiteLLM added ~87 ms; translation to Anthropic is similarly cheap. Profiling attributes ~89% of gateway CPU time to kernel networking (TCP stack), so userland speedups yield small gains and scaling uses multiple workers. Streaming tests deliver near no-gateway token latencies (~30.8 ms to first token) with only 55-131 µs added per token and sustain huge concurrent stream counts (16k-24k streams on a single worker with >99.9% delivery), with reproduction instructions and raw data included.