Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Buoyant launched Kinference, an open‑source inference control plane for vLLM that runs on Kubernetes. It provides declarative routing, KV‑cache‑aware load balancing, and per‑tenant metrics, reducing median time‑to‑first‑token by 95% in internal tests.
Key signals
- 01Open‑source control plane for vLLM on Kubernetes
- 02Declarative routing policies and KV‑cache‑aware load balancing
- 03Per‑tenant metrics and 95% median TTFT reduction
Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM, published by MarkTechPost.
Buoyant has released Kinference, an Apache 2.0 inference control plane for vLLM on Kubernetes. It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics, and cut median TTFT 95% in Buoyant’s own benchmark. The post Buoyant Releases Kinference:…
MarkTechPost
Read the full story at the source: marktechpost.com.
Curated automatically from MarkTechPost. Infrenic News links to the original publisher and does not republish its content.
Daily briefing
Get the next briefing before everyone else.
Five AI stories, five minutes, every morning.
