FreeLLMBuoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Kinference adds declarative routing, KV‑cache‑aware load balancing and per‑tenant metrics, cutting median TTFT by 95% in Buoyant’s benchmark.
Buoyant announced Kinference, an Apache‑2.0 licensed control plane designed for large‑language‑model inference on Kubernetes.
The project runs on top of inference engines such as vLLM and introduces token rate limiting, tenant usage tracking, intelligent load balancing and policy‑driven routing.
Kinference’s load balancer considers GPU queue sizes, historical performance and KV‑cache content, directing requests to the worker that will do the least work.
In Buoyant’s own benchmark, the new control plane reduced median time‑to‑first‑token by 95% compared with standard Kubernetes service load balancing.
The system can be paired with existing stacks via an HTTP endpoint that accepts OpenAI‑ or Anthropic‑compatible traffic, and it can operate alongside Linkerd though it does not require it.
Was das für dich bedeutet
For AI teams, Kinference offers a way to share limited GPU resources more fairly and efficiently, improving response times for LLM applications. Users of free or locally hosted models gain a scalable platform that can manage multiple tenants and models without building custom routing logic.
Quellen
- Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM, MarkTechPost, 7 Nov 2026
- Kinference: Scalable Inference Control Plane for Kubernetes, buoyant.io (official), 2 Nov 2026
Automatisch aus den obigen Quellen geschrieben, dann von unserem Code geprüft: Jede Zahl und jeder Name muss darin vorkommen, und kein Satz darf eine Quelle kopieren. Einen Fehler entdeckt? Melde ihn.
Welche KI kannst du heute kostenlos nutzen?
Jedes kostenlose KI‑Modell, bewertet durch blinde menschliche Stimmen, mit dem Speicher und der Geschwindigkeit, die es auf deiner eigenen GPU benötigt.