FreeLLMBuoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Kinference adds declarative routing, KV‑cache‑aware load balancing and per‑tenant metrics, cutting median TTFT by 95% in Buoyant’s benchmark.
Buoyant announced Kinference, an Apache‑2.0 licensed control plane designed for large‑language‑model inference on Kubernetes.
The project runs on top of inference engines such as vLLM and introduces token rate limiting, tenant usage tracking, intelligent load balancing and policy‑driven routing.
Kinference’s load balancer considers GPU queue sizes, historical performance and KV‑cache content, directing requests to the worker that will do the least work.
In Buoyant’s own benchmark, the new control plane reduced median time‑to‑first‑token by 95% compared with standard Kubernetes service load balancing.
The system can be paired with existing stacks via an HTTP endpoint that accepts OpenAI‑ or Anthropic‑compatible traffic, and it can operate alongside Linkerd though it does not require it.
Cosa significa per te
For AI teams, Kinference offers a way to share limited GPU resources more fairly and efficiently, improving response times for LLM applications. Users of free or locally hosted models gain a scalable platform that can manage multiple tenants and models without building custom routing logic.
Fonti
- Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM, MarkTechPost, 7 Nov 2026
- Kinference: Scalable Inference Control Plane for Kubernetes, buoyant.io (official), 2 Nov 2026
Scritto automaticamente dalle fonti sopra, poi controllato dal nostro codice: ogni numero e nome deve comparire, e nessuna frase può copiare una fonte. Hai trovato un errore? Segnalalo.
Quale AI puoi usare gratis oggi?
Tutti i modelli AI gratuiti, classificati da voti umani anonimi, con la memoria e la velocità necessarie sulla tua GPU.