FreeLLMBuoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Kinference adds declarative routing, KV‑cache‑aware load balancing and per‑tenant metrics, cutting median TTFT by 95% in Buoyant’s benchmark.
Buoyant announced Kinference, an Apache‑2.0 licensed control plane designed for large‑language‑model inference on Kubernetes.
The project runs on top of inference engines such as vLLM and introduces token rate limiting, tenant usage tracking, intelligent load balancing and policy‑driven routing.
Kinference’s load balancer considers GPU queue sizes, historical performance and KV‑cache content, directing requests to the worker that will do the least work.
In Buoyant’s own benchmark, the new control plane reduced median time‑to‑first‑token by 95% compared with standard Kubernetes service load balancing.
The system can be paired with existing stacks via an HTTP endpoint that accepts OpenAI‑ or Anthropic‑compatible traffic, and it can operate alongside Linkerd though it does not require it.
Ce que cela signifie pour vous
For AI teams, Kinference offers a way to share limited GPU resources more fairly and efficiently, improving response times for LLM applications. Users of free or locally hosted models gain a scalable platform that can manage multiple tenants and models without building custom routing logic.
Sources
- Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM, MarkTechPost, 7 Nov 2026
- Kinference: Scalable Inference Control Plane for Kubernetes, buoyant.io (official), 2 Nov 2026
Rédigé automatiquement à partir des sources ci‑dessus, puis vérifié par notre code : chaque nombre et chaque nom doivent y figurer, et aucune phrase ne doit copier une source. Une erreur ? Signalez‑la.
Quelle IA pouvez‑vous utiliser gratuitement aujourd’hui ?
Tous les modèles IA gratuits, classés par votes humains anonymes, avec la mémoire et la vitesse dont ils ont besoin sur votre propre GPU.