FreeLLMBuoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Kinference adds declarative routing, KV‑cache‑aware load balancing and per‑tenant metrics, cutting median TTFT by 95% in Buoyant’s benchmark.
Buoyant announced Kinference, an Apache‑2.0 licensed control plane designed for large‑language‑model inference on Kubernetes.
The project runs on top of inference engines such as vLLM and introduces token rate limiting, tenant usage tracking, intelligent load balancing and policy‑driven routing.
Kinference’s load balancer considers GPU queue sizes, historical performance and KV‑cache content, directing requests to the worker that will do the least work.
In Buoyant’s own benchmark, the new control plane reduced median time‑to‑first‑token by 95% compared with standard Kubernetes service load balancing.
The system can be paired with existing stacks via an HTTP endpoint that accepts OpenAI‑ or Anthropic‑compatible traffic, and it can operate alongside Linkerd though it does not require it.
Qué significa para ti
For AI teams, Kinference offers a way to share limited GPU resources more fairly and efficiently, improving response times for LLM applications. Users of free or locally hosted models gain a scalable platform that can manage multiple tenants and models without building custom routing logic.
Fuentes
- Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM, MarkTechPost, 7 Nov 2026
- Kinference: Scalable Inference Control Plane for Kubernetes, buoyant.io (official), 2 Nov 2026
Escrito automáticamente a partir de las fuentes anteriores, luego revisado por nuestro código: cada número y nombre debe aparecer en ellas, y ninguna oración puede copiar una fuente. ¿Detectaste un error? Infórmalo.
¿Qué IA puedes usar gratis hoy?
Todos los modelos de IA gratuitos, clasificados por votos humanos ciegos, con la memoria y velocidad que necesita tu GPU.