FreeLLMNVIDIA serves 753B GLM‑5.3 model across two DGX stations using vLLM
NVIDIA’s latest blog details how to run the 753‑billion‑parameter GLM‑5.3 NVFP4 model on two DGX stations linked by ConnectX‑8 RDMA.
NVIDIA’s latest blog details how to run the 753‑billion‑parameter GLM‑5.3 NVFP4 model on two DGX stations linked by ConnectX‑8 RDMA.
The model needs about 464 GB of GPU memory, far beyond the 252 GB HBM3e on a single GB300 GPU, so tensor parallelism (TP=2) splits the weights across both GPUs.
Using vLLM, the two stations present a single OpenAI‑compatible endpoint, with each GPU holding and computing roughly half the model.
AIPerf benchmarks on the SPEED‑Bench Coding dataset show peak throughput of 607.8 tokens per second at 16 concurrent requests, dropping to 385.0 tokens per second at 32 concurrency with a mean latency of 453.5 seconds.
The guide walks users through prerequisites, network setup, container deployment, and performance testing, linking to the Connect Two DGX Stations playbook for hardware validation.
O que isso significa para você
Running GLM‑5.3 locally requires multi‑GPU setups; this tutorial gives practitioners a concrete path to deploy frontier‑scale models without cloud services. For users of free or open‑source models, it demonstrates that large‑scale inference is possible on-premises, though hardware costs remain high.
Fontes
- When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM, NVIDIA Developer (official), 13 Nov 2026
Escrito automaticamente a partir das fontes acima, depois verificado pelo nosso código: todo número e nome deve aparecer nelas, e nenhuma frase pode copiar uma fonte. Encontrou um erro? Reporte‑o.
Qual IA você pode usar grátis hoje?
Todo modelo de IA gratuito, classificado por votos humanos cegos, com a memória e velocidade que ele precisa na sua própria GPU.