FreeLLMNVIDIA serves 753B GLM‑5.3 model across two DGX stations using vLLM
NVIDIA’s latest blog details how to run the 753‑billion‑parameter GLM‑5.3 NVFP4 model on two DGX stations linked by ConnectX‑8 RDMA.
NVIDIA’s latest blog details how to run the 753‑billion‑parameter GLM‑5.3 NVFP4 model on two DGX stations linked by ConnectX‑8 RDMA.
The model needs about 464 GB of GPU memory, far beyond the 252 GB HBM3e on a single GB300 GPU, so tensor parallelism (TP=2) splits the weights across both GPUs.
Using vLLM, the two stations present a single OpenAI‑compatible endpoint, with each GPU holding and computing roughly half the model.
AIPerf benchmarks on the SPEED‑Bench Coding dataset show peak throughput of 607.8 tokens per second at 16 concurrent requests, dropping to 385.0 tokens per second at 32 concurrency with a mean latency of 453.5 seconds.
The guide walks users through prerequisites, network setup, container deployment, and performance testing, linking to the Connect Two DGX Stations playbook for hardware validation.
What it means for you
Running GLM‑5.3 locally requires multi‑GPU setups; this tutorial gives practitioners a concrete path to deploy frontier‑scale models without cloud services. For users of free or open‑source models, it demonstrates that large‑scale inference is possible on-premises, though hardware costs remain high.
Sources
- When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM, NVIDIA Developer (official), 13 Nov 2026
Written automatically from the sources above, then checked by our code: every number and name must appear in them, and no sentence may copy a source. Spotted a mistake? Report it.
Which AI can you use for free today?
Every free AI model, ranked by blind human votes, with the memory and speed it needs on your own GPU.