How to use llama.cpp locally

This tool runs language models locally and in the cloud. It supports a wide range of hardware with minimal setup.

This installation example uses Homebrew on Mac or Linux. Choose another installation method from the documentation if you use a different system.

1. Step 1. Set up the official tool

   Use the installation instructions for your system.

   brew install llama.cpp

   The documented tool is available for the usage steps.

2. Step 2. Download and run a model

   Download and run a model directly from Hugging Face. The example combines downloading the selected model with running it on your computer.

   llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

   Download and run a model directly from Hugging Face.

3. Step 3. Launch the application server

   Launch the compatible API server. The quick start provides this server operation as another way to run the selected model locally.

   llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

   Launch the compatible API server.

4. Step 4. Send a completion request

   Send the documented completion request. The example asks the running model to continue a prompt through the local application server.

   curl --request POST \
       --url http://localhost:9931/completion \
       --header "Content-Type: application/json" \
       --data '{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128}'

   Send the documented completion request.

Check the health endpoint shown here. A successful response means that the model has loaded and the server is ready.

You can visit the web front end at the same url.

Read the documented requirements before running the commands.

Commands were checked against official documentation. This guide does not include a repository security certification.

• Official documentation

• Official documentation

• Official documentation