llama.cpp includes a built-in HTTP server that exposes an OpenAI-compatible API. This guide covers compiling llama.cpp with GPU support, starting the server, and deploying it for team use.
Ollama is the easiest local model runner but has limited configurability. llama.cpp's `llama-server` exposes more fine-grained control: parallel request processing with continuous batching, speculative decoding with a draft model, detailed per-request metrics, and GGUF model loading with full quantisation control.
For team deployments where multiple users will be hitting the same server simultaneously, llama.cpp's continuous batching feature is critical — it batches incoming requests and processes them together, significantly improving GPU utilisation.
Clone the repository: `git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp`.
For NVIDIA (CUDA): `cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j`. Requires CUDA toolkit 12.x installed.
For Apple Silicon (Metal): `cmake -B build -DGGML_METAL=ON && cmake --build build --config Release -j`. Metal is the default on macOS; this explicitly enables it.
For CPU-only: `cmake -B build && cmake --build build --config Release -j`. Much slower but works on any hardware.
Download a GGUF model (from Hugging Face) and start the server: `./build/bin/llama-server -m models/phi-4-q4_k_m.gguf --host 0.0.0.0 --port 8080 -ngl 99 -c 8192 --parallel 4`.
`-ngl 99` offloads all layers to GPU. `--parallel 4` allows 4 simultaneous requests with continuous batching. `-c 8192` sets the context per slot. Adjust `--parallel` based on your VRAM — each parallel slot requires the full KV cache allocation.
The server exposes `/v1/chat/completions`, `/v1/completions`, and `/v1/models` — compatible with any OpenAI SDK client. Point your client at `http://your-server:8080/v1` with any API key.
For team access on a local network, bind to `--host 0.0.0.0` and share the server's IP with your team. Add nginx as a reverse proxy for HTTPS if accessing over the public internet. Add basic auth to the nginx config to prevent unauthorised access.
Speculative decoding uses a small draft model to propose multiple tokens at once, which the larger model then verifies in parallel. This can increase throughput by 2–3× for long generations. Use a model from the same family as your main model: a 3B draft model for a 70B main model.
Start the server with speculative decoding: `./build/bin/llama-server -m models/llama-70b-q4.gguf --draft models/llama-3b-q4.gguf --n-draft 8 ...`. The `--n-draft 8` flag specifies how many draft tokens to generate and verify per step.
For healthcare, legal, financial, and personal data, running AI completely offline ensures your data never leaves your machine. This guide covers the full offline setup stack and practical workflows.
Read guidePrompt injection attacks trick LLMs into ignoring their system prompt or executing unintended instructions. This guide covers the attack patterns, defence strategies, and code-level mitigations for production LLM applications.
Read guide