Llama 3.3 70B is Meta's best dense open-source model, matching GPT-4o on many benchmarks. This guide covers hardware requirements, quantisation selection, and optimal settings for running it on a high-end workstation or Apple Silicon Mac.
At Q4_K_M quantisation, the Llama 3.3 70B model weighs around 42 GB. You need either: an Apple Silicon Mac with 48 GB+ of unified memory (M2 Max/Ultra, M3 Max/Ultra, M4 Max/Ultra), or a workstation with an NVIDIA GPU with 48 GB+ VRAM (RTX 6000 Ada, A100 80GB, dual RTX 3090/4090), or a system with 64 GB+ RAM if you are willing to run on CPU (much slower).
CPU inference is feasible for non-interactive tasks. At Q4 on a modern desktop CPU with 64 GB RAM, you can expect 2–4 tokens per second — acceptable for batch processing but too slow for chat. Apple Silicon with 48 GB unified memory runs at 12–18 tok/s at Q4, which is comfortable for real-time use.
With 48 GB unified memory, run: `ollama pull llama3.3:70b`. The download is around 43 GB. Ollama automatically uses Metal acceleration on Apple Silicon and CUDA on NVIDIA GPUs.
Start a session: `ollama run llama3.3:70b`. On M2 Max (32 GPU cores, 48 GB), you will see around 14 tok/s for the 70B model — fast enough for comfortable interactive use. On M3 Max, expect 18–22 tok/s.
Q4_K_M is the recommended starting point — it reduces model size by ~75% versus BF16 with only a ~1–2% quality drop on most benchmarks. Q5_K_M improves quality slightly (0.5–1%) at 20% larger size — worth it if you have 52+ GB VRAM. Q8_0 is essentially lossless quality at about half the BF16 size but requires 70+ GB.
For the 70B model: Q4_K_M is ~42 GB (fits 48 GB), Q5_K_M is ~50 GB (fits 52 GB), Q8_0 is ~74 GB (requires 80 GB). If your system is right on the edge, go one notch down — running comfortably at Q4 beats thrashing memory at Q5.
In Ollama, set the number of GPU layers to max: add `OLLAMA_NUM_GPU=99` to your environment (or set it in the Modelfile). On Apple Silicon this is handled automatically — all layers go to the GPU.
Reduce context length for shorter conversations. Llama 3.3 supports 128K context, but loading a 128K KV cache uses significant memory. Set context to 8K for most tasks: `ollama run llama3.3:70b --ctx-size 8192`. This alone can increase throughput by 20–30% on memory-constrained systems.
Batch multiple prompts together when processing documents. Llama 3.3's prompt processing (prefill) is very fast — it is the generation (decoding) step that is slow. Pre-processing 10 documents into a queue and letting the model decode continuously is more efficient than sending one prompt at a time.
Ollama is the fastest way to run open-source LLMs locally. This guide walks you through installation on macOS, Linux, and Windows, pulling your first model, and running it from the terminal or via its REST API.
Read guideLM Studio is a desktop app for running local AI models without any terminal commands. This guide covers installation, downloading a model from Hugging Face, and starting a local inference server.
Read guide