Running LLMs locally

Evergreen note

Mostly settled. I update it when the tooling changes.

Running models locally gives you privacy, zero per-token cost, and a fast feedback loop. The trade-off is that you are limited by your hardware.

Picking a model size

A rough rule: you need about 0.6 GB of memory per billion parameters at 4-bit quantization.

$$\text{memory (GB)}\approx \text{params (B)}\times \frac{\text{bits}}{8}\times 1.2$$

Model size	4-bit memory	Good for
3B	~2 GB	Autocomplete, classification
8B	~5 GB	Chat, summarisation
32B	~20 GB	Reasoning, coding

Getting started

# Install and run a model with Ollama
brew install ollama
ollama run llama3.1:8b

Then call it from Python:

import ollama
 
reply = ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Explain quantization in one line."}],
)
print(reply["message"]["content"])

How the pieces fit

flowchart LR
  A[Your app] --> B[Ollama / llama.cpp]
  B --> C[(Quantized model weights)]
  B --> D[GPU / Apple Silicon]

Tip

Start with the smallest model that does the job. You can always go bigger.

Related

• Agentic systems use local models as the “brain”
• Evaluating RAG pipelines covers grounding local models in your own data