Ollama vs vLLM vs llama.cpp: Choosing the Right Local LLM Runtime

Running an open-weight model locally is only half the decision. The runtime you pick determines your throughput, your hardware ceiling, your operational model, and how much engineering you absorb. Ollama, vLLM, and llama.cpp solve the same core problem from three different directions, and they rarely compete directly: each wins in a different context. This guide breaks down their architectures, what the benchmarks actually show, and gives you a decision framework instead of a "it depends" answer.

The Three Approaches

llama.cpp: the CPU-first engine that made local inference possible

llama.cpp started in 2023 as a dependency-free C++ implementation of Llama inference, designed to run on the hardware developers already own: CPUs, laptops, and consumer GPUs without a CUDA datacenter behind them. It owns three things that define the local-model ecosystem:

• The GGUF format: a single-file model format bundling weights, tokenizer, and quantization metadata into one portable binary. GGUF is the de facto standard for local model distribution.
• Quantization: converting FP16 weights down to 4-bit or 2-bit integers, shrinking a 30GB model to around 4GB so it fits in laptop RAM.
• Portable backends: Metal (Apple), CUDA (NVIDIA), Vulkan, and pure CPU, with optional GPU offload controlled by a single flag.

You interact with it directly through llama-cli (experimentation) and llama-server (serving). It is the engine underneath most other local tools.

Ollama: model management plus serving, zero setup

Ollama is a model manager and server wrapped around llama.cpp-style inference, with an MLX engine on Apple Silicon. Its contribution is the experience: ollama pull qwen3-coder:30b then ollama run qwen3-coder:30b gets a working OpenAI-compatible API at localhost:11434 with no configuration. It handles model versioning, quantization selection per tag, concurrent requests, and a registry (ollama.com/library) that is the closest thing local AI has to a package manager.

Ollama 0.32.6 (August 2026) aligned /v1/chat/completions streaming with OpenAI's wire format and added automatic speculative decoding on Apple GPUs by using a model's MTP head. That makes it the default choice for single-machine development.

vLLM: production GPU serving at scale

vLLM is built for the opposite question: what happens when 10, 100, or 1,000 requests hit one model concurrently on server-class GPUs? Two mechanisms make it the throughput king:

• PagedAttention: manages the KV cache like OS virtual memory, allocating pages on demand instead of reserving full context per request. This eliminates 60-80% of KV cache waste and is why vLLM achieves 10-24x naive throughput.
• Continuous batching: interleaves token generation across in-flight requests so GPU slots never sit idle waiting for a slow request to finish.

It adds production features local tools lack: tensor parallelism across GPUs, prefix caching, LoRA adapter serving, chunked prefill, and Kubernetes-native deployment. It targets NVIDIA, AMD, and Intel GPUs plus TPUs; it is not a laptop tool.

Head-to-Head

Dimension	llama.cpp	Ollama	vLLM
Primary use	CPU/edge/local, offline	Developer machine, single box	Production serving, many users
Backends	CPU, Metal, CUDA, Vulkan	llama.cpp + MLX (Apple)	CUDA, ROCm, TPU, Intel
Model format	GGUF	GGUF (registry)	HF safetensors, AWQ, GPTQ, FP8
Concurrency	Sequential queue	Basic parallel	Continuous batching
Throughput scaling	Flat	Flat	Scales with concurrency
OpenAI-compatible API	Yes (llama-server)	Yes (/v1)	Yes (/v1)
Multi-GPU	Manual offload	No	Tensor/pipeline parallel
Setup effort	Moderate	Minimal	Moderate (production config)
Best for	Edge, IoT, offline, prototyping	Experimentation, dev machines	Team inference, SLAs, agents at scale

What the Benchmarks Actually Show

Red Hat benchmarked llama.cpp and vLLM head-to-head on a single NVIDIA H200 running Llama 3.1 8B at FP16, scaling concurrency from 1 to 64 simultaneous users:

• Single user: both engines produce tokens at a comparable rate. For one interactive session, the runtime choice barely matters.
• 64 concurrent users: vLLM generated roughly 44x more tokens per second than llama.cpp. Continuous batching keeps GPU utilization high; llama.cpp's sequential queue serializes everyone.
• Time to first token (TTFT): vLLM's p99 TTFT stays low and stable at every concurrency level. llama.cpp's TTFT grows exponentially with load — at 64 users it took over three minutes before the first token arrived.

The honest reading: these numbers are a production-serving comparison on a datacenter GPU. They tell you when to stop using a local tool, not that llama.cpp is slow on your laptop.

Decision Framework

Your situation	Pick	Why
Prototyping on a laptop, no GPU	llama.cpp or Ollama	Zero config, GGUF, offline
Development machine, want pull-and-run	Ollama	Model registry, OpenAI API, MLX on Mac
Offline/edge/IoT (factory floor, air-gapped)	llama.cpp	No dependencies, tiny footprint
Serving a model to a team or app	vLLM	Throughput, latency SLAs, batching
Kubernetes deployment with autoscaling	vLLM	Native metrics, K8s maturity
Apple Silicon laptop	Ollama or LM Studio	Metal/MLX backends, MTP speculative decoding

The pattern is almost always the same: start with Ollama on your laptop, and when the model needs to serve real traffic, swap the endpoint to vLLM. Because both expose the OpenAI API, the migration is a base URL change.

The OpenAI-Compatible Contract

All three runtimes speak the same protocol, which is why swapping between them is a configuration change, not a rewrite:

from openai import OpenAI

# Ollama on a laptop
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

# vLLM in production
client = OpenAI(base_url="http://vllm.internal:8000/v1", api_key="dummy")

# llama.cpp server
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="qwen3-coder:30b",
    messages=[{"role": "user", "content": "Refactor this module to async/await"}],
)

Your prompts, tools, streaming code, and embedding calls survive the move untouched. Only base_url and model change.

Getting Started

# llama.cpp: build and serve a GGUF
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release
./build/bin/llama-server -m models/qwen3-coder-30b-q4_k_m.gguf -ngl 99 -c 8192

# Ollama: pull and run
ollama pull qwen3-coder:30b
ollama run qwen3-coder:30b "Find the null pointer in src/auth.ts"
curl http://localhost:11434/v1/models

# vLLM: serve an HF model (GPU required)
pip install vllm
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --gpu-memory-utilization 0.9 \
    --max-model-len 32768 \
    --enable-prefix-caching

Migration Path: Ollama to vLLM

The classic growth path for a team:

1. Prototype with Ollama on a dev machine, using localhost:11434/v1.
2. Prove quality against a golden set before investing in serving infrastructure.
3. Deploy vLLM on GPU nodes (or a managed GPU provider) with the same model at higher precision than the laptop quant.
4. Switch clients by changing base_url. Keep the OpenAI SDK calls identical.

The trap to avoid: tuning your prompts against the Q4 quant on the laptop, then silently upgrading precision in production. Eval on the deployment precision, or you will ship a model whose behavior you never measured.

Implementation Checklist

• Start with Ollama or llama.cpp on developer machines for experimentation
• Use vLLM when the model serves multiple concurrent users or needs latency SLAs
• Choose llama.cpp for offline, edge, and no-GPU environments
• Keep all client code on the OpenAI-compatible API so runtimes are swappable
• Benchmark with real concurrency (1, 8, 32, 64 users) before picking a runtime
• Eval on the same precision you deploy, not the laptop quant
• For Kubernetes, prefer vLLM for metrics, autoscaling, and multi-GPU serving

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub