Written by Jakub Rusinowski · Last updated July 10, 2026
vLLM serves local LLMs at production throughput — typically 5–20x more requests per second than Ollama under concurrent load — using PagedAttention and continuous batching. This guide installs it, exp
vLLM serves local LLMs at production throughput — typically 5–20x more requests per second than Ollama under concurrent load — using PagedAttention and continuous batching. This guide installs it, explains those two ideas in plain terms, picks the right quantization (AWQ/GPTQ/FP8), and stands up an OpenAI-compatible endpoint you can put real traffic on.
Last Updated: July 2026
Be clear-eyed before installing anything — for a single user, vLLM is *not* faster:
| Ollama / llama.cpp | vLLM | |
|---|---|---|
| One user chatting | Excellent | Comparable, no win |
| 10–100 concurrent requests | Queues up, throughput collapses | This is what it's built for |
| Model format | GGUF (quantized, CPU offload) | HF safetensors, AWQ/GPTQ/FP8 |
| Hardware | Anything, incl. Macs | NVIDIA (CUDA) first-class; AMD ROCm supported |
| Ops model | Desktop app simplicity | A real server you administer |
If you're serving yourself, stay with Ollama's local API. If you're serving a team, an app, or batch pipelines — anything where requests *overlap* — vLLM is the standard answer, and the gap is dramatic: on one RTX 4090 serving an 8B model, Ollama sustains roughly 100–150 total tok/s across concurrent users, while vLLM sustains 1,000–2,500+ tok/s of aggregate throughput at 30–100 parallel requests.
PagedAttention. Every conversation needs a KV cache (the model's working memory), which naive servers allocate as one contiguous block sized for the *maximum possible* context — wasting 60–80% of VRAM on padding. vLLM manages the KV cache the way an operating system manages RAM: in small pages, allocated on demand. Same GPU, several times more concurrent conversations held in memory.
Continuous batching. Naive batching waits for every request in a batch to finish before starting the next batch — one long answer stalls everyone. vLLM schedules at the *token* level: the moment any sequence finishes, a waiting request takes its slot in the very next forward pass. The GPU never idles while there's work queued.
Neither trick helps a single request; both multiply aggregate throughput. That's why the "is vLLM faster?" answer is "at what concurrency?"
Requirements: Linux (or WSL2), Python 3.10–3.12, an NVIDIA GPU with a recent driver (Compute Capability 7.0+; 16GB+ VRAM recommended for 7–8B models at full quality).
python -m venv .venv && source .venv/bin/activate
pip install vllmThat single package includes the inference engine and the OpenAI-compatible server. Docs: docs.vllm.ai.
vLLM pulls straight from Hugging Face. For a 24GB card, an AWQ-quantized 7–8B model is the right first test:
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--api-key localtokenThe flags you'll actually touch:
--max-model-len — max context per request. Shorter = more concurrent sequences fit in the paged KV cache. Don't default to 128K "just in case."--gpu-memory-utilization — fraction of VRAM vLLM pre-claims (default 0.9). Lower it if the GPU also drives your monitor.--api-key — unlike Ollama, vLLM ships with auth. Use it.--tensor-parallel-size 2 — split across 2 GPUs when the model doesn't fit in one (multi-GPU guide).The endpoint is drop-in OpenAI-compatible — existing SDK code needs only a base URL:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="localtoken")
resp = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct-AWQ",
messages=[{"role": "user", "content": "Summarize PagedAttention in one sentence."}],
)
print(resp.choices[0].message.content)Structured output (response_format with a JSON schema) and tool calling work here too — the patterns from our structured output guide carry over unchanged.
vLLM doesn't use GGUF well — its quantization world is different:
| Format | Quality | Speed in vLLM | Use when |
|---|---|---|---|
| None (BF16) | Reference | Fast, most VRAM | VRAM is plentiful |
| FP8 | ~99% of BF16 | Fastest on Ada/Hopper (RTX 40xx+) | Modern GPU, best default |
| AWQ (4-bit) | ~97–98% | Fast | Fitting 2x the model in VRAM |
| GPTQ (4-bit) | ~96–98% | Fast | Widest choice of pre-quantized repos |
Practical rule: search Hugging Face for your model plus "AWQ" or "FP8" and serve a pre-quantized repo; only quantize yourself when nobody has. A 7B at AWQ needs ~6GB for weights, leaving the rest of a 24GB card for KV-cache pages — i.e., for concurrency.
vllm/vllm-openai), with restart-on-failure./metrics — track queue depth (num_requests_waiting) and KV-cache usage; those, not tokens/sec, tell you when to scale.For one user: no — single-stream latency is similar, and Ollama is far simpler. For concurrent load: dramatically, 5–20x aggregate throughput, because continuous batching keeps the GPU saturated while Ollama processes a shallow queue.
GGUF support exists but is experimental and slow — it bypasses vLLM's optimized paths. Serve the original HF weights, or an AWQ/GPTQ/FP8 quantization of them. Keep GGUF for the llama.cpp/Ollama world.
Not meaningfully — vLLM targets CUDA (and ROCm) datacenter-style serving. On Apple Silicon, llama.cpp-based servers are the right tool; a Mac serving a household doesn't need continuous batching anyway.
Weights once, plus KV cache per active sequence. An 8B AWQ model (~6GB) on a 24GB card leaves ~15GB of page pool — roughly 40–80 active 2K-context sequences. Longer contexts shrink that fast, which is why capping --max-model-len is the first tuning lever.