Written by Jakub Rusinowski · Last updated July 12, 2026
Two consumer GPUs give you roughly 1.5x — not 2x — the single-request speed of one, so multi-GPU inference is worth it for exactly one reason: running models too big for a single card's VRAM. This gui
Two consumer GPUs give you roughly 1.5x — not 2x — the single-request speed of one, so multi-GPU inference is worth it for exactly one reason: running models too big for a single card's VRAM. This guide explains tensor vs pipeline parallelism in plain terms, shows the honest benchmark numbers, and covers the setup in Ollama, llama.cpp, and vLLM.
Tensor parallelism slices every weight matrix across GPUs; each computes its share of *every* layer, syncing after each one. Both GPUs work simultaneously — this is the only mode that speeds up a single request — but it syncs dozens of times per token, so interconnect bandwidth decides how much of the win survives. This is vLLM's --tensor-parallel-size.
Pipeline (layer) parallelism puts layers 1–40 on GPU 0 and 41–80 on GPU 1. A token flows through one GPU, *then* the other — they take turns, so a single request runs at single-GPU speed (minus a hop). This is what llama.cpp and Ollama do by default: it adds VRAM, not velocity.
Data parallelism loads a full copy of the model on each GPU and splits *requests* between them. Single-request speed unchanged; total throughput doubles. Only relevant when the model already fits on one card and you're serving many users.
Representative llama.cpp/vLLM figures for the patterns above:
| Setup | 8B Q4 model | 70B Q4 model |
|---|---|---|
| 1× RTX 3090 (24GB) | ~100–110 t/s | Doesn't fit |
| 2× RTX 3090, layer split (Ollama default) | ~95–105 t/s (no gain) | ~15–18 t/s (now it runs!) |
| 2× RTX 3090, tensor parallel (vLLM) | ~140–160 t/s (~1.5x) | ~20–25 t/s |
| 2× RTX 3090 + NVLink, tensor parallel | ~160–175 t/s | ~24–28 t/s |
Why not 2x? Tensor parallelism synchronizes the GPUs after every layer — 80 syncs per token for a 70B model. Over PCIe 4.0 x16 (~32 GB/s each way) that latency is the tax. The RTX 3090 is the last consumer card with NVLink (~112 GB/s), which is why used 3090 pairs remain the classic budget 70B rig — see the used GPU guide.
One more line item people forget: a dual-GPU box draws 700–850W under load and wants a 1200W+ PSU — the electricity cost guide has the monthly damage.
Ollama uses every visible GPU automatically, splitting layers. There is genuinely nothing to configure:
nvidia-smi # confirm both cards are visible
ollama run llama3.3:70b # splits across both automatically
# Pin to one card when you don't want splitting:
CUDA_VISIBLE_DEVICES=0 ollama serveIn raw llama.cpp you can control the ratio — useful with mismatched cards:
# 24GB card takes 60%, 16GB card takes 40%
./llama-cli -m llama-70b-q4.gguf --n-gpu-layers 999 --tensor-split 60,40Mismatched GPUs work in llama.cpp (the smaller card just caps its share); keep expectations at "the model runs," not "the model is fast."
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--quantization awq --tensor-parallel-size 2 --max-model-len 8192Rules that save you an evening of debugging: tensor parallelism wants *identical* GPUs (it splits evenly — a 24GB+16GB pair behaves like 2× 16GB), the model's attention-head count must divide by the GPU count (2, 4, 8 are safe), and going past 2 GPUs on consumer PCIe hits diminishing returns fast. Full serving context in the vLLM setup guide.
| Your situation | Verdict |
|---|---|
| Model fits on your current card, want speed | Don't add a GPU — upgrade the card or quantize down |
| Want 70B and own one 24GB card | Second identical card: the classic move |
| Choosing between 1× 4090 and 2× 3090 | 4090 for anything ≤24GB; 2× 3090 only for 70B+ ambitions |
| Serving many users, model fits on one card | Data parallelism (two instances + a load balancer), not tensor parallel |
| Mac user reading this | Skip it all — a 64GB+ Mac runs 70B on one chip (Mac guide) |
No. With tensor parallelism expect ~1.4–1.6x on consumer PCIe; with Ollama's default layer splitting expect ~1.0x (it adds memory, not speed). The doubling you're imagining only applies to *throughput* across many parallel requests, not to your single chat.
For vLLM tensor parallelism: effectively yes (even split, matched kernels). For llama.cpp/Ollama layer splitting: no — mixed cards work, with --tensor-split to balance, and total VRAM is what matters.
It roughly halves the tensor-parallel sync penalty on 3090s (the last consumer card to support it; bridges run $80–150 used). Worth it for a dedicated 2× 3090 70B rig; irrelevant for layer-split capacity mode.
Practically, no — pick one backend. Vulkan builds of llama.cpp can technically enumerate mixed vendors, but the stability and performance tax makes it a science project, not a setup.