Written by Jakub Rusinowski · Last updated July 10, 2026
Two consumer GPUs give you roughly 1.5x — not 2x — the single-request speed of one, so multi-GPU inference is worth it for exactly one reason: running models too big for a single card's VRAM. This gui
Two consumer GPUs give you roughly 1.5x — not 2x — the single-request speed of one, so multi-GPU inference is worth it for exactly one reason: running models too big for a single card's VRAM. This guide explains tensor vs pipeline parallelism in plain terms, shows the honest benchmark numbers, and covers the setup in Ollama, llama.cpp, and vLLM.
Last Updated: July 2026
Tensor parallelism slices every weight matrix across GPUs; each computes its share of *every* layer, syncing after each one. Both GPUs work simultaneously — this is the only mode that speeds up a single request — but it syncs dozens of times per token, so interconnect bandwidth decides how much of the win survives. This is vLLM's --tensor-parallel-size.
Pipeline (layer) parallelism puts layers 1–40 on GPU 0 and 41–80 on GPU 1. A token flows through one GPU, *then* the other — they take turns, so a single request runs at single-GPU speed (minus a hop). This is what llama.cpp and Ollama do by default: it adds VRAM, not velocity.
Data parallelism loads a full copy of the model on each GPU and splits *requests* between them. Single-request speed unchanged; total throughput doubles. Only relevant when the model already fits on one card and you're serving many users.
Representative llama.cpp/vLLM figures for the patterns above:
| Setup | 8B Q4 model | 70B Q4 model |
|---|---|---|
| 1× RTX 3090 (24GB) | ~100–110 t/s | Doesn't fit |
| 2× RTX 3090, layer split (Ollama default) | ~95–105 t/s (no gain) | ~15–18 t/s (now it runs!) |
| 2× RTX 3090, tensor parallel (vLLM) | ~140–160 t/s (~1.5x) | ~20–25 t/s |
| 2× RTX 3090 + NVLink, tensor parallel | ~160–175 t/s | ~24–28 t/s |
Why not 2x? Tensor parallelism synchronizes the GPUs after every layer — 80 syncs per token for a 70B model. Over PCIe 4.0 x16 (~32 GB/s each way) that latency is the tax. The RTX 3090 is the last consumer card with NVLink (~112 GB/s), which is why used 3090 pairs remain the classic budget 70B rig — see the used GPU guide.
One more line item people forget: a dual-GPU box draws 700–850W under load and wants a 1200W+ PSU — the electricity cost guide has the monthly damage.
Ollama uses every visible GPU automatically, splitting layers. There is genuinely nothing to configure:
nvidia-smi # confirm both cards are visible
ollama run llama3.3:70b # splits across both automatically
# Pin to one card when you don't want splitting:
CUDA_VISIBLE_DEVICES=0 ollama serveIn raw llama.cpp you can control the ratio — useful with mismatched cards:
# 24GB card takes 60%, 16GB card takes 40%
./llama-cli -m llama-70b-q4.gguf --n-gpu-layers 999 --tensor-split 60,40Mismatched GPUs work in llama.cpp (the smaller card just caps its share); keep expectations at "the model runs," not "the model is fast."
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--quantization awq --tensor-parallel-size 2 --max-model-len 8192Rules that save you an evening of debugging: tensor parallelism wants *identical* GPUs (it splits evenly — a 24GB+16GB pair behaves like 2× 16GB), the model's attention-head count must divide by the GPU count (2, 4, 8 are safe), and going past 2 GPUs on consumer PCIe hits diminishing returns fast. Full serving context in the vLLM setup guide.
| Your situation | Verdict |
|---|---|
| Model fits on your current card, want speed | Don't add a GPU — upgrade the card or quantize down |
| Want 70B and own one 24GB card | Second identical card: the classic move |
| Choosing between 1× 4090 and 2× 3090 | 4090 for anything ≤24GB; 2× 3090 only for 70B+ ambitions |
| Serving many users, model fits on one card | Data parallelism (two instances + a load balancer), not tensor parallel |
| Mac user reading this | Skip it all — a 64GB+ Mac runs 70B on one chip (Mac guide) |
No. With tensor parallelism expect ~1.4–1.6x on consumer PCIe; with Ollama's default layer splitting expect ~1.0x (it adds memory, not speed). The doubling you're imagining only applies to *throughput* across many parallel requests, not to your single chat.
For vLLM tensor parallelism: effectively yes (even split, matched kernels). For llama.cpp/Ollama layer splitting: no — mixed cards work, with --tensor-split to balance, and total VRAM is what matters.
It roughly halves the tensor-parallel sync penalty on 3090s (the last consumer card to support it; bridges run $80–150 used). Worth it for a dedicated 2× 3090 70B rig; irrelevant for layer-split capacity mode.
Practically, no — pick one backend. Vulkan builds of llama.cpp can technically enumerate mixed vendors, but the stability and performance tax makes it a science project, not a setup.