Efficiency leaderboard

Tokens per watt: which hardware runs local AI most efficiently

Speed leaderboards are everywhere. This one ranks Qwen 3 8B by how many tokens each machine produces per watt it draws — the number that decides whether a model runs quietly in the corner of your desk or turns your room into a server closet.

For a machine that lives where you work, watts are the constraint. They set the heat, the heat sets the fan speed, the fan speed sets the noise, and the total sets whether your power supply has headroom and what the electricity costs over a year of daily use. A card that is 20% faster and draws twice the power is not a better desk machine; it is a worse one that finishes sooner.

Unified-memory systems — Apple Silicon above all — lead this ranking by a wide margin, and the caveat matters as much as the result: their power figure is SoC package power, not discrete-card board power. Those are different envelopes. The comparison is still worth making, which is why it is here, but it is not like-for-like and every such row says so.

Most efficient overall1.50 tok/WApple M5 Max · SoC power

Most efficient discrete card0.82 tok/WNVIDIA RTX PRO 4000 Blackwell SFF · 70 W board

Machines that run it108Qwen 3 8B at Q4_K_M, 4K context

What this ranking rests on: 1 measured of 108 rows.Three minutes with llama-bench moves your hardware out of the estimate column →

1 measured17 modelled on a constant fitted from real runs90 on a constant nobody has calibrated

Model

Q4_K_M · 4K context · decode speed

Speed against power

Discrete card, board powerUnified memory, SoC power

Each diagonal is a constant tokens-per-watt line. Points further up and to the left do more work per watt.

Ranked by tokens per watt

Open a row to see where its numbers come from.

#	Hardware	Speed	Power	Tokens per watt
1	Apple M5 Max128 GB unified · 614 GB/s	51 tok/s	35 W SoC package	1.50
2	Apple M4 Max128 GB unified · 546 GB/s	46 tok/s	35 W SoC package	1.30
3	Apple M1 Max64 GB unified · 400 GB/s	35 tok/s	30 W SoC package	1.20
4	Apple M1 Ultra128 GB unified · 800 GB/s	64 tok/s	60 W SoC package	1.10
5	Apple M2 Ultra192 GB unified · 800 GB/s	64 tok/s	60 W SoC package	1.10
6	Apple M3 Ultra512 GB unified · 819 GB/s	65 tok/s	60 W SoC package	1.10
7	Apple M5 Max (32-core GPU)36 GB unified · 460 GB/s	40 tok/s	35 W SoC package	1.10
8	Apple M3 Max128 GB unified · 400 GB/s	35 tok/s	35 W SoC package	1.00
9	Apple M4 Max (32-core GPU)36 GB unified · 410 GB/s	36 tok/s	35 W SoC package	1.00
10	Apple M5 Pro64 GB unified · 307 GB/s	28 tok/s	30 W SoC package	0.93
11	Apple M2 Max96 GB unified · 400 GB/s	35 tok/s	40 W SoC package	0.88
12	Apple M4 Pro24 GB unified · 273 GB/s	25 tok/s	30 W SoC package	0.83
13	NVIDIA RTX PRO 4000 Blackwell SFF24 GB VRAM · 432 GB/s	58 tok/s	70 W board	0.82
14	Apple M3 Max (30-core GPU)96 GB unified · 300 GB/s	27 tok/s	35 W SoC package	0.78
15	Apple M5 Ultra512 GB unified · 1229 GB/s	89 tok/s	120 W SoC package	0.74
16	NVIDIA GeForce RTX 5070 Ti Laptop GPU12 GB VRAM · 672 GB/s	84 tok/s	115 W board	0.73
17	NVIDIA GeForce RTX 5080 Laptop GPU16 GB VRAM · 896 GB/s	105 tok/s	150 W board	0.70
18	NVIDIA GeForce RTX 5090 Laptop GPU24 GB VRAM · 896 GB/s	105 tok/s	150 W board	0.70
19	Apple M632 GB unified · 170 GB/s	16 tok/s	25 W SoC package	0.64
20	Apple M532 GB unified · 153 GB/s	14 tok/s	23 W SoC package	0.63
21	NVIDIA L424 GB VRAM · 300 GB/s	45 tok/s	72 W board	0.63
22	Apple M1 Pro32 GB unified · 200 GB/s	19 tok/s	30 W SoC package	0.62
23	Apple M2 Pro32 GB unified · 200 GB/s	19 tok/s	30 W SoC package	0.62
24	NVIDIA RTX PRO 2000 Blackwell16 GB VRAM · 288 GB/s	40 tok/s	70 W board	0.57
25	NVIDIA GeForce RTX 4090 Laptop GPU16 GB VRAM · 576 GB/s	79 tok/s	150 W board	0.53

Showing 25 of 108 · ranked by tokens per watt, ties broken by hardware id

SoC marks the 30 unified-memory systems in this ranking. Their power figure is whole-package power — CPU, GPU and memory controller on one die — where a discrete card reports board power for the card alone. Neither is the wall draw of the complete machine. They are not the same measurement, so read the ranking as a real difference in efficiency that is partly a difference in what is being counted.

How this is calculated

Decode speed comes from the same engine as the rest of the site — a three-term latency model with efficiency fitted per GPU architecture. Full methodology

tokens_per_watt = decode_tok_s / watts

watts resolves in this order:
  1. a measured draw on a submitted run, with its method   -> measured
  2. the hardware's spec power                             -> estimated
       discrete card      board power (TGP/TBP)
       unified memory     SoC package power
  3. nothing                                               -> no figure shown

We never estimate a power figure for hardware that lacks one.

购买此硬件 Apple MacBook Pro M5 Max — 128 GB unified memory · 614 GB/s 立即云端部署 RunPod 上的 RTX 4090 — 低至 $0.34/小时 · 价格核实于 2026-07

或在 Vast.ai 比较，低至 $0.35/小时 (typical low · varies)

作为亚马逊联盟成员，我们从符合条件的购买中获得收入。云 GPU 链接为推荐链接——我们可能获得佣金，您无需额外付费。

Cost per 1M tokens Performance / $ / W / VRAM The full database What running LLMs 24/7 costs