Best Local LLMs for the NVIDIA L40

作者： Jakub Rusinowski · 最后更新： 2026年10月7日

Best all-round pick: GLM-5.1 72B

The NVIDIA L40 has 48 GB of VRAM, of which about 48 GB is available to a model. The largest model it can hold is Nemotron 3.5 Bolt 74B (74B, 22.8 GB at Q4_K_M).

Best models overall on the NVIDIA L40

Model	Score	Memory at Q4_K_M	Context	Licence
GLM-5.1 72B	94.7	44.3 GB	125K	MIT
Qwen 3.5 72B	94.4	44.3 GB	125K	Apache 2.0
Gemma 4 27B ⭐	94	17.1 GB	125K	Gemma License (commercial OK)
Llama 3.3 70B Instruct	93.8	43.1 GB	128K	Llama Community
Qwen 3.7 35B-A3B	93.2	21.9 GB	256K	Apache-2.0
Qwen3.8 27B	92.7	17.6 GB	256K	Apache-2.0

Best model by what you are doing

Workload	Recommended on this GPU
Coding	Qwen 3.7 35B-A3B (96.9) Qwen 3.5 72B (96.6) Qwen 3.6 35B-A3B (96.2)
General assistant	GLM-5.1 72B (94.7) Qwen 3.5 72B (94.4) Gemma 4 27B ⭐ (94)
Reasoning	GLM-5.1 72B (100) Gemma 4 27B ⭐ (98.1) GLM-4.7 / GLM-Z1 GLM-Z1 32B (Reasoning) (97.7)
RAG	Gemma 4 31B (95.6) Qwen 3.6 27B (94.5) Qwen 3.7 35B-A3B (93.8)
Agents	Nemotron 3.5 Bolt 74B (91.9) Nemotron-Cascade 2 30B-A3B (91) Seed-OSS-36B-Instruct (90.5)
Vision	Qwen 3.5 72B (99.7) Qwen 3.7 35B-A3B (96.5) Qwen 3.6 35B-A3B (95.8)

How these numbers are calculated

Usable memory is 48 GB of the card's 48 GB.
Fit is measured at Q4_K_M — quantized weights plus KV cache plus 0.8 GB runtime overhead.
Within this card's budget, models are ranked on how well they use the memory available, not on being small — so the recommendation scales with the hardware.

FAQ

What is the best LLM for the NVIDIA L40?

GLM-5.1 72B is the strongest all-round pick that fits its 48 GB.

What is the largest model the NVIDIA L40 can run?

Nemotron 3.5 Bolt 74B — 74B parameters, needing 22.8 GB at Q4_K_M.

How much of the NVIDIA L40's memory can a model actually use?

About 48 GB of its 48 GB, before the desktop and runtime overhead are accounted for.

Similar GPUs

By Workload

← All GPUs | NVIDIA L40 specs