Inference Engines Compared: Ollama, llama.cpp, LM Studio, vLLM, SGLang, MLX, TensorRT-LLM and Transformers

Pick the engine by the job, not by its benchmark chart. For one person on a laptop or desktop, use Ollama or LM Studio. On a Mac you can also run MLX. To serve many users from NVIDIA or AMD GPUs, use vLLM or SGLang. For a frontier MoE model split across several GPUs, use whichever of vLLM or SGLang the model's page lists as supported. Everything else on this page explains why, and what to check before you commit.

The reason one engine rarely wins at everything is that the engines optimise for different things. Generating tokens for a single request is limited by memory bandwidth: every engine has to stream the active weights for each token, and none can beat the bandwidth ceiling our methodology page describes. Where engines really differ is in everything around that: which weight files they read, which hardware they run on, how they handle many requests at once, and how much setup they ask of you.

The decision rule

Two warnings before the details. First, a model being "open" does not mean every engine can run it. New architectures land in Transformers first and reach the others over weeks; the serving-status chips on each model page record what was true on the date shown. Second, the file you download decides the engine more often than your preference does. A GGUF file means llama.cpp, Ollama or LM Studio. An AWQ, GPTQ or FP8 checkpoint means vLLM, SGLang or TensorRT-LLM. An MLX folder means MLX or LM Studio on a Mac.

The matrix

Read from each engine's own documentation on 2026-10-14; every cell's source is listed in that engine's section below. "—" means the documentation we read does not say, not that the feature is missing.

What actually differs

1. The weight format

GGUF is llama.cpp's format, and Ollama and LM Studio read it too. It packs the weights, the tokenizer and the chat template into one file, quantised anywhere from 1.5 to 8 bits per weight. That one-file design is why these tools are so easy to start with. The serving engines (vLLM, SGLang, TensorRT-LLM) work mainly with Hugging Face checkpoints and GPU-oriented quantisations such as FP8, AWQ and GPTQ. vLLM can load GGUF, but its own documentation calls that support "highly experimental and under-optimized" and ships it as a separate plugin. MLX uses its own converted weights and runs only on Apple Silicon.

2. One user versus many

For one request at a time, the engines land closer together than their marketing suggests, because all of them are waiting on memory bandwidth. They separate under load. vLLM and SGLang batch incoming requests continuously, store the KV cache in pages, and reuse cached prefixes across requests, so ten users sharing one system prompt cost far less than ten times one user. The llama.cpp server batches across parallel slots too, and LM Studio exposes that as "Max Concurrent Predictions". Ollama can process requests to one model in parallel, but its FAQ notes that each parallel slot adds its own context to the memory the model needs. If you serve a team, size the machine for the concurrency, not for one chat window.

3. Hardware

llama.cpp has the widest reach: NVIDIA, AMD, Apple, Intel, Vulkan GPUs and plain CPUs. Ollama and LM Studio inherit much of that. vLLM runs on NVIDIA, AMD, Intel XPU and CPU, with Apple Silicon only through a separate plugin. SGLang covers NVIDIA, AMD Instinct, Google TPU, Intel and Ascend. TensorRT-LLM is NVIDIA only, and its support matrix names the architectures it is developed and tested on. MLX is Apple only.

4. Splitting a model across devices

llama.cpp splits a model by layers across GPUs by default, which adds memory, not speed. It can also keep the experts of an MoE model in system RAM while the rest stays on the GPU. vLLM and SGLang use tensor parallelism within a machine, add pipeline parallelism across machines, and use expert parallelism for MoE models. The multi-GPU guide explains which mode buys you what.

5. Tool calls and thinking output

Every engine here offers an OpenAI-compatible API, and most can return tool calls. The difference is configuration. vLLM needs --enable-auto-tool-choice and a parser matched to the model's tool-call format. SGLang uses a per-model tool parser. Ollama and LM Studio pick that up from the model's template. Reasoning models add a second wrinkle: the engine has to separate the thinking from the answer. vLLM and SGLang do that with reasoning parsers, and llama.cpp with --reasoning-format and a thinking budget. The thinking-budget guide covers why that matters for speed.

6. Decision models

Ollama, the llama.cpp server, LM Studio and transformers serve all document a /v1/systemone endpoint for decision models: models that return a probability per option instead of text. The beyond-Ollama decision-model guide compares them.

Each engine, with its sources

Every fact below comes from the engine's own documentation, read on the date shown. A dash means the documentation we read does not say. It does not mean the feature is missing.

Ollama

Best for: one person on a laptop or desktop who wants a model running with one command. Documentation.

• Runs on: NVIDIA (compute 5.0+), AMD Radeon, Apple M-series, CPU (source)
• Weight formats: GGUF, Safetensors import (source)
• Multimodal: supported (vision models) (source)
• Tool calling: supported (source)
• Many users at once: parallel requests per loaded model; each slot adds its own context (source)
• OpenAI-compatible API: yes (source)
• Thinking control: per-model: on/off or named levels, discovered with /api/show (source)
• Decision models: /v1/systemone from v0.35.0 (source)

Checked 2026-10-14.

llama.cpp

Best for: the widest hardware reach and GGUF on anything, including CPU+GPU split for models larger than VRAM. Documentation.

• Runs on: NVIDIA (CUDA), AMD (HIP), Apple Silicon (Metal), Intel GPU (SYCL), Vulkan GPUs, CPU (x86, ARM, RISC-V) (source)
• Weight formats: GGUF, 1.5- to 8-bit (source)
• Multimodal: experimental (needs the model's multimodal projector (--mmproj)) (source)
• Tool calling: supported (source)
• Many users at once: continuous batching across parallel slots (on by default) (source)
• OpenAI-compatible API: yes (source)
• Thinking control: --reasoning on/off, --reasoning-effort, --reasoning-budget N (source)
• Decision models: /v1/systemone (source)

Checked 2026-10-14.

LM Studio

Best for: a desktop app with a model browser, and a local server when you need one. Documentation.

• Runs on: Apple Silicon, Windows x64 and ARM, Linux x64 and ARM64 (source)
• Weight formats: GGUF (llama.cpp engine), MLX (Apple Silicon) (source)
• Multimodal: supported (image input) (source)
• Tool calling: supported (source)
• Many users at once: continuous batching on the llama.cpp engine (Max Concurrent Predictions, default 4) (source)
• OpenAI-compatible API: yes (source)
• Thinking control: —
• Decision models: /v1/systemone and /v1/decisions (source)

Checked 2026-10-14.

vLLM

Best for: many users on NVIDIA or AMD GPUs, and models split across several GPUs. Documentation.

• Runs on: NVIDIA (CUDA), AMD (ROCm), Intel XPU, CPU, Apple Silicon via the vLLM-Metal plugin (source)
• Weight formats: FP8, NVFP4 / MXFP4, INT8, INT4, GPTQ, AWQ, bitsandbytes, GGUF (experimental, via plugin) (source)
• Multimodal: supported (source)
• Tool calling: supported (source)
• Many users at once: continuous batching, PagedAttention, prefix caching (source)
• OpenAI-compatible API: yes (source)
• Thinking control: reasoning parsers separate the thinking from the answer (source)

Checked 2026-10-14.

SGLang

Best for: high-throughput serving with heavy prefix reuse (agents, shared system prompts) and large MoE models. Documentation.

• Runs on: NVIDIA, AMD Instinct, Google TPU, Intel GPU and Xeon CPU, Ascend NPU (source)
• Weight formats: FP8, NVFP4, MXFP4, GPTQ, AWQ (source)
• Multimodal: supported (source)
• Tool calling: supported (source)
• Many users at once: batched serving with RadixAttention prefix caching (source)
• OpenAI-compatible API: yes (source)
• Thinking control: reasoning parser separates the thinking from the answer (source)

Checked 2026-10-14.

MLX

Best for: Apple Silicon, from Python, when you want MLX-format weights. Documentation.

• Runs on: Apple Silicon (source)
• Weight formats: MLX quantised weights (e.g. 4-bit) (source)
• Multimodal: via mlx-vlm (mlx-lm itself is text-only; vision models use the separate mlx-vlm package) (source)
• Tool calling: —
• Many users at once: batching, except with a quantised KV cache (then one request at a time) (source)
• OpenAI-compatible API: yes (similar to the OpenAI chat API; the docs say the server is not recommended for production) (source)
• Thinking control: —

Checked 2026-10-14.

TensorRT-LLM

Best for: squeezing the most throughput out of NVIDIA datacenter GPUs, at the cost of setup. Documentation.

• Runs on: NVIDIA Blackwell, Hopper, Ada Lovelace, Ampere (source)
• Weight formats: FP8, NVFP4, INT8, W4A16 / W4A8 AWQ, W4A16 / W4A8 GPTQ (source)
• Multimodal: supported (source)
• Tool calling: —
• Many users at once: in-flight batching (source)
• OpenAI-compatible API: yes (source)
• Thinking control: —

Checked 2026-10-14.

Transformers

Best for: evaluation and research, and models no other engine supports yet. Documentation.

• Runs on: —
• Weight formats: —
• Multimodal: supported (text, image, audio and video requests) (source)
• Tool calling: supported (models whose tokenizer declares tool-call tokens) (source)
• Many users at once: continuous batching (flag) (source)
• OpenAI-compatible API: yes (source)
• Thinking control: —
• Decision models: /v1/systemone (source)

Checked 2026-10-14.

How model pages use this

Each model page carries a "Runs with" row: one chip per engine, with a status (supported, experimental, pending or not supported), the URL the status was read from, and the date. The chip links to that engine's section above. A status describes the model at that date, not the engine in general. If a model's page lists no chips, we hold no dated statement for it yet; check the model card.

Before you move a working setup to a new engine, measure on your own machine: the benchmarking guide shows how to compare two engines fairly.

Frequently Asked Questions

Is vLLM faster than Ollama?

For one person sending one request at a time, the two are much closer than their reputations suggest, because both are limited by how fast the GPU can read the weights. vLLM pulls ahead when many requests arrive together, which is what continuous batching and paged KV caching are for. If you are the only user, choose by convenience; if you serve a team, test both at your real concurrency.

Can vLLM load a GGUF file?

Yes, through the separate vllm-gguf-plugin, but vLLM's own documentation calls GGUF support highly experimental and under-optimised. If GGUF is what you have, llama.cpp, Ollama or LM Studio will serve it with less friction. If you want vLLM, download the model's Hugging Face checkpoint or an AWQ, GPTQ or FP8 build instead.

Which engine should I use on a Mac?

Ollama or LM Studio for most people. LM Studio can run both GGUF (through llama.cpp) and MLX models on Apple Silicon, so you can compare the two formats in one app. Use mlx-lm directly if you work in Python. Note that the mlx-lm server's own documentation says it is not recommended for production.

Related Guides

• → Ollama vs LM Studio vs Jan
• → vLLM: Production-Speed Serving
• → Setting up a Local API Server
• → Multi-GPU LLM Inference
• → vLLM in 2026: when to use it instead of Ollama
• → Benchmark your local LLM setup

Sources, checked 2026-10-14: vLLM: GGUF support is experimental, via plugin · vLLM: tool calling flags · vLLM: reasoning outputs · llama.cpp server: split modes, --cpu-moe, reasoning flags, continuous batching · Ollama FAQ: parallel requests and context memory · LM Studio: parallel requests · LM Studio: GGUF and MLX models · SGLang: tool parser · TensorRT-LLM: support matrix · mlx-lm server: not recommended for production.