Every thinking token is a generated token, and on local hardware generated tokens are the slow part. A model that thinks for 2,000 tokens before answering makes you wait for 2,000 decode steps before the first word appears. Leave thinking on for maths, multi-step code and hard planning. Turn it off, or cap it, for classification, extraction, routing and short tool calls, where it adds seconds and rarely adds accuracy. This guide shows what thinking costs on a real card, which models document a switch for it, and how to set one in the engine you use.
Why thinking is expensive on local hardware
Answering a request has two phases. The prompt is processed in parallel (prefill), then the answer is generated one token at a time (decode). Decode is limited by memory bandwidth: every token streams the model's active weights from memory, so a card can only produce so many tokens per second, however simple the question. The MoE guide and our methodology page explain that ceiling.
Thinking tokens go through exactly the same decode path as answer tokens. There is no cheaper "internal" mode. So the extra wait is simple arithmetic: the number of thinking tokens divided by the model's decode speed. Here it is for two Qwen 3 models on an RTX 4090, one dense and one MoE, both with a documented thinking switch:
| Before the first word of the answer | Qwen 3 8B (measured, 141 tok/s) | Qwen 3 30B-A3B (MoE) (estimated, 142–274 tok/s) |
|---|---|---|
| thinking off | 0 s | 0 s |
| 512 thinking tokens | 3.6 s | 1.9–3.6 s |
| 2,048 thinking tokens | 14 s | 7.5–14 s |
| 8,192 thinking tokens | 58 s | 30–58 s |
Extra wait = thinking tokens ÷ decode speed at Q4_K_M on the RTX 4090: the measured speed where we hold benchmark runs, otherwise the modelled range, labelled in each column heading. Prompt processing comes on top. The thinking tokens also sit in the context window and the KV cache. Three things follow from that table.
- A thinking budget is a latency budget. Doubling the budget doubles the wait. A reasoning trace can run to many thousands of tokens on a hard problem, and on a local card you wait for every one before the answer starts.
- MoE models pay less per thinking token. The MoE reads only its active experts per token, so on the same card it can get through a long trace faster than a much smaller dense model, even though it is the larger model in memory. Compare the two columns: the MoE's range starts about where the dense model's speed sits.
- The trace also uses context. Thinking tokens sit in the context window and in the KV cache like any other tokens. In a long agent session, a model that keeps its old traces fills its window faster; see the context window guide.
When to turn thinking off
Thinking earns its cost when the task needs intermediate steps the model cannot do in one pass: arithmetic, multi-step code changes, planning with constraints, proofs. It rarely helps when the answer is a direct mapping from the input:
| Task | Thinking | Why |
|---|---|---|
| Classifying a ticket, routing a request, yes/no checks | Off (or use a decision model) | The answer is one label. A decision model returns a calibrated probability per option in one pass, with no generated text at all |
| Extracting fields into JSON | Off | The values are in the input; thinking adds latency and sometimes paraphrases what it should copy |
| Short tool calls in an agent loop | Off or low | Each step pays the thinking cost again; a ten-step loop multiplies it by ten |
| Summaries and rewrites | Off or low | Fluency, not deliberation |
| Maths, logic puzzles, competition-style problems | On | This is what the reasoning training is for |
| Multi-file code changes, debugging | On, with a budget | Large gains, but cap it so one hard turn cannot stall the session |
| Planning with many constraints | On | Intermediate steps are the work |
If you run an agent, the cheapest pattern is to route: a fast non-thinking call handles the easy steps, and only the steps that fail or are flagged as hard go to a thinking call. That is the same accept–escalate pattern the confidence and thresholds guide uses for decision models.
Setting a budget you can live with
Work backwards from the wait, not forwards from the model. Decide how long a user, or an agent step, can afford to wait before the answer starts. Multiply that by the model's decode speed on your hardware, and you have the most thinking tokens you can afford. The table above gives the speed for two models on one card; the VRAM calculator and each model's page give it for others, as a range, because it is modelled rather than measured unless we hold benchmark runs.
Then apply the budget per task, not per model. One cap that suits a debugging session is far too generous for a routing step, and a cap tuned for routing will cut a hard proof off halfway. Most engines let you set thinking per request, so the cheapest setup is a short list: which tasks run with thinking off, which run with a low level or a small token cap, and which get the full budget.
In an agent loop, multiply. Ten tool-calling steps that each think for a couple of thousand tokens cost ten times the single-step wait in the table, before any tool runs. That is usually the moment a "slow model" complaint turns out to be a thinking-budget problem, and the fix is turning thinking off for the routine steps rather than buying a faster card.
The documented controls, per model
Only controls the model's own card documents are listed. If a card documents no switch, the model is not listed. Behaviour there depends on the engine and the chat template, so test it rather than assume.
| Model family | Documented control | Settings | Notes from the card |
|---|---|---|---|
| gpt-oss | Reasoning level, set in the system prompt | low, medium, high (e.g. "Reasoning: high") | Low is for fast general dialogue, high for deep analysis |
| Qwen 3 | enable_thinking in the chat template (hard switch); /think and /no_think in a message (soft switch) | on by default | Do not use greedy decoding in thinking mode; the card warns of degraded output and endless repetition |
| Qwen 3.6 | enable_thinking through the API's chat-template arguments | on by default | The Qwen 3 soft switch is not officially supported. The card advises keeping at least 128K of context to preserve thinking |
| GLM-5.3 | reasoning_effort | low, high, max (max by default) | The chat template keeps earlier thinking unless clear_thinking=true is passed; the card recommends passing it for chat |
| DeepSeek V4.1 Flash | Thinking mode and a numeric reasoning effort, in DeepSeek's reference encoder | see the encoder's tests | The release ships no Jinja chat template, so whether your engine exposes the switch depends on the engine's own support |
Two of these are frontier-scale models that no single consumer machine holds (see their pages); their controls matter if you rent the hardware or serve them for a team.
Setting it in your engine
Ollama. Thinking controls vary by model, so ask the model first: /api/show returns a thinking object listing the values it supports (booleans for an on/off switch, named levels for models like gpt-oss) and its default. Then set think on the chat or generate request: true, false, or one of the listed level names, exactly as spelled. The answer comes back in message.content and the trace in message.thinking, so you can hide it or log it.
# What does this model support?
curl http://localhost:11434/api/show -d '{"model": "gpt-oss"}'
# Ask without thinking (if the model permits it)
curl http://localhost:11434/api/chat -d '{
"model": "qwen3",
"messages": [{"role": "user", "content": "Label this ticket: billing, bug or account?"}],
"think": false,
"stream": false
}'llama.cpp server. --reasoning on|off|auto turns thinking on or off, --reasoning-effort passes a level to the chat template, and --reasoning-budget N caps the trace at N tokens (0 ends it immediately, -1 is unlimited). --reasoning-format decides whether the trace comes back separated from the answer. A hard budget is the most direct way to bound latency: whatever the model wanted, it cannot think for more than N decode steps.
vLLM and SGLang. Both separate the reasoning from the answer with a reasoning parser, and the Qwen card notes that enable_thinking passes through to both as a chat-template argument. That is the usual way to switch thinking off per request on a server.
Measure it, do not assume it
Two things to check on your own workload before you settle a policy:
- Accuracy with and without thinking, on 50–100 real examples of each task. For extraction and classification, thinking often makes no measurable difference; for multi-step tasks it can be the whole difference.
- Tokens actually spent. Log the length of the trace per request. Averages hide the long tail: one request in fifty that thinks for ten times the usual length is what your users remember.
The benchmarking guide covers measuring speed fairly. Record whether thinking was on, because a run with thinking and a run without are not comparable figures.
Frequently Asked Questions
Does thinking make a local model smarter?
On tasks that need intermediate steps, such as maths, logic and multi-step code, reasoning-trained models usually do better with thinking on. On direct tasks like classification, extraction and short rewrites, the gain is usually small or absent. The cost is the same in both cases: every thinking token is generated at the model's normal decode speed.
How do I stop a model from thinking forever?
Cap the trace. llama.cpp's server takes a token budget with --reasoning-budget N. Models with named levels, such as gpt-oss, GLM-5.3 and DeepSeek V4.1 Flash, document a lower setting. Ollama lets you pick a level the model lists in /api/show, or switch thinking off where the model permits it. For Qwen 3 in thinking mode, also follow the card's sampling advice: greedy decoding is what it warns can cause endless repetition.
Is a thinking model the right choice for classifying tickets?
Usually not. For a fixed set of labels, a decision model gives you a probability per option in one forward pass, which is faster than any amount of thinking and easier to threshold. Use a thinking model when the label genuinely needs reasoning to decide.
Related Guides
Sources, checked 2026-10-14: gpt-oss card: reasoning levels · Qwen 3 card: enable_thinking, /think and /no_think, sampling advice · Qwen 3.6 card: non-thinking mode, 128K advice · GLM-5.3 card: reasoning_effort, clear_thinking · DeepSeek V4.1 Flash card: reference encoder, no Jinja template · Ollama: thinking capability, /api/show and think · llama.cpp server: --reasoning, --reasoning-effort, --reasoning-budget · vLLM: reasoning outputs.