Inkling (NVFP4) — VRAM, Speed & Local Setup

Written by Jakub Rusinowski · Last updated July 16, 2026

Model library → Inkling → Inkling (NVFP4)

The 4-bit deployment build. NVFP4 quantization brings the 975B / 41B-active MoE down to ~600 GB — a single high-memory GPU server rather than a multi-node cluster — while keeping the full 1M-token context and native text / image / audio input. Includes MTP speculative-decoding layers for faster generation and runs under vLLM, SGLang, Transformers 5.14+ and llama.cpp. Blackwell GPUs are recommended for native FP4. Apache 2.0. Thinking Machines reports AIME 2026 97.1%, GPQA Diamond 87.2%, SWE-bench Verified 77.6% and VoiceBench 91.4%. Specs from launch coverage — verify on the Hugging Face model card.

Inkling (NVFP4) needs about 589 GB of VRAM at NVFP4 (4-bit) — quantized weights plus framework overhead, before any KV cache. On Apple Silicon that figure comes out of unified memory.

Specifications

Parameters	975 Billion (41B active)
Context window	1,000,000
Architecture	Multimodal Mixture-of-Experts — 256 experts, top-6 routed + 2 shared; NVFP4 4-bit
Provider	Thinking Machines
Licence	Apache 2.0
Specified at	NVFP4 (4-bit)
System RAM	1024 GB
Record updated	2026-07-16

Licence

Apache-2.0 — commercial use permitted. Commercial use permitted. No usage restrictions beyond attribution.

VRAM and Speed by Quantization

Modelled on a reference NVIDIA RTX 4090 (24 GB), with no KV cache (this record has no published architecture). Speed figures are ESTIMATES from the memory-bandwidth roofline described on the methodology page, not benchmarks we ran — rows marked measured come from published or reader-submitted runs. VRAM here includes the KV cache, so it reads higher than the headline figure above, which does not.

Quant	Weights	VRAM needed	Est. speed	Fit on 24 GB
Q2_K	320.5 GB	321.3 GB	—	Won't fit
Q3_K_M	415.6 GB	416.4 GB	—	Won't fit
Q4_K_M	588.7 GB	589.5 GB	—	Won't fit
Q5_K_M	691.0 GB	691.8 GB	—	Won't fit
Q6_K	799.5 GB	800.3 GB	—	Won't fit
Q8_0	1035.9 GB	1036.7 GB	—	Won't fit
F16	1950.0 GB	1950.8 GB	—	Won't fit

Want the memory numbers alone, at every quantization level and your own context length? Use the Inkling (NVFP4) VRAM calculator.

Deploy in the Cloud NowRunPod

or compare on Vast.ai

As an Amazon Associate we earn from qualifying purchases. Cloud GPU links are referral links — we may earn a commission at no extra cost to you.

How to Run Inkling (NVFP4)

Install Ollama, then run:

ollama run inkling

Weights on Hugging Face: thinkingmachines/Inkling-NVFP4.

Published Benchmark Scores

Quality scores as published by the model's authors or an independent evaluator — not throughput, and not measured by us.

Benchmark	Score	Provenance
AIME 2026	97.1 %	vendor-claimed · Thinking Machines (launch)
GPQA Diamond	87.2 %	vendor-claimed · Thinking Machines (launch)
HLE (with tools)	46 %	vendor-claimed · Thinking Machines (launch)
SWE-bench Verified	77.6 %	vendor-claimed · Thinking Machines (launch)
MMMU Pro (Standard 10)	73.3 %	vendor-claimed · Thinking Machines (launch)
VoiceBench	91.4 %	vendor-claimed · Thinking Machines (launch)
Global-MMLU-Lite	88.7 %	vendor-claimed · Thinking Machines (launch)

Best for: multimodal, agentic tasks, long context, fine tuning, enterprise.

Can I Run Inkling (NVFP4) on My GPU?

Inkling on Apple M3 Ultra

Other Inkling Sizes

Inkling (BF16) — about 589 GB VRAM at Q4_K_M

Inkling (NVFP4) — Frequently Asked Questions

How much VRAM does Inkling (NVFP4) need?

About 589 GB at NVFP4 (4-bit) — quantized weights plus framework overhead, before any KV cache. The cache grows with context length and is added on top; the table above folds it in. Apple Silicon counts unified memory toward the same figure.

Does Inkling (NVFP4) run on an RTX 4090 (24 GB)?

No. Inkling (NVFP4) needs about 589 GB at NVFP4 (4-bit), more than a single RTX 4090's 24 GB. It needs a larger card, several GPUs, or Apple Silicon with enough unified memory — or it runs with part of the weights offloaded to system RAM, which is much slower.

How do I run Inkling (NVFP4) locally?

Install Ollama and run `ollama run inkling`. That pulls the weights and starts a local OpenAI-compatible endpoint; after the download nothing leaves the machine.

What other sizes does Inkling come in?

Inkling (NVFP4) (589 GB), Inkling (BF16) (589 GB). Every size shares the family's training and licence; the larger ones score higher and need proportionally more memory.

← All Inkling models | VRAM calculator | Check your own hardware