LLM Glossary

Large Language Model (LLM)#

Also called: LLM, language model

A neural network trained on very large amounts of text to predict the next token, which turns out to be enough to summarise, translate, write code, and answer questions. Modern LLMs are transformers with billions of parameters. "Local LLM" means one whose weights you have downloaded and run on your own hardware, with no request leaving your machine.

Tokens#

The basic unit of text for an LLM. Words, sub-words, or characters. Roughly 1,000 tokens ≈ 750 English words. Tokenization varies by model — GPT-4 uses tiktoken, Llama uses SentencePiece. Pricing for cloud APIs is per token.

Tokenizer#

The component that converts raw text into token IDs (numbers) before feeding it to the model, and converts model output IDs back to text. Each model family has its own tokenizer — Llama uses SentencePiece, GPT-4 uses tiktoken. Different tokenizers produce different token counts for the same text.

Parameters (Weights)#

Also called: weights, model size

The numbers learned during training that define what the model knows. A '7B' model has 7 billion parameters. More parameters = more world knowledge and reasoning ability, but requires more VRAM (roughly 2GB per 1B params at full precision, 0.5GB/B at Q4).

Inference#

Running the model to generate text from an input prompt — the 'use' phase, distinct from training. Inference can be done on CPU or GPU. GPU inference is 5–50x faster. Local inference = running the model on your own hardware without sending data to the cloud.

Base Model#

Also called: pretrained model, foundation model

A model straight out of pretraining, before any instruction tuning. It continues text rather than answering questions, so a base model given "What is the capital of France?" may produce more quiz questions instead of an answer. Base models are the starting point for fine-tuning; for chat use, download the instruct variant instead.

Instruct Model#

Also called: chat model, instruction-tuned model

A base model that has been fine-tuned to follow instructions and hold a conversation. Filenames and tags mark it with "-Instruct", "-it", or "-chat". This is what you want for almost every local use case — an instruct model expects a chat template with roles, while a base model expects raw text to continue.

Open-Weight Model#

Also called: open weights

A model whose trained weights are published for download, so anyone can run, quantize, and fine-tune it locally. Open weights are not the same as open source: the training data and code usually stay private, and the licence may restrict commercial use. Llama, Qwen, DeepSeek, Mistral, Gemma, and gpt-oss are all open-weight families.

Open-Source Model#

A model that ships weights, training code, and enough data documentation under an OSI-approved licence for someone else to reproduce it. Most models called "open source" in marketing are only open-weight. The distinction matters for compliance reviews: an Apache 2.0 weights release still is not a reproducible open-source system.

Local AI#

Also called: on-device AI, self-hosted AI, private AI

Running AI models on hardware you control — a laptop, desktop GPU, or your own server — instead of calling a hosted API. The payoffs are privacy (no prompt leaves the machine), fixed cost after hardware, offline availability, and no rate limits. The trade-off is that you are capped by your own VRAM and bandwidth.

Small Language Model (SLM)#

Also called: SLM, tiny model

A language model small enough to run on modest hardware — usually under about 4B parameters, and often under 1B. SLMs like Qwen3 0.6B, Gemma 3 270M, and SmolLM run on phones, laptops without discrete GPUs, and edge devices. They are weaker at open-ended reasoning but excellent at classification, extraction, and routing.

Prompt#

The full text sent to a model for one generation — system instructions, conversation history, retrieved documents, and your latest message combined. Everything in the prompt consumes context and, on a cloud API, costs input tokens. On local hardware, a longer prompt means a longer prefill before the first token appears.

System Prompt#

Also called: system message

A separate instruction block placed before the conversation that sets the model's role, tone, and rules. Because it sits at the very start of the context, it is the ideal thing to cache: prefix caching makes an unchanged system prompt nearly free on every subsequent request in the same session.

Chat Template#

Also called: prompt template, ChatML

The exact formatting a chat model expects, with special tokens marking where each role's turn starts and ends. Every family has its own — ChatML, Llama 3, Gemma, and Mistral templates are all different. Using the wrong template is the single most common cause of a local model rambling, ignoring instructions, or never stopping.

Special Tokens#

Also called: BOS, EOS, control tokens

Reserved tokens that carry structure rather than text — beginning-of-sequence (BOS), end-of-sequence (EOS), turn delimiters, and tool-call markers. The runtime must know the model's EOS token to stop generating; a mismatch between the GGUF metadata and the template is why some models produce endless output.

Model Card#

The README published alongside a model's weights, documenting its architecture, training data, intended use, licence, benchmark scores, and known limitations. On Hugging Face it is the repository's main page. Read the licence and the recommended chat template sections before downloading anything for production use.

Checkpoint#

A saved snapshot of a model's weights at a point in training. Released models are simply the checkpoint the authors chose to publish. During fine-tuning you save checkpoints periodically so you can roll back if quality degrades, and pick the one with the best validation score rather than the last one.

Hallucination#

Also called: confabulation

A fluent, confident model output that is factually wrong. It is a direct consequence of next-token prediction: the model optimises for plausible text, not truth. Retrieval grounding, lower temperature, and asking for citations reduce it; no local or cloud model eliminates it, so anything consequential needs verification.

Reasoning Model#

Also called: thinking model, chain-of-thought model

A model trained to produce an internal chain of thought before its final answer, trading latency and tokens for accuracy on maths, logic, and code. DeepSeek R1 and the QwQ line popularised it locally. Budget for the extra output: a reasoning model can spend thousands of thinking tokens on one question, and they occupy context like any others.

Multimodal Model#

Also called: VLM, vision-language model

A model that accepts more than text — usually images, sometimes audio or video — by encoding them into the same token stream. Locally these need a vision projector file alongside the weights, and images consume a surprising number of tokens: a single high-resolution image can cost more context than a page of text.

Distillation#

Training a small "student" model to imitate a larger "teacher" model's outputs, transferring much of its capability into a fraction of the parameters. Most strong small models today are distilled. The name in a filename — for example a "DeepSeek-R1-Distill" variant — tells you it is a smaller base model taught by a larger reasoning model, not the original.

Zero-Shot & Few-Shot Prompting#

Also called: in-context learning

Zero-shot means asking the model to do a task with instructions only; few-shot means including a handful of worked examples in the prompt first. Few-shot reliably improves format adherence on smaller local models, at the cost of the context those examples occupy — and prefix caching makes a fixed example block cheap to reuse.

VRAM (Video RAM)#

The memory on your GPU. This is the critical bottleneck for local LLM inference — the entire model (weights) must fit in VRAM or the GPU falls back to slow CPU RAM. More VRAM = bigger models. RTX 4090 has 24GB; RTX 4060 has 8GB.

GPU Offloading#

Also called: layer offloading, n_gpu_layers, ngl

Splitting a model between GPU and CPU by placing only some layers in VRAM. In llama.cpp this is the `n_gpu_layers` (or `-ngl`) setting. Offloading lets a model larger than your VRAM run at all, but every layer left on the CPU costs speed dramatically — partial offload is typically 3–10x slower than a full fit.

Unified Memory#

Also called: UMA, shared memory

A design where CPU and GPU share one pool of RAM instead of each having its own — the defining feature of Apple Silicon for local AI. A 64GB Mac can allocate most of that to a model, so it runs LLMs that would need a data-centre GPU. The catch is bandwidth: unified memory is far slower than a discrete card's GDDR or HBM, so large models load but generate slowly.

Apple Silicon#

Also called: M-series, Mac

Apple's M-series chips (M1 through M5 and their Pro/Max/Ultra variants), which run local LLMs through the GPU and Neural Engine using unified memory. Practical capacity is roughly 70% of total RAM. Max and Ultra tiers have several times the memory bandwidth of the base chip, which is what actually determines tokens per second.

Memory Bandwidth#

Also called: GB/s, bandwidth

How fast a GPU can read data from its own memory, in GB/s — the number that actually predicts local token generation speed. Generating each token requires reading the whole active model once, so tokens per second is roughly bandwidth divided by model size in bytes. This is why an 8GB model on a 1,000 GB/s card runs near 100 t/s regardless of the card's raw compute.

Memory-Bound vs Compute-Bound#

Whether a workload is limited by memory bandwidth or by arithmetic throughput. Single-user local generation is memory-bound — the GPU sits idle waiting for weights — which is why bandwidth beats TFLOPS for chat. Prefill and large-batch serving are compute-bound, which is where tensor cores and FLOPS start to matter.

CUDA#

NVIDIA's parallel-computing platform and the default path for GPU acceleration in nearly every LLM runtime. Practically, it means an NVIDIA card plus a matching driver and toolkit version. Most local-AI installation failures are CUDA version mismatches between the driver, the toolkit, and the compiled runtime.

ROCm#

Also called: HIP, AMD GPU support

AMD's open compute stack, the CUDA equivalent for Radeon and Instinct GPUs. Support in llama.cpp, Ollama, and vLLM is real but narrower than NVIDIA's: officially supported cards, kernel driver versions, and the HSA override environment variables are the usual friction points on consumer RDNA hardware.

Metal (MPS)#

Also called: MPS, Metal Performance Shaders

Apple's GPU API, and the backend llama.cpp and PyTorch use to accelerate models on Mac. Metal support is why a MacBook runs local LLMs at usable speed without any CUDA equivalent. In PyTorch the device is called `mps`; in llama.cpp it is compiled in by default on macOS.

Vulkan#

A cross-vendor graphics and compute API that llama.cpp can use as a GPU backend when CUDA or ROCm is unavailable. It runs on NVIDIA, AMD, Intel Arc, and many integrated GPUs from a single build. Expect a speed penalty versus the vendor-native backend, but it is often the fastest route to any GPU acceleration at all.

Tensor Cores#

Dedicated matrix-multiply units on NVIDIA GPUs since Volta, and the reason modern cards are so much faster at low-precision maths. Each generation adds formats — FP16 and BF16, then FP8 on Hopper and Ada, then FP4 on Blackwell. A quantization format only runs at full speed if the GPU has tensor cores for that precision.

Compute Capability#

Also called: SM version, CC

NVIDIA's version number for a GPU architecture's feature set — 8.6 for Ampere consumer cards, 8.9 for Ada, 9.0 for Hopper, 12.0 for Blackwell. Runtimes use it to decide which kernels can run. A card below the minimum compute capability of a prebuilt binary will fail to load the model even with a current driver.

TFLOPS#

Also called: FLOPS

Trillions of floating-point operations per second — a GPU's raw arithmetic throughput, always quoted for a specific precision. It predicts prompt-processing and batch-serving speed well and single-user token generation poorly, because generation is bandwidth-limited. Compare TFLOPS at the same precision or the numbers are meaningless.

HBM#

High Bandwidth Memory — stacked DRAM placed next to the GPU die, used on data-centre accelerators like the H100, H200, and MI300 series. It delivers several terabytes per second, an order of magnitude beyond consumer GDDR, which is why one server GPU can serve many concurrent users at speeds a desktop card cannot match.

GDDR#

The graphics DRAM used on consumer GPUs — GDDR6, GDDR6X, and GDDR7 on current cards. Bandwidth is the memory bus width multiplied by the effective clock, so a 384-bit card massively outruns a 128-bit one at the same generation. For local LLM work, bus width and generation matter more than the advertised clock speed.

System RAM#

Also called: DDR, CPU RAM

Your motherboard's DDR memory, used for CPU inference, for layers offloaded from the GPU, and to stage weights during loading. As a rule, keep at least as much system RAM as the largest model file you intend to load. DDR5 bandwidth is roughly 5–10% of a discrete GPU's, so CPU-resident layers dominate total generation time.

Out of Memory (OOM)#

Also called: CUDA out of memory, OOM error

The error you get when a model, its KV cache, and framework overhead exceed available VRAM. The fix order is: reduce context length, quantize the KV cache, drop to a smaller quantization, offload fewer layers, or pick a smaller model. Context length is the usual culprit — the KV cache grows linearly with it while the weights stay fixed.

TDP#

Also called: power draw, wattage

Thermal Design Power — the sustained wattage a GPU is built to dissipate, and a good proxy for both PSU sizing and running cost. A 450W card under continuous inference draws close to that figure, so a full-time local setup costs real money in electricity; that draw is a line item in any honest local-versus-cloud comparison.

NVLink#

NVIDIA's high-speed direct GPU-to-GPU interconnect, far faster than routing traffic over PCIe. It matters for tensor parallelism, where cards exchange activations every layer. Consumer cards after the RTX 3090 dropped NVLink, so multi-GPU desktop rigs communicate over PCIe and lose throughput on tensor-parallel serving.

PCIe Bandwidth#

Also called: PCIe lanes, x16

The link speed between GPU and motherboard, which governs model load time and any traffic between cards. For single-GPU inference it barely matters once the model is resident — even x4 works. It becomes a real bottleneck for multi-GPU tensor parallelism and for setups that stream layers from system RAM each token.

NPU#

A Neural Processing Unit — a low-power accelerator for neural networks built into recent laptop SoCs from Intel, AMD, Qualcomm, and Apple. NPUs excel at small always-on models within a tight power budget. Software support for general LLM inference is still thin, so most local runtimes still target the GPU instead.

Quantization#

Reducing the precision of a model's weights (e.g., from 32-bit float to 4-bit integer) to shrink memory usage with minimal quality loss. A 7B model in FP16 needs ~14GB VRAM; in Q4_K_M it needs ~4GB.

Bits per Weight (BPW)#

Also called: BPW, bits per parameter

The average number of bits used to store each parameter, and the single number that predicts a quantized file's size. Multiply parameters by BPW and divide by eight for bytes: 8B parameters at 4.5 BPW is about 4.5GB. Formats are not exactly their nominal bit count — Q4_K_M averages roughly 4.8 BPW because some tensors are kept at higher precision.

Q4_K_M#

The default recommendation for local LLMs: a 4-bit K-quant, medium size, averaging roughly 4.8 bits per weight. It is the point where the size/quality curve bends — perplexity is within about 1% of FP16 while the file is roughly a quarter the size. Go to Q5_K_M or Q6_K if you have VRAM to spare, Q3_K_M only if you must.

K-Quants#

Also called: Q4_K, Q5_K, Q6_K

llama.cpp's family of block-wise quantization formats, written Q<bits>_K_<size> — for example Q4_K_S, Q4_K_M, Q6_K. Weights are grouped into blocks with their own scale factors, and important tensors (attention and embeddings) are kept at higher precision than the rest, which is why K-quants beat older flat formats at the same file size.

I-Quants (IQ)#

Also called: IQ quants

llama.cpp's newer importance-aware quantization formats, written IQ2_XS, IQ3_M, IQ4_XS and similar. They use codebook lookups guided by an importance matrix to hold quality at very low bit rates, so IQ3 often beats Q3_K. The trade-off is more compute per token, which costs speed on weak GPUs and CPU-only setups.

Importance Matrix (imatrix)#

Also called: imatrix

A profile of which weights matter most, computed by running calibration text through the model and recording activation magnitudes. llama.cpp uses it to allocate precision where it counts during quantization. An imatrix-quantized file is measurably better than the same format without one, at zero extra size or inference cost.

Calibration Dataset#

A small sample of representative text — typically a few hundred sequences — pushed through a model to measure activation ranges before quantizing. GPTQ, AWQ, and imatrix all need one. Calibrating on text unlike your real workload is a quiet quality killer: a model calibrated only on English prose degrades more on code or other languages.

FP16 & BF16#

Also called: half precision, bfloat16

The two 16-bit floating-point formats models are trained and released in. Both use two bytes per parameter, so weights alone need roughly 2GB per billion parameters. BF16 trades mantissa bits for FP32's exponent range, which makes it far more stable in training; FP16 has more precision but overflows more easily. For inference the difference is negligible.

FP8#

Also called: E4M3, E5M2

An 8-bit floating-point format with hardware support from NVIDIA Hopper and Ada onward, and on recent AMD accelerators. It halves memory versus BF16 with very little quality loss and is the default production precision in vLLM and SGLang on modern data-centre cards. FP8 KV cache is a separate, equally useful lever.

FP4 (NVFP4 & MXFP4)#

Also called: NVFP4, MXFP4, FP4

Two competing 4-bit floating-point formats with native hardware support on Blackwell-class GPUs. NVFP4 is NVIDIA's, using small blocks with FP8 scales for higher accuracy; MXFP4 is the Open Compute Project standard, with larger blocks and simpler hardware. Both deliver 4-bit memory savings at close to FP8 quality — gpt-oss shipped natively in MXFP4.

INT8 & INT4#

Also called: W8A8, W4A16

Integer quantization formats that map floating-point weights onto 8-bit or 4-bit integers using a scale and zero point. INT8 is nearly lossless and widely supported; INT4 needs group-wise scales and a good calibration set to hold up. GPTQ and AWQ both produce INT4 weights — the difference is in how they choose the scales.

GPTQ#

A one-shot post-training quantization method that compresses weights layer by layer, using second-order error information to compensate as it goes. It produces 4-bit GPU-ready weights that run fast in vLLM, ExLlama, and Transformers. GPTQ needs a calibration pass, and is generally a little more calibration-sensitive than AWQ.

AWQ#

Activation-aware Weight Quantization — a 4-bit method that identifies the small fraction of weight channels most sensitive to activation magnitude and protects them with per-channel scaling. It generally holds instruction-following quality better than GPTQ at the same bit width and is a common default for serving quantized models in vLLM.

EXL2 & EXL3#

Also called: ExLlamaV2 format, exl2

The ExLlama quantization formats, built for fast single-user inference on consumer NVIDIA GPUs. Their distinguishing feature is fractional bit rates — you can target 4.65 bits per weight to fill your VRAM exactly instead of jumping between fixed tiers. EXL3 improves accuracy at low bit widths over EXL2.

bitsandbytes#

Also called: NF4, load_in_4bit

The library that adds on-the-fly 8-bit and 4-bit quantization to Hugging Face Transformers, via `load_in_4bit` and the NF4 data type. It is the standard way to load a large model onto a small GPU for QLoRA fine-tuning. For pure inference throughput, a pre-quantized GPTQ, AWQ, or GGUF file is faster.

KV Cache Quantization#

Also called: KV cache quant, quantized KV

Storing the attention key/value cache at lower precision — typically 8-bit, sometimes 4-bit — instead of FP16. It roughly halves or quarters the memory that long contexts consume, which is often the difference between a 32k context fitting or not. Q8 KV is close to free in quality terms; Q4 keys visibly degrade recall on long documents.

Quantization-Aware Training (QAT)#

Also called: QAT

Training or fine-tuning a model while simulating quantization error, so the final weights are already robust to being compressed. QAT releases — Gemma has shipped them — beat post-training quantization of the same model at the same bit width. Look for "QAT" in a repository name when choosing between 4-bit versions.

GGUF#

The file format used by llama.cpp and Ollama for quantized models. Stands for 'GPT-Generated Unified Format'. A single .gguf file contains the full model weights and metadata. Quantization level is encoded in the filename (e.g., Q4_K_M, Q8_0, F16).

GGML#

The predecessor to GGUF, and still the name of the underlying C tensor library that llama.cpp is built on. As a file format it is obsolete — modern llama.cpp cannot load .ggml files. If you find one in an old tutorial, look for the GGUF re-upload of the same model instead of trying to convert it.

safetensors#

The standard format for distributing unquantized model weights, designed to replace Python pickle. It stores tensors in a flat binary layout with a JSON header, so files memory-map instantly and cannot execute code on load. Anything you download from Hugging Face for use with Transformers, vLLM, or fine-tuning will be safetensors.

PyTorch .bin (Pickle)#

Also called: pytorch_model.bin

The legacy PyTorch checkpoint format, serialized with Python pickle. It is a security risk: loading a .bin file can execute arbitrary code embedded by whoever uploaded it. Prefer safetensors for anything from an untrusted source, and treat a repository that offers only .bin files as a reason for caution.

Sharded Weights#

Also called: model shards

A model split across several files — model-00001-of-00004.safetensors and so on — with an index JSON mapping tensor names to shards. Sharding keeps individual files under hosting limits and lets loaders stream. You need every shard plus the index; a partial download fails with a missing-tensor error rather than a clear message.

Hugging Face Hub#

Also called: HF Hub, huggingface.co

The main repository for open-weight models, datasets, and quantized conversions, addressed as `org/model-name`. Practically every local runtime downloads from it. Gated models require accepting a licence and supplying an access token, which is the usual reason an otherwise correct download command returns a 401.

Repo ID#

The `organization/model` identifier that uniquely names a model on Hugging Face — for example `meta-llama/Llama-3.3-70B-Instruct`. Quantized re-uploads keep the original name with a suffix and a different owner, so the repo ID is how you tell an official release from a community conversion of it.

Modelfile#

Ollama's build recipe — a short text file that names a base model and layers on a system prompt, chat template, sampling defaults, and adapters. `ollama create mymodel -f Modelfile` turns it into a reusable local model. It is the cleanest way to pin one set of parameters for a whole team without editing application code.

Ollama Tag#

The `model:variant` identifier Ollama uses to pull and run weights — for example `llama3.3:70b` or `qwen3:8b-q4_K_M`. The bare tag `:latest` resolves to a default quantization, usually Q4_K_M, which surprises people who assume they downloaded full precision. Name the quantization explicitly for reproducible setups.

Multimodal Projector (mmproj)#

Also called: mmproj, vision projector

A separate weights file that maps image encoder output into the language model's token embedding space. Vision models in GGUF need both the model file and its matching mmproj file — loading the model alone yields a text-only assistant that appears to ignore images. The projector must come from the same conversion as the model.

config.json#

The architecture manifest in a Hugging Face model repository: layer count, hidden size, attention head counts, vocabulary size, RoPE settings, and maximum position embeddings. Loaders read it to build the model before loading weights. It is also where you verify a model's real trained context length rather than a marketing claim.

Context Window#

The maximum number of tokens a model can process at once — its working memory. Llama 3.1 has 128k tokens (~96,000 words). Larger context = more expensive in VRAM. Context is consumed by your prompt, conversation history, and the response.

KV Cache#

Key-Value cache: stores intermediate attention computations so the model doesn't recompute the entire context on every new token. Larger context windows require more KV cache memory. On a 7B model with 8k context, the KV cache can use 1–2GB of VRAM on top of the model weights.

Max Output Tokens#

Also called: max_tokens, num_predict

The cap on how many tokens a single response may contain, set separately from the context window. Output tokens are drawn from the same context budget as the prompt, so prompt length plus max output must fit inside the window. Setting it too low is why answers get cut off mid-sentence.

Long Context#

Context windows beyond roughly 32k tokens, now common at 128k and reaching 1M in some releases. The constraint locally is almost never the model — it is KV cache VRAM, which scales linearly with context length. Retrieval quality also degrades well before the advertised maximum, so a 128k window rarely means 128k of usable recall.

Effective Context#

The context length at which a model still reliably uses information, as opposed to the maximum it technically accepts. Long-context benchmarks routinely show accuracy falling sharply past a fraction of the advertised window. Treat the number on the model card as a ceiling and validate on your own documents.

Lost in the Middle#

The well-documented tendency of LLMs to use information at the start and end of a long context more reliably than material buried in the middle. The practical response is ordering, not length: put the critical instruction last, place key documents at the edges, and retrieve fewer, better chunks instead of padding the window.

Context Overflow#

Also called: truncation, context shift

What happens when a conversation exceeds the context window. Runtimes respond differently — some truncate the oldest turns, some error out, some silently shift the window and drop the system prompt with it. Losing the system prompt to a silent shift is a common cause of an assistant "forgetting" its instructions mid-session.

Prompt Caching#

Reusing the computed KV cache for a prompt prefix that has already been processed, so repeated system prompts, few-shot examples, and conversation history do not have to be recomputed. Locally this is prefix caching in llama.cpp, vLLM, and SGLang; on cloud APIs it appears as a discounted cached-input token rate.

Token Budget#

The deliberate allocation of a context window across system prompt, retrieved documents, conversation history, tool definitions, and room for the answer. Agent stacks blow their budget on tool schemas more often than on content — verbose tool definitions can eat a large share of the window before any real work starts.

Ollama#

The most popular local LLM runtime. A CLI + REST API that manages model downloads, quantization selection, and inference with a simple interface. Compatible with the OpenAI API format. Run a model with one command: `ollama run llama3.2`. Supports Mac, Windows, and Linux.

llama.cpp#

The C/C++ inference engine that most consumer local-AI tools are built on, including Ollama and LM Studio. It runs GGUF models across CUDA, ROCm, Metal, Vulkan, and plain CPU, with partial GPU offload when a model does not fit. Using it directly gives you every sampling and cache flag the wrappers hide.

llama-server#

llama.cpp's built-in HTTP server, exposing an OpenAI-compatible endpoint plus a web UI. It is the lightest way to put a GGUF model behind an API for other applications, with direct control over context size, GPU layers, KV cache type, and slot count for concurrent requests.

LM Studio#

A desktop application for running local models with a graphical interface — model browser, chat window, and a local OpenAI-compatible server. It handles GGUF and MLX, and shows estimated VRAM fit before you download. The usual first stop for people who want local AI without a terminal.

vLLM#

A high-throughput inference server built for serving many concurrent users, using PagedAttention, continuous batching, and prefix caching. It targets full-precision and GPU-quantized formats — FP8, AWQ, GPTQ, and increasingly FP4 — rather than CPU offload. For one user on one desktop GPU, llama.cpp is simpler; past a handful of concurrent requests, vLLM wins decisively.

SGLang#

A serving engine focused on structured generation and aggressive prefix reuse, built around RadixAttention — a radix-tree KV cache that shares prefixes across requests automatically. It is particularly strong for agent workloads where many calls share a long system prompt or few-shot block.

TensorRT-LLM#

NVIDIA's compiled inference library, which builds a hardware-specific engine for a given model, precision, and batch shape. It delivers the best throughput available on NVIDIA hardware at the cost of a build step and much less flexibility — change the model or the max batch size and you rebuild the engine.

ExLlamaV2 / ExLlamaV3#

Also called: ExLlama

A fast inference library for quantized models on consumer NVIDIA GPUs, paired with the EXL2/EXL3 formats. Its niche is squeezing maximum single-user speed out of a card whose VRAM you want to fill exactly, thanks to fractional bit-rate quantization. No CPU offload — the model must fit.

MLX#

Apple's array framework for machine learning on Apple Silicon, with an LLM stack (`mlx-lm`) that runs and fine-tunes models using unified memory. MLX builds are typically faster than GGUF on the same Mac and support LoRA fine-tuning locally — the practical route to training on a laptop.

Hugging Face Transformers#

Also called: transformers

The reference Python library for loading and running models from the Hub. It supports every architecture first, which makes it the way to run a brand-new model on release day before quantized conversions appear. It is the slowest option for serving, so it is a research and prototyping tool, not a production server.

Open WebUI#

A self-hosted browser front end for local models, typically pointed at Ollama or any OpenAI-compatible endpoint. It adds multi-user accounts, conversation history, document upload with built-in RAG, and model switching — the missing interface layer between a raw inference server and something a team will actually use.

llamafile#

A single executable file that bundles model weights with the llama.cpp runtime, producing one artefact that runs on Windows, macOS, and Linux without installation. It is the simplest way to hand a working local model to a non-technical colleague or to run one on a machine where you cannot install software.

OpenAI-Compatible API#

Also called: /v1/chat/completions

The de facto standard local-inference interface: `/v1/chat/completions` with the same request and response shape as OpenAI's API. Ollama, llama-server, vLLM, SGLang, and LM Studio all expose it, so switching a client from a cloud provider to a local model usually means changing a base URL and a dummy API key.

Tokens per Second (t/s)#

Also called: t/s, tok/s, TPS

The standard speed metric for local inference: how many tokens the model generates each second. Roughly 5 t/s is painful, 15–20 t/s reads as comfortable real-time, and above 40 t/s is faster than most people read. For single-user generation it is governed by memory bandwidth divided by the size of the active weights.

Time to First Token (TTFT)#

Also called: TTFT, first-token latency

How long you wait between sending a prompt and seeing the first token — the latency users actually perceive. It is dominated by prefill, so it grows with prompt length while generation speed stays flat. Prefix caching is the highest-leverage fix, since it removes prefill entirely for a repeated prefix.

Inter-Token Latency (ITL)#

Also called: ITL, TPOT, time per output token

The gap between consecutive generated tokens, the reciprocal of tokens per second. It is the metric that matters once streaming has started: steady 50ms gaps feel smooth, while erratic gaps feel broken even at the same average. Under batching, ITL is where contention between concurrent requests shows up first.

Prefill#

Also called: prompt processing, prompt eval

The first phase of inference, where the model processes the entire prompt in parallel and builds the KV cache. It is compute-bound and fast per token — thousands of tokens per second — but its total cost scales with prompt length, which is why long prompts delay the first token even on fast hardware.

Decode#

Also called: generation phase, autoregressive decoding

The second phase of inference, generating one token at a time, each conditioned on everything before it. Decode is memory-bound: every token requires reading the model's active weights from VRAM again. This sequential, bandwidth-limited loop is why generation is orders of magnitude slower per token than prefill.

Chunked Prefill#

Splitting a long prompt into fixed-size chunks — often 512 or 2,048 tokens — so prefill work interleaves with ongoing decoding instead of blocking it. Without it, one user pasting a huge document stalls every other request on the server. It is standard in vLLM and SGLang and trades a little TTFT for far better tail latency.

Prefix Caching#

Also called: automatic prefix caching, APC, RadixAttention

Keeping the KV cache for a shared prompt prefix so later requests skip recomputing it. System prompts, few-shot blocks, RAG context, and multi-turn history all repeat, so hit rates are high in practice. The effect is large: a cached 8k-token prefix can cut time to first token by an order of magnitude.

Continuous Batching#

Also called: in-flight batching, dynamic batching

A scheduling technique where new requests join the running batch as soon as any sequence finishes, instead of waiting for the whole batch to complete. It keeps the GPU saturated and can multiply serving throughput several times over static batching. It is the core reason vLLM-class servers outperform naive loops under load.

PagedAttention#

A KV cache allocator that stores attention state in fixed-size blocks, like virtual memory pages, rather than one contiguous slab per sequence. It nearly eliminates the memory fragmentation that forced servers to over-reserve VRAM, so many more sequences fit at once. It is the mechanism behind vLLM's throughput advantage.

Speculative Decoding#

Also called: speculative sampling, assisted generation

A speed technique where a small draft model proposes several tokens ahead and the full model verifies them all in one parallel pass. Accepted guesses cost roughly what the small model cost, and output is mathematically identical to normal decoding. Typical local speedups are 1.5–3x, best on predictable text like code.

Draft Model#

The small, fast model that proposes candidate tokens in speculative decoding. It must share the target model's tokenizer, and works best when it is from the same family — a 0.5B drafting for a 32B, for example. Both models occupy VRAM simultaneously, so the draft has to be small enough to be worth its footprint.

Batch Size#

How many sequences the GPU processes together in one forward pass. Because decode is memory-bound, batching is nearly free at first: doubling from one to two concurrent users barely reduces per-user speed while doubling total throughput. Gains flatten once the workload becomes compute-bound or the KV cache fills VRAM.

Throughput vs Latency#

The central serving trade-off: throughput is total tokens per second across all users, latency is the wait experienced by one. Larger batches raise throughput and worsen per-user latency. A single-user desktop should optimise latency; a shared team server should optimise throughput, and they call for different engines and settings.

Concurrency#

The number of requests in flight at once, and the main input to sizing a shared local deployment. VRAM sets the ceiling, because each active sequence holds its own KV cache. Estimating peak concurrency, multiplying by per-sequence KV cost, and adding weights is how you decide whether one GPU is enough.

mmap#

Also called: memory mapping, use_mmap

Memory-mapping a model file so the OS pages weights in on demand rather than reading the whole file into RAM first. It makes a second launch of the same model nearly instant, since the pages are still in the file cache. Disable it when running from slow network storage, where page faults are worse than one upfront read.

Model Load Time#

Also called: cold load, warm-up

How long it takes to move weights from disk into VRAM before the first token can be produced — seconds on NVMe, minutes over a slow network share. Runtimes that unload idle models trade VRAM for this cost on every request. Keeping a model resident is the difference between a snappy assistant and a 30-second pause.

KV Cache Offloading#

Moving part of the KV cache to system RAM when it will not fit in VRAM. It lets you run a longer context than the GPU can hold, but every offloaded block crosses PCIe on each attention step, so the slowdown is severe. Quantizing the KV cache is almost always the better first move.

Flash Attention#

An optimized implementation of the attention algorithm that is significantly faster and more memory-efficient. It processes attention in tiles to minimize HBM (GPU memory) reads/writes. Flash Attention 2 & 3 are standard in most modern local inference frameworks. Enables longer context windows.

Transformer#

The neural network architecture behind essentially every modern LLM, introduced in 2017. It stacks identical blocks, each combining self-attention with a feed-forward network, so every token can look at every other token in the sequence. Its parallelism during training is what made scaling to billions of parameters practical.

Attention (Self-Attention)#

The core mechanism in Transformers. For each token in the input, attention computes how much to 'attend to' (focus on) every other token. This is what allows the model to understand relationships between words regardless of distance. Multi-head attention runs several attention operations in parallel.

Multi-Head Attention (MHA)#

Also called: MHA

The original attention design, where each head keeps its own query, key, and value projections. It is expressive but expensive at inference, because the KV cache stores keys and values for every head at every layer. Nearly all recent models replace it with GQA or MLA to shrink that cache.

Grouped-Query Attention (GQA)#

Also called: GQA

An attention variant where several query heads share one set of key/value heads, cutting KV cache size by the sharing ratio with almost no quality loss. It is why a modern 8B model holds a 128k context in memory that older designs would need many times more VRAM for. Standard across the Llama 3, Qwen, and Mistral families.

Multi-Query Attention (MQA)#

Also called: MQA

The most aggressive form of key/value sharing: all query heads share a single KV head. It gives the smallest possible KV cache, at a measurable cost in quality. GQA is the compromise that largely replaced it, keeping most of the memory saving without the accuracy hit.

Multi-Head Latent Attention (MLA)#

Also called: MLA

DeepSeek's attention design, which compresses keys and values into a low-rank latent vector and reconstructs them on the fly. The KV cache shrinks dramatically compared with GQA while quality holds or improves. It is a large part of why DeepSeek models serve long contexts at unusually low memory cost.

Sliding Window Attention (SWA)#

Also called: SWA, local attention

Restricting most layers to attend only to a fixed window of recent tokens — commonly 4k — instead of the entire context. KV cache then grows with the window rather than the sequence, making very long inputs affordable. Models usually interleave a few full-attention layers so global information can still propagate.

MoE (Mixture of Experts)#

An architecture where the model is split into many 'expert' sub-networks. Each token is routed to only 2–8 experts, so total params are huge but active params (and VRAM) are much smaller. DeepSeek R1 has 671B total params but only 37B active at once.

Active Parameters#

Also called: active params

The subset of an MoE model's parameters actually used for a given token. It sets generation speed, while total parameters set the memory you must have available. A 100B model with 3B active generates roughly as fast as a 3B dense model — but you still need to hold all 100B somewhere.

Expert Router#

Also called: gating network, MoE router

The small learned network inside each MoE layer that decides which experts handle each token. It is trained with a load-balancing objective so traffic spreads across experts instead of collapsing onto a few. Routing is per token, not per request, so a single sentence can touch a large share of the model.

Dense Model#

A model where every parameter participates in every token — the conventional design, as opposed to MoE. Dense models are simpler to quantize, serve, and fine-tune, and at small sizes they are usually stronger per gigabyte. The Llama, Qwen dense line, Gemma, and Mistral small models are all dense.

RoPE#

Also called: rotary embeddings, rope_theta

Rotary Position Embedding — the standard way transformers encode token position, by rotating query and key vectors at frequencies that depend on position. Because it encodes relative distance, it extrapolates better than older absolute schemes, and its base frequency (`rope_theta`) is the parameter that context-extension methods adjust.

YaRN#

Also called: RoPE scaling, context extension

A RoPE-scaling method that extends a model's usable context far beyond its training length by interpolating position frequencies unevenly, with a short fine-tune. It is how models trained at 4k–32k ship with 128k windows. Applying it at load time without the matching fine-tune degrades quality at short contexts as well as long.

Layer#

Also called: transformer block, decoder layer

One transformer block — attention plus a feed-forward network — stacked dozens of times to form the model. Layer count and hidden size together determine parameter count. Layers are also the unit of GPU offload: `n_gpu_layers` decides how many blocks live in VRAM and how many stay on the CPU.

Hidden Size#

Also called: d_model, embedding dimension

The width of the vectors flowing through the model, listed as `hidden_size` in config.json. It drives both parameter count and per-token KV cache cost, alongside layer count and head configuration. Together with layers and vocabulary size it is enough to compute a model's memory footprint from first principles.

Vocabulary Size#

The number of distinct tokens a model's tokenizer can emit — around 128k for Llama 3, over 150k for several recent multilingual models. Bigger vocabularies encode text in fewer tokens, especially for non-English languages and code, which means more content fits in the same context window.

Logits#

The raw, unnormalised scores the model produces for every token in the vocabulary at each step, before softmax turns them into probabilities. Every sampling parameter — temperature, top-p, penalties, grammar constraints — operates on logits. Exposing them is what makes constrained decoding and classification with an LLM possible.

State Space Model (Mamba)#

Also called: SSM, Mamba

A non-attention sequence architecture that keeps a fixed-size recurrent state instead of a growing KV cache, so memory stays constant as context grows. Pure state space models trail transformers on recall-heavy tasks, which is why most shipping designs are hybrids that interleave Mamba blocks with attention layers.

Hybrid Architecture#

A model that mixes attention layers with cheaper alternatives — state space blocks, sliding-window attention, or linear attention — to keep long-context memory bounded without losing recall. Nemotron and several 2026 open-weight releases use this pattern, and it is the main reason large context windows became affordable to serve.

Temperature#

A sampling parameter (0.0–2.0) that controls response randomness. Temperature=0 is deterministic (always picks the most probable token). Temperature=1 is balanced. Temperature>1.5 produces creative/chaotic output. For coding tasks, use 0.1–0.3; for creative writing, try 0.7–1.2.

Top-p (Nucleus Sampling)#

Also called: top_p, nucleus sampling

A sampler that considers only the most probable tokens whose cumulative probability reaches p — 0.9 or 0.95 typically — and discards the long tail. Unlike top-k it adapts to the distribution, staying narrow when the model is confident and widening when it is not. Most defaults pair it with temperature.

Top-k#

Also called: top_k

A sampler that keeps only the k highest-probability tokens at each step, then samples among them. It is simple but blunt: a fixed k is too permissive when the model is certain and too restrictive when it is not. Modern configurations prefer top-p or min-p, often disabling top-k entirely by setting it to 0.

Min-p#

Also called: min_p

A sampler that keeps tokens whose probability is at least a fraction of the top token's — 0.05 means "at least 5% as likely as the best option". It scales with model confidence more gracefully than top-p and holds up at higher temperatures, which is why creative-writing setups favour it.

Repetition Penalty#

Also called: repeat_penalty

A multiplier that lowers the probability of tokens already present in the context, used to stop a model looping. Values just above 1.0 — 1.05 to 1.15 — help; higher settings degrade output by penalising ordinary words like "the" and breaking code, where repeated syntax is correct.

Frequency & Presence Penalty#

Also called: frequency_penalty, presence_penalty

Two additive OpenAI-style penalties. Frequency penalty scales with how often a token has already appeared, discouraging overuse; presence penalty applies a flat cost the moment a token appears at all, pushing toward new topics. Both are gentler than a multiplicative repetition penalty.

Greedy Decoding#

Always taking the single highest-probability token — equivalent to temperature 0. It gives the most reproducible output and is the right default for extraction, classification, and structured formats. It also makes repetition loops more likely on weaker models, since there is no randomness to break a cycle.

Beam Search#

Keeping several candidate continuations alive at once and returning the highest-scoring complete sequence. It helps in translation and other constrained tasks but produces bland, repetitive open-ended text, and costs memory and time proportional to beam width. Most local chat runtimes leave it off.

Seed#

The random-number seed that makes sampling reproducible: the same seed, prompt, settings, and build produce the same output. It does not survive changes in batch size, GPU, or runtime version, because floating-point reduction order changes with them. Useful for debugging, not a guarantee of determinism across machines.

Stop Sequences#

Also called: stop tokens, stop strings

Strings that halt generation the moment they appear, on top of the model's own end-of-sequence token. They are how you stop a model from writing the next turn of a conversation for you. A model that never stops usually has a chat-template mismatch, not a missing stop string.

Logprobs#

Also called: log probabilities, token probabilities

The log-probabilities the model assigned to the tokens it chose, and optionally to the alternatives it rejected. They are the basis for confidence scoring, cheap classification, and detecting low-certainty answers before showing them to a user. Most OpenAI-compatible local servers expose them.

Streaming#

Also called: SSE, token streaming

Sending tokens to the client as they are generated instead of waiting for the full response, usually over server-sent events. It does not change total generation time but transforms perceived speed, since the user starts reading after time-to-first-token rather than after the last token.

Structured Output#

Also called: JSON mode, JSON schema output

Forcing a model to emit valid JSON, or JSON matching a specific schema, by constraining which tokens may be sampled at each step. Because invalid tokens are masked out, the output parses by construction — far more reliable than asking politely and retrying. Supported by llama.cpp, vLLM, SGLang, and Ollama.

Constrained Decoding#

Also called: guided decoding, grammar-constrained decoding

The general mechanism behind structured output: at each step, a state machine derived from a grammar or schema masks every token that would make the output invalid. It guarantees syntactic correctness at negligible speed cost, and can enforce anything expressible as a grammar — JSON, SQL, a fixed set of labels.

GBNF Grammar#

Also called: GBNF

llama.cpp's grammar format, a BNF variant that describes exactly which strings a model may produce. Writing a grammar lets you pin output to a rigid shape — a date, an enum, a specific record layout — without post-processing. It is the lowest-level and most flexible form of constrained decoding available locally.

Reasoning Effort#

Also called: thinking budget, reasoning tokens

A control on how much internal thinking a reasoning model does before answering, exposed as a level (low/medium/high) or an explicit token budget. Higher effort improves hard maths and code at a direct cost in latency and tokens. Setting it low, or disabling thinking, is the fix when a reasoning model deliberates over trivial questions.

RAG (Retrieval-Augmented Generation)#

A technique to give an LLM access to external documents without fine-tuning. Your documents are split into chunks, embedded into vectors, and stored in a vector database. At query time, the most relevant chunks are retrieved and injected into the prompt as context.

Embeddings#

Numerical vector representations of text. Every word or chunk of text is mapped to a list of floating-point numbers that capture semantic meaning. Similar concepts have similar vectors. Used in RAG, semantic search, and clustering. Models like nomic-embed-text or all-minilm create them.

Vector Database#

Also called: vector store, vector DB

A store that indexes embeddings for fast nearest-neighbour search. Options range from libraries embedded in your process (FAISS, Chroma, LanceDB) to servers (Qdrant, Weaviate, Milvus) and Postgres with pgvector. For collections under roughly a million chunks, an embedded library on the same machine is usually enough.

Chunking#

Also called: text splitting

Splitting documents into retrievable pieces before embedding them. Chunk size is the highest-leverage decision in a RAG system: too small and context is lost, too large and the embedding blurs across topics. Splitting on structure — headings, sections, function boundaries — beats fixed character counts almost every time.

Reranker#

Also called: cross-encoder

A second-stage model that scores retrieved chunks against the query directly, rather than by vector distance, and reorders them. Cross-encoder rerankers like bge-reranker are slower per document but far more accurate. Retrieve 50 candidates, rerank, keep 5 — usually the single biggest quality win in a RAG pipeline.

Hybrid Search#

Also called: BM25 + vector, reciprocal rank fusion

Combining keyword search (BM25) with vector search and fusing the ranked lists. Keyword search nails exact matches — error codes, product SKUs, function names — that embeddings blur together, while vectors catch paraphrases. Hybrid retrieval consistently beats either method alone on real document sets.

Semantic Search#

Searching by meaning rather than exact words, by embedding the query and finding the nearest document vectors. It answers "how do I stop my GPU running out of memory" with a document titled "CUDA OOM errors" even with no shared keywords. It is the retrieval half of RAG, and is useful on its own.

Cosine Similarity#

The standard measure of how close two embeddings are: the cosine of the angle between them, from -1 to 1. It compares direction while ignoring magnitude, which is what you want for text. Most vector databases normalise vectors so that cosine similarity and dot product become equivalent.

HNSW#

Hierarchical Navigable Small World — the graph index most vector databases use for approximate nearest-neighbour search. It gives sub-linear query time with high recall, tuned by the `ef` and `M` parameters that trade memory and build time for accuracy. Approximate, not exact: a small recall loss buys an enormous speed gain.

Embedding Dimension#

Also called: vector dimension

The length of an embedding vector — 384, 768, and 1024 are common, with some models producing 4096. Larger dimensions capture more nuance and cost proportionally more storage and search time. The dimension is fixed per model, so changing embedding models means re-embedding the entire corpus.

Matryoshka Embeddings#

Also called: MRL, Matryoshka Representation Learning

Embeddings trained so that truncating the vector still leaves a usable representation — take the first 256 dimensions of a 1024-dimension vector and lose only a little accuracy. This lets one model serve a cheap coarse index and an expensive precise one, and cuts storage substantially with minimal quality cost.

Top-k Retrieval#

How many chunks a RAG system pulls into the prompt. More is not better: irrelevant chunks dilute attention and push the useful passage into the middle of the context, where recall is weakest. Retrieve broadly, rerank, and pass a small number of high-confidence chunks.

Grounding#

Constraining a model to answer from supplied sources and to say so when they do not contain the answer. It is the main defence against hallucination in document QA, and it is a prompt-and-evaluation discipline as much as a retrieval one — including asking for span-level citations you can verify programmatically.

GraphRAG#

Also called: graph RAG, knowledge graph RAG

A RAG variant that first extracts entities and relationships into a knowledge graph, then retrieves over graph structure as well as text similarity. It answers questions that need connections across many documents — "which projects share this dependency?" — that flat chunk retrieval cannot. The cost is an expensive indexing pass.

AI Agent#

A system where an LLM chooses actions in a loop — calling tools, reading results, and deciding what to do next — rather than producing one response. Agents need reliable tool calling and long context, which is why local agent work favours models specifically trained for it over general chat models of the same size.

Function Calling#

Also called: tool calling, tool use

A model capability where, given descriptions of available functions, the model emits a structured call — name plus JSON arguments — instead of prose. Your code executes it and returns the result. It is the foundation of every agent, and a model's tool-calling reliability matters far more than its benchmark scores for agent work.

Tool Schema#

Also called: function schema, tool definition

The JSON Schema description of a tool — its name, purpose, and parameters — that goes into the prompt so the model knows what it can call. Schemas are pure context overhead: a dozen verbose tools can consume a large share of the window before any work begins, which is why tool selection and terse descriptions matter.

Model Context Protocol (MCP)#

Also called: MCP

An open standard, introduced by Anthropic in late 2024 and now widely adopted, for connecting models to tools and data through a common client/server interface. One MCP server exposing a filesystem, database, or API works with any MCP-capable client, replacing bespoke integrations per application.

ReAct#

Also called: reason and act

The reason-act-observe loop that most agents implement: the model states its reasoning, picks an action, sees the result, and repeats until done. Its practical value is debuggability — the reasoning trace shows exactly where a run went wrong, which is otherwise nearly impossible to diagnose.

Agentic Workflow#

A multi-step process where a model plans, executes, checks its own work, and retries. Compared with a single call it is far more capable and far more expensive: token use per task can be tens of times higher. That cost profile is a strong argument for running agents on local hardware, where tokens are not metered.

Multi-Agent System#

An architecture that splits work across several specialised agents — planner, researcher, coder, reviewer — coordinating through messages or a shared scratchpad. It helps when subtasks genuinely differ, and hurts when it just adds handoffs: every boundary loses context and multiplies token cost.

Coding Agent#

An agent that reads a repository, edits files, and runs tests in a loop. Local coding agents need strong tool calling, long context for large files, and low latency, since a single task can involve dozens of calls. Model choice here is driven by SWE-bench-style results and tool reliability, not general chat quality.

Chain of Thought#

Also called: CoT, step-by-step reasoning

Having a model work through intermediate steps before answering, either because it was prompted to or because it was trained to. It measurably improves multi-step arithmetic, logic, and code, and costs output tokens and latency. Reasoning models internalise it; ordinary models can be prompted into it.

Test-Time Compute#

Also called: inference-time scaling, test-time scaling

Spending more computation at inference — longer reasoning, multiple sampled attempts, self-verification — to get better answers from the same weights. It is the scaling axis that largely replaced "just train a bigger model", and it is why a mid-size local reasoning model can beat a much larger one given enough thinking budget.

Context Engineering#

The discipline of deciding what goes into the context window at each step of an agent run — which tools to expose, which history to keep, what to summarise, what to drop. As agents got longer-running, it displaced prompt wording as the main determinant of whether a system works reliably.

Pretraining#

The first and by far most expensive training phase: next-token prediction over trillions of tokens of general text, producing a base model. It costs millions of dollars and thousands of GPUs, which is why almost nobody pretrains from scratch — the open-weight ecosystem exists so you can start from someone else's pretraining run.

Fine-Tuning#

Continuing to train an existing model on your own data so it adopts a format, tone, or domain vocabulary. It is the right tool for behaviour and style, and the wrong tool for facts — for knowledge that changes, retrieval beats fine-tuning on both accuracy and update cost. With LoRA it fits on a single consumer GPU.

SFT (Supervised Fine-Tuning)#

Training a pretrained base model on labeled input-output pairs to make it follow instructions. This is the first step in creating an 'instruct' model from a base model. Examples: training on instruction-response pairs from Alpaca, ShareGPT, or your own domain-specific data.

LoRA / QLoRA#

Parameter-Efficient Fine-Tuning methods. LoRA adds small trainable rank-decomposition matrices to attention layers, updating only ~1% of weights instead of all of them. QLoRA = LoRA on a 4-bit quantized model, enabling fine-tuning of 7B models on a single 6GB GPU.

Adapter Weights#

Also called: LoRA adapter, PEFT adapter

The small file a LoRA fine-tune produces — typically tens to hundreds of megabytes rather than gigabytes — holding only the trained low-rank matrices. Adapters load on top of the unchanged base model, so you can ship several task-specific adapters against one set of weights, or merge one in permanently.

LoRA Rank & Alpha#

Also called: r, lora_alpha

The two settings that govern a LoRA fine-tune. Rank (r) sets the capacity of the injected matrices — 8 to 16 for style, 32 to 64 for real new capability. Alpha scales the adapter's contribution, conventionally set to r or 2r. Higher rank means more trainable parameters, more VRAM, and more risk of overfitting.

RLHF / DPO#

Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are alignment techniques. Both use human preference data (chosen vs rejected responses) to make models more helpful and less harmful. DPO is simpler — no reward model needed.

GRPO#

Group Relative Policy Optimization — the reinforcement-learning method DeepSeek used to train R1's reasoning. It samples a group of answers per prompt and scores each relative to the group average, removing the need for a separate value network. With a verifiable reward such as unit tests, it trains reasoning without human preference labels.

Continued Pretraining#

Also called: domain-adaptive pretraining, CPT

Running more next-token training on a base model with a large domain corpus — medical, legal, or a low-resource language — before instruction tuning. It is the method for teaching genuinely new domain knowledge, unlike SFT, and it needs billions of tokens rather than the thousands of examples SFT uses.

Catastrophic Forgetting#

The loss of previously learned capability when a model is fine-tuned narrowly — a model trained hard on one output format often gets worse at everything else. Mitigations are lower learning rates, fewer epochs, LoRA instead of full fine-tuning, and mixing general instruction data into the training set.

Model Merging#

Also called: mergekit, SLERP merge

Combining two or more fine-tunes of the same base model by arithmetic on their weights, using methods like SLERP, TIES, or DARE. It requires no training compute and can blend a coding fine-tune with a chat fine-tune in minutes. Results are unpredictable, so merges must be evaluated, not assumed.

Gradient Checkpointing#

Also called: activation checkpointing

A memory-for-compute trade during training: instead of storing every intermediate activation for the backward pass, recompute them from saved checkpoints. It typically cuts training memory substantially at roughly 20–30% extra compute time, and it is routinely what makes a fine-tune fit on a consumer GPU.

Learning Rate#

Also called: LR

How large a step each optimizer update takes — the hyperparameter most likely to ruin a fine-tune. Too high and the model degrades into repetition; too low and nothing changes. Full fine-tuning uses roughly 1e-5 to 2e-5; LoRA tolerates 1e-4 to 2e-4 because it updates far fewer parameters.

Epoch#

One complete pass over the training dataset. Instruction fine-tunes usually need only one to three: past that, models memorise examples verbatim and lose generality. Watch validation loss rather than counting epochs, and stop when it stops falling.

Instruction Dataset#

A collection of prompt/response pairs used for supervised fine-tuning, in a format such as Alpaca or ShareGPT. Quality dominates quantity — a thousand carefully curated examples routinely beat a hundred thousand scraped ones. Every example must use the target model's chat template.

Synthetic Data#

Training examples generated by another model instead of collected from humans. It is how most modern instruction datasets are built, and it is cheap and scalable. The risks are inherited errors and narrowed diversity, so production pipelines filter, deduplicate, and verify synthetic examples before training on them.

Unsloth#

A fine-tuning library with hand-written kernels that roughly doubles LoRA and QLoRA training speed while cutting memory use, on a single GPU. It is the usual answer to "how do I fine-tune an 8B model on a 16GB card", and it also publishes widely used quantized and QAT model conversions.

Axolotl#

A configuration-driven fine-tuning framework: you describe the model, dataset, and method in YAML and it handles the training loop. It supports full fine-tuning, LoRA, QLoRA, and preference methods across many architectures, and its reproducible configs are why community fine-tunes are often shared as an Axolotl file.

Perplexity#

A measure of how well a language model predicts a text sample. Lower perplexity = the model finds the text more predictable = better quality. Used to compare models and quantization levels. A heavily quantized model will have higher perplexity (more uncertainty) than the FP16 original.

Benchmark#

A standardised test set used to compare models on a task. Benchmarks are useful for coarse ranking and unreliable for predicting performance on your specific workload, because published scores depend on prompt format, sampling settings, and possible contamination. Treat them as a shortlist filter, then evaluate on your own data.

MMLU & MMLU-Pro#

Also called: MMLU

Multiple-choice knowledge benchmarks across dozens of academic subjects. MMLU is the long-standing standard and is now saturated and heavily contaminated at the top of the leaderboard. MMLU-Pro adds harder questions and ten answer options instead of four, restoring separation between strong models.

GPQA#

Graduate-level Google-Proof Q&A — physics, chemistry, and biology questions written so that a non-expert with web access still fails them. Its Diamond subset is the common reporting split. It resists contamination better than MMLU and is a good discriminator among reasoning models.

HumanEval#

A 164-problem Python benchmark where a model writes a function from a docstring and is scored by hidden unit tests. It is the original code benchmark, now largely saturated and present in most training corpora. Contamination-resistant successors like LiveCodeBench draw problems from contests postdating a model's cutoff.

SWE-bench#

A benchmark built from real GitHub issues, where the model must produce a patch that makes the repository's tests pass. Because it requires navigating a codebase and using tools, it correlates with practical coding-agent usefulness far better than function-writing benchmarks. SWE-bench Verified is the human-validated subset most results cite.

AIME#

The American Invitational Mathematics Examination, adopted as a standard reasoning benchmark because each answer is an integer, making scoring unambiguous. New AIME papers each year provide fresh, uncontaminated problems, which is why AIME scores became the headline metric for reasoning models.

IFEval#

A benchmark that tests literal instruction following with programmatically checkable constraints — "answer in exactly three bullet points", "do not use the letter e". It measures compliance rather than knowledge, which makes it unusually predictive of whether a model will behave inside an automated pipeline.

LMArena Elo#

Also called: Chatbot Arena, Arena Elo

A ranking derived from blind human pairwise comparisons of model responses, scored with an Elo system. It captures preference and style in a way static benchmarks cannot, and it also rewards them — chattier, better-formatted answers score well independently of correctness. Read it alongside task benchmarks, not instead of them.

Pass@k#

Also called: pass@1

The probability that at least one of k sampled attempts is correct — the standard metric for code benchmarks. Pass@1 measures single-shot reliability; pass@10 measures whether the model can find the answer given retries. A large gap between them means sampling more and verifying is a cheap accuracy win.

LLM-as-a-Judge#

Using a strong model to grade another model's outputs against a rubric, in place of human review. It scales evaluation enormously and carries known biases — toward longer answers, toward its own family's style, and toward whichever response appears first. Randomise order and calibrate against human labels on a sample.

Benchmark Contamination#

Also called: data contamination, test set leakage

When benchmark questions appear in a model's training data, so its score reflects memorisation rather than capability. It is widespread on older public benchmarks. The defences are private held-out sets, problems published after the training cutoff, and being suspicious of a model that tops a leaderboard but disappoints in use.

Needle in a Haystack#

Also called: NIAH

A long-context test that hides a specific fact in a long document and asks the model to retrieve it, varying both depth and total length. Passing it proves retrieval, not comprehension — multi-fact and reasoning-over-context variants are much harder, and are where advertised context windows usually break down.

KL Divergence#

Also called: KLD

A measure of how far one probability distribution is from another, used to judge quantization damage by comparing a quantized model's token distribution against the original's. It is more sensitive than perplexity, which can look almost unchanged while specific behaviours have measurably shifted.

Eval Harness#

Also called: lm-eval, lm-evaluation-harness

A framework that runs standardised benchmark suites against a model with consistent prompts and scoring — lm-evaluation-harness is the common one. Using a harness is what makes numbers comparable, since prompt formatting and sampling settings alone can move a score by several points.

Self-Hosting#

Running models on infrastructure you own or rent exclusively, rather than calling a shared API. It gives you data control, fixed costs, no rate limits, and version stability — a hosted model cannot be deprecated out from under you. The costs are hardware, electricity, and the operational work of keeping it running.

Inference Server#

A long-running process that holds a model in VRAM and answers requests over HTTP. Keeping weights resident is the point: it removes load time from every request. Production servers add batching, queuing, prefix caching, metrics, and health checks that a bare CLI loop does not have.

Tensor Parallelism#

Also called: TP

Splitting each layer's weight matrices across multiple GPUs so they compute one token together. It is how a model larger than any single card runs, and how you cut latency across cards. Every layer requires an all-reduce between GPUs, so interconnect speed — NVLink versus PCIe — sets the ceiling.

Pipeline Parallelism#

Also called: PP, layer split

Assigning whole layers to different GPUs so a token passes through card one, then card two. Communication is far lighter than tensor parallelism, making it viable over ordinary PCIe or even across machines. It does not reduce single-request latency — it only lets a bigger model fit.

Multi-GPU Inference#

Running one model across several GPUs, by splitting layers (pipeline) or splitting each layer (tensor). Two 24GB cards can hold a model a single 24GB card cannot, but they do not double speed — expect meaningfully less than 2x, and less still without a fast interconnect.

Cost per Million Tokens#

Also called: $/M tokens, token pricing

The standard unit for comparing inference costs, quoted separately for input and output because output is several times more expensive to produce. Comparing it against local inference means converting your hardware amortisation and electricity into the same unit at your actual monthly token volume.

GPU-Hour#

The rental unit for cloud GPUs — one hour of one accelerator, from well under a dollar for a consumer-class card to several dollars for an H100-class one. It is the right comparison basis when you are serving your own model rather than paying per token, and the basis for estimating a hardware purchase's payback.

Total Cost of Ownership (TCO)#

Also called: TCO

The full cost of running models locally: hardware amortised over its useful life, electricity at your rate, cooling, and the staff time to operate it. TCO is the only honest basis for a local-versus-cloud comparison — the hardware price alone understates it, and ignoring it makes local look free after purchase.

Break-Even Point#

The monthly token volume at which local inference becomes cheaper than a cloud API, once hardware and electricity are counted. It falls as usage rises: light use never repays a GPU, while sustained high-volume or agentic workloads — which burn tokens by the million — often pay one back within months.

On-Premise#

Also called: on-prem

Running AI infrastructure in your own data centre or office rather than in a provider's cloud. Organisations choose it for data residency, regulatory obligations, and predictable cost at scale. It is distinct from air-gapped: on-prem hardware usually still has network access.

Air-Gapped Deployment#

Also called: air gap, offline deployment

Running models on hardware with no network connection to the outside world — the standard for classified, defence, and some clinical environments. Everything must be staged in physically: weights, runtime, dependencies, and updates. Open-weight models are the only practical option, since no API call can leave the enclave.

Sovereign AI#

AI capability a country or organisation controls end to end — data, model weights, compute, and operations — without dependence on a foreign provider. It is the driver behind national model programmes and much European enterprise interest in open weights, since only downloadable weights can be operated entirely under local jurisdiction.

Cold Start#

The delay when a request arrives and no model is loaded — allocating a GPU, pulling weights, and filling VRAM, which can take from seconds to minutes. It is the central trade-off in serverless GPU hosting: you stop paying for idle time and pay in first-request latency instead.

Serverless GPU#

GPU capacity billed per second of actual use, scaling to zero when idle. It suits spiky or occasional workloads where a dedicated instance would sit unused, and it is a poor fit for latency-sensitive interactive use because of cold starts. Sustained load is cheaper on a reserved instance or your own hardware.

Rate Limit#

Also called: TPM, RPM

A cap a hosted API places on requests or tokens per minute. Rate limits are a recurring reason teams move to local inference: batch jobs and agent loops hit them constantly, and self-hosted models are limited only by your own hardware. The local equivalent is a queue, which delays rather than rejects.

Apache 2.0#

A permissive open-source licence with an explicit patent grant, and the most business-friendly licence common for model weights. It allows commercial use, modification, and redistribution with attribution, and imposes no user-count threshold or field-of-use restriction. Qwen and gpt-oss weights ship under it.

MIT License#

The shortest and most permissive common licence: do anything, keep the copyright notice, no warranty. Unlike Apache 2.0 it contains no explicit patent grant, which some legal teams care about. DeepSeek has released recent model weights under it.

Llama Community License#

Also called: Llama license

Meta's custom licence for Llama weights. It permits commercial use but adds conditions absent from Apache 2.0: attribution requirements, naming rules for derivative models, an acceptable-use policy, and a special-permission threshold for very large deployments. Read it directly rather than assuming it behaves like open source.

Non-Commercial License#

Also called: CC-BY-NC, research-only license

A licence permitting research and personal use only, most often CC-BY-NC. Models under it cannot be used in a product, even internally at a company, without separate permission. It is common for research releases and for some fine-tunes whose training data carries the restriction — always check derivatives, not just the base model.

OpenRAIL#

Also called: RAIL license

A family of Responsible AI licences that allow commercial use while prohibiting specific applications — surveillance, discrimination, generating disinformation. The restrictions bind downstream users too, so anyone shipping an OpenRAIL model must pass the same use limits on to their own customers.

EU AI Act#

The European Union's risk-tiered AI regulation, with obligations phased in from 2025 onward. It imposes transparency and documentation duties on general-purpose model providers and stricter requirements on high-risk deployments in areas like hiring, credit, and medical devices. Self-hosting does not remove the obligations of a deployer.

Data Residency#

A requirement that data be stored and processed within a specific jurisdiction. It is one of the strongest arguments for local models: a prompt processed on hardware in your own building never crosses a border, which is far simpler to demonstrate to an auditor than a cloud provider's regional-processing commitments.

Zero Data Retention#

Also called: ZDR, no-retention

A contractual commitment that a provider does not store prompts or outputs after serving a request. It is the strongest privacy guarantee a hosted API can offer, and it is still a promise about someone else's systems. Local inference makes the question moot, because the data never leaves the machine.

Prompt Injection#

An attack where instructions hidden in content the model reads — a web page, a document, a tool result — are followed as if they came from the user. It is the defining security problem for agents, because a model cannot reliably distinguish data from instructions. Mitigate with least-privilege tools, output validation, and human approval for consequential actions.

Jailbreak#

A prompt crafted to bypass a model's safety training and elicit content it was trained to refuse. Distinct from prompt injection, which targets the application around the model rather than its alignment. Open-weight models are inherently more exposed, since anyone can also modify the weights directly.

Guardrails#

Checks placed around a model rather than inside it — input filters, output classifiers, schema validation, and tool permission boundaries. Because alignment training can be bypassed, guardrails are the layer that actually enforces policy in production. Llama Guard and similar small classifiers are commonly used locally for this.

Alignment#

Training a model to behave according to intended values — helpful, honest, and refusing genuinely harmful requests — usually via preference optimization after instruction tuning. Alignment is a property of the weights, so it can be fine-tuned away by anyone with the open weights, which is why deployed systems need guardrails as well.

Red Teaming#

Deliberately adversarial testing of a model or the system around it, looking for jailbreaks, injection paths, data leaks, and harmful outputs before users find them. For agents the target is the whole loop — tools, permissions, and data flow — not the model in isolation.

Abliterated Model#

Also called: uncensored model, abliteration

A model modified after training to remove its refusal behaviour, typically by ablating the direction in activation space associated with refusing. Variants labelled "abliterated" or "uncensored" appear widely in community repositories. They also lose some instruction-following quality, and shipping one to end users transfers all safety responsibility to you.

PII Redaction#

Also called: data masking

Stripping or masking personal data before it reaches a model. It is a standard control for cloud APIs and often relaxed for local inference, where the data never leaves the trusted boundary — one of the concrete operational savings of running models yourself, alongside the compliance review it removes.