LLM Glossary

Plain-English definitions of the 20 terms you'll meet running local AI — from quantization and VRAM to Mixture-of-Experts and RAG. New to local LLMs? Start with the GPU & VRAM checker or browse the model library.

Quantization

Reducing the precision of a model's weights (e.g., from 32-bit float to 4-bit integer) to shrink memory usage with minimal quality loss. A 7B model in FP16 needs ~14GB VRAM; in Q4_K_M it needs ~4GB. (vram, memory, gguf, performance)

GGUF

The file format used by llama.cpp and Ollama for quantized models. Stands for 'GPT-Generated Unified Format'. A single .gguf file contains the full model weights and metadata. Quantization level is encoded in the filename (e.g., Q4_K_M, Q8_0, F16). (format, ollama, llamacpp, file)

Context Window

The maximum number of tokens a model can process at once — its working memory. Llama 3.1 has 128k tokens (~96,000 words). Larger context = more expensive in VRAM. Context is consumed by your prompt, conversation history, and the response. (memory, tokens, context, performance)

Tokens

The basic unit of text for an LLM. Words, sub-words, or characters. Roughly 1,000 tokens ≈ 750 English words. Tokenization varies by model — GPT-4 uses tiktoken, Llama uses SentencePiece. Pricing for cloud APIs is per token. (text, pricing, input, output)

Parameters (Weights)

The numbers learned during training that define what the model knows. A '7B' model has 7 billion parameters. More parameters = more world knowledge and reasoning ability, but requires more VRAM (roughly 2GB per 1B params at full precision, 0.5GB/B at Q4). (model size, vram, quality)

MoE (Mixture of Experts)

An architecture where the model is split into many 'expert' sub-networks. Each token is routed to only 2–8 experts, so total params are huge but active params (and VRAM) are much smaller. DeepSeek R1 has 671B total params but only 37B active at once. (architecture, vram, deepseek, llama 4)

Inference

Running the model to generate text from an input prompt — the 'use' phase, distinct from training. Inference can be done on CPU or GPU. GPU inference is 5–50x faster. Local inference = running the model on your own hardware without sending data to the cloud. (runtime, gpu, cpu, local)

VRAM (Video RAM)

The memory on your GPU. This is the critical bottleneck for local LLM inference — the entire model (weights) must fit in VRAM or the GPU falls back to slow CPU RAM. More VRAM = bigger models. RTX 4090 has 24GB; RTX 4060 has 8GB. (hardware, gpu, memory, bottleneck)

RAG (Retrieval-Augmented Generation)

A technique to give an LLM access to external documents without fine-tuning. Your documents are split into chunks, embedded into vectors, and stored in a vector database. At query time, the most relevant chunks are retrieved and injected into the prompt as context. (architecture, documents, search, embeddings)

LoRA / QLoRA

Parameter-Efficient Fine-Tuning methods. LoRA adds small trainable rank-decomposition matrices to attention layers, updating only ~1% of weights instead of all of them. QLoRA = LoRA on a 4-bit quantized model, enabling fine-tuning of 7B models on a single 6GB GPU. (fine-tuning, training, lora, qlora, adapter)

RLHF / DPO

Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are alignment techniques. Both use human preference data (chosen vs rejected responses) to make models more helpful and less harmful. DPO is simpler — no reward model needed. (alignment, training, preference, safety)

Embeddings

Numerical vector representations of text. Every word or chunk of text is mapped to a list of floating-point numbers that capture semantic meaning. Similar concepts have similar vectors. Used in RAG, semantic search, and clustering. Models like nomic-embed-text or all-minilm create them. (vectors, search, rag, similarity)

KV Cache

Key-Value cache: stores intermediate attention computations so the model doesn't recompute the entire context on every new token. Larger context windows require more KV cache memory. On a 7B model with 8k context, the KV cache can use 1–2GB of VRAM on top of the model weights. (memory, attention, vram, performance)

Perplexity

A measure of how well a language model predicts a text sample. Lower perplexity = the model finds the text more predictable = better quality. Used to compare models and quantization levels. A heavily quantized model will have higher perplexity (more uncertainty) than the FP16 original. (evaluation, benchmark, quality, quantization)

SFT (Supervised Fine-Tuning)

Training a pretrained base model on labeled input-output pairs to make it follow instructions. This is the first step in creating an 'instruct' model from a base model. Examples: training on instruction-response pairs from Alpaca, ShareGPT, or your own domain-specific data. (training, fine-tuning, instruction, alignment)

Attention (Self-Attention)

The core mechanism in Transformers. For each token in the input, attention computes how much to 'attend to' (focus on) every other token. This is what allows the model to understand relationships between words regardless of distance. Multi-head attention runs several attention operations in parallel. (architecture, transformer, mechanism)

Flash Attention

An optimized implementation of the attention algorithm that is significantly faster and more memory-efficient. It processes attention in tiles to minimize HBM (GPU memory) reads/writes. Flash Attention 2 & 3 are standard in most modern local inference frameworks. Enables longer context windows. (performance, memory, attention, optimization)

Tokenizer

The component that converts raw text into token IDs (numbers) before feeding it to the model, and converts model output IDs back to text. Each model family has its own tokenizer — Llama uses SentencePiece, GPT-4 uses tiktoken. Different tokenizers produce different token counts for the same text. (text processing, input, tokens)

Temperature

A sampling parameter (0.0–2.0) that controls response randomness. Temperature=0 is deterministic (always picks the most probable token). Temperature=1 is balanced. Temperature>1.5 produces creative/chaotic output. For coding tasks, use 0.1–0.3; for creative writing, try 0.7–1.2. (sampling, generation, creativity, output)

Ollama

The most popular local LLM runtime. A CLI + REST API that manages model downloads, quantization selection, and inference with a simple interface. Compatible with the OpenAI API format. Run a model with one command: `ollama run llama3.2`. Supports Mac, Windows, and Linux. (runtime, tool, cli, api, setup)