GPU & VRAM checker

It depends on the model's size. At Q4_K_M with an 8K context, a small model such as Llama 3.2 3B needs about 3.7 GB, an 8B such as Llama 3.1 8B about 6.7 GB, a 14B about 11.1 GB and a 32B about 22.8 GB: compare those with your card's VRAM. A 70B model (Llama 3.3 70B) needs about 45.7 GB, more than any single consumer GPU holds (the RTX 5090 has 32 GB), so it takes several GPUs or a Mac with enough unified memory. On Apple Silicon the GPU can use about 75% of unified memory by default, so a Mac with 8 GB runs the small models and one with 64 GB fits the 70B with little to spare (48 GB is not enough). Leave system RAM free for the OS and for any layers offloaded from the GPU. The checker above matches your exact GPU and RAM against 180 published model variants so you don't have to guess.

At Q4_K_M with an 8K context, a 7B model (Qwen 2.5 7B) needs about 5.9 GB, a 13–14B model (Qwen3 14B) about 11.1 GB, and a 70B model (Llama 3.3 70B) about 45.7 GB. Each total covers the weights, the KV cache and about 0.8 GB of runtime overhead. Q8_0 stores 1.8× as many bits per weight as Q4_K_M and FP16 3.3×, so the same 70B needs about 77.9 GB at Q8_0 and 143.5 GB at FP16, which means multiple GPUs. Longer context grows the KV cache: for Llama 3.1 8B it is 1.1 GB at 8K. On Apple Silicon the model draws from unified memory instead of dedicated VRAM, so a 64 GB Mac can run the 70B at Q4_K_M.

Full-precision models keep every weight in 16-bit floating point (FP16/BF16): the format they were trained in, and the highest quality. Quantized models compress those weights to 8-bit, 4-bit or lower, cutting memory use by 1.9× at Q8_0 and 3.3× at Q4_K_M for a small and usually unnoticeable drop in quality. The weights of Qwen 2.5 7B, for example, shrink from about 15.2 GB at FP16 to about 4.6 GB at Q4_K_M. Unless you are fine-tuning or running research evaluations, a 4-bit or 5-bit quant is the right choice for local inference.

Temperature controls randomness: lower values make output focused and repeatable, higher values make it more varied and creative. Use 0.0–0.3 for factual Q&A, coding, and data extraction; 0.6–0.8 for general chat and assistants; and 0.9–1.2 for brainstorming or creative writing. Most tools default to around 0.7–0.8, which is a sensible starting point. If you want more diversity, it is usually better to adjust top_p (nucleus sampling) than to push temperature very high.

The software is free and open source: Ollama, LM Studio, llama.cpp and the open-weight models themselves cost nothing to download or run. Your only real costs are the hardware (a GPU or a capable Mac) and the electricity to power it. Unlike cloud APIs there are no per-token fees, no rate limits, and your data never leaves your machine. With steady daily use a local setup can pay for itself; the cost calculator on this site estimates your break-even point from your own usage and electricity price.

llama.cpp is the underlying inference engine most local tools are built on — pick it if you want maximum control, scripting, and the newest features from the command line. Ollama wraps llama.cpp in a simple CLI plus a background server with an OpenAI-compatible API, making it the easiest option for developers who want to call models from code. LM Studio is a polished desktop GUI with a model browser and chat interface, ideal for beginners or anyone who prefers clicking to typing. All three are free, run the same GGUF model files, and can happily coexist on one machine.

Yes. Ollama and llama.cpp both support CPU-only inference, and it works for small models. It is much slower than a GPU, because generation speed is set by memory bandwidth and system RAM is far slower than VRAM, so it suits short prompts better than long generations. The whole model loads into system RAM, so size your RAM by the model: about 3.7 GB for Llama 3.2 3B and 6.7 GB for Llama 3.1 8B at Q4_K_M with an 8K context, plus room for the OS. For the best CPU-only experience, stick to small models at Q4 quantization.

Q4_K_M is the best all-round default: it stores 3.3× fewer bits per weight than FP16, with quality loss that is hard to notice in day-to-day use. Step up to Q5_K_M or Q6_K if you have spare VRAM and want a little more accuracy, or Q8_0 for near-lossless output at 1.8× the Q4_K_M size. Reserve full FP16/BF16 for fine-tuning, evaluation, or research where every bit of precision counts. Below Q4 (Q3, Q2) the file keeps shrinking but the model starts making noticeably more mistakes, so only go there to squeeze a model that otherwise will not fit.