Written by Jakub Rusinowski · Last updated July 10, 2026
To run a fine-tuned model in Ollama or llama.cpp, you merge the LoRA adapter into its base model, convert the result to GGUF with llama.cpp's conversion script, quantize it, and wrap it in a Modelfile
To run a fine-tuned model in Ollama or llama.cpp, you merge the LoRA adapter into its base model, convert the result to GGUF with llama.cpp's conversion script, quantize it, and wrap it in a Modelfile. The whole pipeline is four commands, and this guide walks each one — including the chat-template step everyone gets wrong the first time.
Last Updated: July 2026
LoRA adapter + base model → merge → HF folder → convert_hf_to_gguf.py
→ model-f16.gguf → llama-quantize → model-Q4_K_M.gguf → Modelfile → ollama createYou finished fine-tuning your first LLM and have a folder of adapter weights. Training frameworks save you a *LoRA adapter* — a few hundred MB of weight deltas, not a runnable model. Ollama wants a single GGUF file. Bridging that gap is what this guide does.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt # for the conversion scripts
cmake -B build && cmake --build build --config Release -t llama-quantizeYou need disk headroom: an 8B pipeline briefly holds the merged FP16 model (~16GB) plus the F16 GGUF (~16GB) plus the quant (~5GB). Clean up intermediates as you go if space is tight.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
base = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B-Instruct", # must be the EXACT base you trained on
torch_dtype=torch.float16, device_map="cpu")
model = PeftModel.from_pretrained(base, "./my-lora-adapter")
merged = model.merge_and_unload() # bakes the deltas into the weights
merged.save_pretrained("./merged-model", safe_serialization=True)
AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct").save_pretrained("./merged-model")Two gotchas: the base model string must match what you fine-tuned on *exactly* (an adapter trained on the Instruct variant merged into the base variant produces a subtly broken model), and don't forget the tokenizer save — conversion fails without it. device_map="cpu" keeps this runnable on machines without spare VRAM; merging needs system RAM, not GPU.
(Shortcut: if you trained with Unsloth, model.save_pretrained_gguf(...) does Steps 1–3 in one call — this guide is the standard path for everyone else, and for when you need control over each stage.)
python convert_hf_to_gguf.py ./merged-model \
--outfile my-model-f16.gguf --outtype f16Convert at F16 and quantize as a separate step — quantizing straight from the converter locks you into redoing the conversion for every quant level you want to try.
If the script errors with Model architecture not supported, your base model is newer than your llama.cpp checkout — git pull and retry; support for new architectures lands there within days of major releases.
./build/bin/llama-quantize my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_MQ4_K_M is the right default (best quality-per-GB; ~4.9GB for an 8B). Go Q5_K_M or Q8_0 if VRAM is plentiful, Q3_K_M only under real pressure — the quant selection guide has the full decision table. It's worth producing two quant levels and A/B-testing on *your* task: fine-tuned behaviors (rigid output formats especially) are sometimes more quant-sensitive than general chat.
FROM ./my-model-Q4_K_M.gguf
# Llama 3.1 chat template - MUST match what you trained with
TEMPLATE """<|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|><|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""
PARAMETER stop <|eot_id|>
SYSTEM You are a helpful assistant.ollama create my-finetune -f Modelfile
ollama run my-finetune "A test prompt from your training distribution"The critical line is TEMPLATE. Your fine-tune learned to expect the exact prompt format it was trained with (Llama 3.1 headers above; Qwen/ChatML-family tunes use <|im_start|> markers instead — copy a correct template from a same-family model with ollama show llama3.1 --template). A mismatched template is the number-one reason a freshly converted model outputs nonsense, ignores its training, or never stops generating — dramatic-looking failures, one wrong string. See gibberish output: wrong prompt template if your first test looks broken.
PARAMETER stop doesn't match the tokens the fine-tune actually emits.ollama push (registry account required), or just distribute the GGUF — recipients only need your Modelfile alongside it.llama.cpp supports runtime GGUF-converted adapters (convert_lora_to_gguf.py + --lora), but the merged path is what Ollama expects and what you should ship: one artifact, no adapter/base version skew, marginally faster inference.
Steps 1–4, yes — but it's mechanical; script it once. The Modelfile only changes when your template or params change, so retraining iterations reduce to merge → convert → quantize → ollama create.
In order of likelihood: wrong chat template in the Modelfile (fixable in minutes), merged into a different base variant than you trained on, or quantization on a quant-sensitive task — test the F16 GGUF to isolate which one it is.
Peak ~37GB: merged FP16 (~16GB) + F16 GGUF (~16GB) + Q4 quant (~5GB). Delete the intermediates after validating and you keep only the ~5GB quant. A 70B pipeline peaks around 300GB — plan for it, or run the conversion on a rented box (cloud GPU guide).