Guide 14: Fine-Tune Your First LLM in 1 Hour

作者： Jakub Rusinowski · 最后更新： 2026年6月15日

By the end of this hour you'll have a small language model that answers in YOUR format, running locally in Ollama — total cost: $0. No PhD required. You need exactly one of: an NVIDIA GPU with 6GB+ VR

In This Guide

• Step 0 — Should you even fine-tune? (5 min)
• Step 1 — Pick your path (2 min)
• Step 2 — Pick a starter dataset (5 min)
• Step 3A — Train on NVIDIA with Unsloth (25 min)
• Step 3B — Train on Mac with MLX (25 min)
• Step 4 — Test and export to Ollama (10 min)
• Step 5 — What went wrong? (5 min)
• What's next

By the end of this hour you'll have a small language model that answers in YOUR format, running locally in Ollama — total cost: $0. No PhD required. You need exactly one of: an NVIDIA GPU with 6GB+ VRAM or an Apple Silicon Mac with 16GB+ RAM. We'll pick a starter dataset, train a QLoRA adapter with conservative settings that won't blow up, export to GGUF, and chat with your own model. This is the recipe card; Guide 13: Fine-Tuning LLMs Locally is the full cookbook — go there whenever you want the theory behind a step.

Step 0 — Should you even fine-tune? (5 min)

Honest gate before you spend an hour: fine-tuning is the *third* tool to reach for, not the first.

Prompting first. A good system prompt plus two or three example outputs solves most "make it answer like X" problems in five minutes. RAG second. If the problem is *knowledge* — your docs, your product, current facts — retrieval wins, because fine-tuning does not reliably add facts. Fine-tune when you need consistent style or format on every response, domain-specific phrasing, structured output that must always parse, or a short prompt for latency reasons. That's the one-paragraph version — the full decision checklist with examples is in Guide 13's "When Should You Fine-Tune?" section.

Still here? Good — style, format, and tone are exactly what a one-hour fine-tune does well.

Step 1 — Pick your path (2 min)

QLoRA is the trick that makes this possible on consumer hardware: the base model is loaded in 4-bit and you train only small adapter matrices, so a 7B fine-tune fits in about 6GB of VRAM — and a 4-bit 7B fine-tune on a Mac uses roughly 8GB of working memory, leaving headroom on a 16GB machine. On Path B, note that training a LoRA adapter on an already-4-bit model *is* QLoRA — MLX handles it with no extra setup.

Not sure your card qualifies? Check your GPU in the hardware analyzer. No compatible hardware at all? Rent an RTX 4090 for ~$0.35/hr — see the Cloud GPU Rental Guide; everything below works identically on a rented box.

If you're shopping for a first fine-tuning card, the 16GB RTX 4060 Ti is the sweet spot — enough VRAM for 13B QLoRA runs:

Step 2 — Pick a starter dataset (5 min)

Three good options, in order:

1. No Robots — 10k examples written entirely by human annotators. Small, clean, diverse. The best first dataset for *learning* the workflow, and it's what the scripts below use. 2. Databricks Dolly 15k — 15k human-written pairs under CC BY-SA 3.0. Pick this one if your fine-tune might ever ship in a product. 3. Your own 200–500 examples — the moment you want the model to sound like *you*, nothing beats your own data. Use the Alpaca JSONL format (same one Guide 13 uses):

{"instruction": "Summarize this support ticket in two sentences.", "input": "Customer reports login fails after password reset...", "output": "Customer cannot log in following a password reset. Issue traced to a stale session cookie; cleared and resolved."}

> ⚠️ License check before you train. Alpaca and ShareGPT are CC BY-NC (non-commercial) — and so is No Robots, so treat it as a learning dataset. For anything commercial, use Dolly 15k (CC BY-SA) or another permissive set — filter the dataset hub by "Permissive" to see all of them.

Step 3A — Train on NVIDIA with Unsloth (25 min)

One script, top to bottom. Install first (a fresh venv keeps CUDA dependency pain away):

pip install unsloth

from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTTrainer
from transformers import TrainingArguments

# 1. Load a small 4-bit base — 3B keeps this comfortable even on 6GB cards
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-3B-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,
)

# 2. Add LoRA adapters (r=16 is the standard starting point)
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    lora_dropout=0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

# 3. Load No Robots and render each conversation with the chat template.
#    Columns on the HF card: prompt, prompt_id, messages, category —
#    "messages" is the list of {role, content} turns we train on.
dataset = load_dataset("HuggingFaceH4/no_robots", split="train[:1000]")

def to_text(row):
    return {"text": tokenizer.apply_chat_template(row["messages"], tokenize=False)}

dataset = dataset.map(to_text)

# 4. Train — conservative defaults, same as Guide 13
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        num_train_epochs=2,
        learning_rate=2e-4,
        logging_steps=10,
        optim="adamw_8bit",
        output_dir="./outputs",
        save_strategy="epoch",
    ),
)
trainer.train()

# 5. Save the adapter
model.save_pretrained("my-first-lora")
tokenizer.save_pretrained("my-first-lora")

Expect roughly 20–60 minutes for the 1,000-example subset (2 epochs) on an RTX 3060/4060-class GPU — the spread depends on example length; older 6GB cards land at the top of the range. Loss should start around 2 and fall steadily; if it doesn't, jump to Step 5.

Step 3B — Train on Mac with MLX (25 min)

pip install mlx-lm

MLX expects a data/ folder with train.jsonl and valid.jsonl in its chat format — each line is {"messages": [...]} and the final message is what the model learns to produce. Convert No Robots in a few lines:

import json
from datasets import load_dataset

ds = load_dataset("HuggingFaceH4/no_robots")
def dump(rows, path):
    with open(path, "w") as f:
        for row in rows:
            f.write(json.dumps({"messages": row["messages"]}) + "\n")

dump(ds["train"].select(range(1000)), "data/train.jsonl")
dump(ds["test"].select(range(100)), "data/valid.jsonl")

Then train — one command:

mlx_lm.lora \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --data ./data \
  --train \
  --batch-size 2 \
  --iters 600

Because the base model is already 4-bit, this LoRA run is QLoRA — no flags needed. Expect 20–40 minutes on an M2/M3/M4 with 16GB; validation loss prints as it goes. Test the adapter immediately:

mlx_lm.generate \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --adapter-path adapters \
  --prompt "Summarize what QLoRA does in one sentence."

Step 4 — Test and export to Ollama (10 min)

Path A (Unsloth): export straight to GGUF — one line added to your script:

model.save_pretrained_gguf("my-first-model", tokenizer, quantization_method="q4_k_m")

Path B (MLX): fuse the adapter into the weights and export. GGUF export needs fp16 weights, so de-quantize while fusing:

mlx_lm.fuse \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --adapter-path adapters \
  --de-quantize \
  --export-gguf

Now wire it into Ollama with a two-line Modelfile (point FROM at the .gguf file you just produced):

FROM ./my-first-model/unsloth.Q4_K_M.gguf
SYSTEM You are a concise assistant that answers in two sentences maximum.

ollama create my-first-model -f Modelfile
ollama run my-first-model

The before/after is the payoff. Same prompt — "Explain what a VPN does":

Before (base model): *"Great question! A VPN, or Virtual Private Network, is a technology that creates a secure, encrypted connection over a less secure network... [three more paragraphs]"*

After (your fine-tune): *"A VPN encrypts your traffic and routes it through a remote server, hiding your activity from local networks and masking your IP address. It protects privacy on public Wi-Fi but doesn't make you anonymous."*

That's the shape of every successful style fine-tune: same knowledge, your format.

Step 5 — What went wrong? (5 min)

The four failures everyone hits, in order of frequency:

1. CUDA out of memory — lower per_device_train_batch_size to 1, cut max_seq_length to 1024, or (MLX) add --grad-checkpoint. QLoRA on a 3B base fits in ~4GB; if you OOM there, something else is using VRAM — close it. 2. The model parrots training examples verbatim — you overfit. Drop to 1 epoch or add more varied data. 3. The model got dumber at everything else — learning rate too high or too many epochs erased general ability. Go back to 2e-4 and 2 epochs; that's why they're the defaults. 4. Loss never falls — almost always a data-format problem: the text field isn't going through the chat template, or your JSONL keys don't match. Print one formatted sample and read it.

Weird driver or install errors instead? The troubleshooting section covers CUDA OOM and friends error-by-error.

What's next

• Sharpen answers with DPO — after SFT, a preference pass makes the model *choose* better answers, not just format them. Guide 13 has the working DPO script.
• Level up your data — browse all 47 datasets in the hub, filtered by license and category.
• Show it off — trained something fun? We feature reader fine-tunes in the Local AI Digest; reply to any issue with your before/after.

Frequently Asked Questions

What is the minimum GPU to fine-tune an LLM?

A 6GB NVIDIA card (RTX 3060/4060 class) comfortably fine-tunes a 7B model with QLoRA, and a 3B model fits in about 4GB. Full fine-tuning without QLoRA needs 10–20x more memory — that's why every step in this guide uses 4-bit bases with LoRA adapters.

Can I fine-tune an LLM on a Mac?

Yes. Apple's MLX framework trains LoRA adapters on 4-bit models using unified memory — a 16GB M-series Mac handles a 7B QLoRA fine-tune in roughly 8GB of working memory. It's slower than a desktop NVIDIA card but completely local and silent.

How many examples do I need to fine-tune?

For style, tone, and format: 200–1,000 good examples. LIMA demonstrated 1,000 curated samples can align a 65B model. For new domain skills, aim for 10k+. Quality beats quantity every time — one bad example teaches a bad habit.

How long does fine-tuning take?

With this guide's settings, roughly 20–60 minutes for 1,000 examples on an RTX 3060/4060-class GPU, and 20–40 minutes on a 16GB Apple Silicon Mac. Bigger datasets scale linearly: 10k examples is an overnight run, not a weekend.

Does fine-tuning add new knowledge to a model?

Not reliably — fine-tuning changes *how* a model answers, not *what it knows*. To make a model answer from your documents or current facts, use RAG (retrieval-augmented generation) instead, or combine both: RAG for the facts, a fine-tune for the voice.

← All Guides | Check GPU Compatibility