Guide 13: Fine-Tuning LLMs Locally — SFT, LoRA, and DPO Workflows

作者： Jakub Rusinowski · 最后更新： 2026年7月10日

Fine-tuning adapts a pretrained model to your specific task. This guide covers when to fine-tune, the three main techniques, and working code for all of them.

本指南内容

• When Should You Fine-Tune?
• The Three Techniques
• Dataset Preparation
• SFT with Unsloth (Fastest Method)
• DPO Training with TRL
• LLaMA-Factory (GUI Alternative)
• Evaluating Your Fine-Tuned Model
• Common Mistakes to Avoid

Fine-tuning adapts a pretrained model to your specific task. This guide covers when to fine-tune, the three main techniques, and working code for all of them.

First time? Start with Guide 14: Fine-Tune Your First LLM in 1 Hour — the hands-on recipe card. This guide is the full reference it links back to.

---

When Should You Fine-Tune?

Fine-tuning is powerful but not always the right tool. Follow this decision checklist:

1. Try prompting first — system prompts + few-shot examples solve 80% of cases 2. Try RAG next — if the problem is knowledge (facts, documents, private data), use RAG 3. Fine-tune when: consistent style/format, domain-specific language, latency constraints (no long system prompts), or behavior that can't be prompted

Good fine-tuning use cases:

• Customer support bot with your company's tone
• Code assistant that follows your team's conventions
• Medical/legal domain specialist
• Structured output extraction (always return JSON)

Poor fine-tuning use cases:

• "Make it smarter" — fine-tuning doesn't add world knowledge
• One-off tasks — prompting is faster
• Very small datasets (<100 examples) — will overfit

---

The Three Techniques

1. Supervised Fine-Tuning (SFT)

Train on input/output pairs. The model learns to produce the output given the input. This is the foundation of instruction-following models.

• Data format: prompt + ideal response
• Minimum dataset size: 200–500 examples
• Key parameters: learning rate (2e-4), epochs (1–3), max sequence length

2. LoRA / QLoRA (Parameter-Efficient Fine-Tuning)

Instead of updating all model weights (full fine-tuning requires 2x VRAM), LoRA adds small trainable rank-decomposition matrices to attention layers. QLoRA = LoRA on a 4-bit quantized base model — enables fine-tuning 7B models on a single 6GB GPU.

• r (rank): 8–64 — higher = more capacity, more memory
• alpha: typically 2x rank
• target_modules: q_proj,v_proj (minimum), k_proj,o_proj,gate_proj,up_proj,down_proj (full)
• VRAM: 7B QLoRA ≈ 6GB, 13B QLoRA ≈ 10GB, 70B QLoRA ≈ 48GB

3. DPO (Direct Preference Optimization)

Aligns the model with human preferences without a separate reward model. Requires paired data: a 'chosen' response and a 'rejected' response for the same prompt. Usually run after SFT.

• Data format: prompt + chosen response + rejected response
• Key parameter: beta (0.1–0.5) — controls how much the model deviates from the base
• When to use: when SFT gives correct but bland/unhelpful responses

---

Dataset Preparation

SFT Alpaca Format (simplest)

import json

# Write your training data as JSONL
examples = [
    {"instruction": "Summarize this in one sentence", "input": "Long text...", "output": "Short summary."},
    {"instruction": "Classify the sentiment", "input": "I love this product!", "output": "Positive"},
]

with open('train.jsonl', 'w') as f:
    for ex in examples:
        f.write(json.dumps(ex) + '\n')

ChatML Multi-Turn Format

{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello"}, {"role": "assistant", "content": "Hi! How can I help?"}]}

DPO Format

{"prompt": "Explain photosynthesis", "chosen": "Photosynthesis is the process by which plants convert sunlight into sugar using CO2 and water...", "rejected": "Plants make food from sun."}

---

SFT with Unsloth (Fastest Method)

Unsloth is 2x faster than standard training with 60% less VRAM.

pip install unsloth

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

# Load model with QLoRA config
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-3B-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,  # QLoRA
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_alpha=32,
    lora_dropout=0,
    bias="none",
)

# Load your dataset (Alpaca format)
dataset = load_dataset("tatsu-lab/alpaca", split="train[:1000]")  # first 1k for demo

# Format as instruction prompt
def format_prompt(example):
    return {"text": f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"}

dataset = dataset.map(format_prompt)

# Train
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        num_train_epochs=2,
        learning_rate=2e-4,
        output_dir="./output",
        save_strategy="epoch",
    ),
)
trainer.train()

# Save merged model for Ollama
model.save_pretrained_merged("merged_model", tokenizer, save_method="merged_16bit")

---

DPO Training with TRL

pip install trl transformers peft

from trl import DPOTrainer, DPOConfig
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

model_name = "meta-llama/Llama-3.2-3B-Instruct"  # your SFT checkpoint
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

dataset = load_dataset("argilla/dpo-mix-7k", split="train")
# dataset columns: prompt, chosen, rejected

lora_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])

training_args = DPOConfig(
    beta=0.1,                    # preference strength
    max_length=1024,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    learning_rate=5e-5,
    output_dir="./dpo_output",
)

trainer = DPOTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    peft_config=lora_config,
)
trainer.train()

---

LLaMA-Factory (GUI Alternative)

For those who prefer a visual interface over code:

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e '.[torch,metrics]'
llamafactory-cli webui

Then open http://localhost:7860 in your browser. LLaMA-Factory supports:

• SFT, RLHF, DPO, PPO training methods
• 100+ model families including Llama, Qwen, Gemma, Mistral
• Built-in dataset browser (supports all formats)
• LoRA merge and export to GGUF (Ollama-compatible)

---

Evaluating Your Fine-Tuned Model

Baseline comparison:

# Before vs After test
prompt = "[YOUR TASK-SPECIFIC PROMPT HERE]"
print("BASE:", base_model.generate(prompt))
print("FINE-TUNED:", finetuned_model.generate(prompt))

For code models: Run HumanEval Pass@1 before and after

For chat models: Use MT-Bench with GPT-4 as judge

For domain-specific tasks: Build a holdout test set (10–20% of your data) before training

Watch for catastrophic forgetting: Test general tasks (math, reasoning) to ensure the model didn't lose base capabilities.

---

Common Mistakes to Avoid

Mistake	Fix
Too little data (<100 examples)	Minimum 200–500 for meaningful fine-tuning
Too many epochs (>3)	Usually 1–3 epochs; more causes overfitting
Wrong JSONL format	Validate with python -c "import json; [json.loads(l) for l in open('data.jsonl')]"
Using wrong base model	Pick a model already close to your task (coding task → CodeLlama)
Forgetting to merge adapter	Unsloth's save_pretrained_merged() is required before Ollama import
No eval set	Always hold out 10% of data to measure improvement

---

→ Browse training datasets in the Dataset Hub | → Which dataset to pick? | → Convert the result to GGUF for Ollama

← 所有指南 | 检查 GPU 兼容性