Written by Jakub Rusinowski · Last updated July 10, 2026
Choosing the right dataset is the single biggest factor in a successful fine-tuning run. This guide cuts through the confusion with a practical decision framework.
Choosing the right dataset is the single biggest factor in a successful fine-tuning run. This guide cuts through the confusion with a practical decision framework.
| Category | What It Is | When to Use |
|---|---|---|
| Pretraining | Raw text at massive scale (web, books, code) | Training a model from scratch — requires 100B+ tokens and serious GPU clusters |
| Instruction | Prompt + response pairs teaching a model to follow instructions | Most common fine-tuning goal — makes a base model into an assistant |
| Preference | Chosen/rejected response pairs for alignment (RLHF/DPO) | After SFT, when you want to make responses more helpful, safe, or stylistically aligned |
| Code | Source code, coding problems, and solutions | Building a coding assistant or improving a model's programming ability |
| Evaluation | Benchmarks to measure model capability (not for training) | Testing and comparing models — run these after fine-tuning to measure improvement |
| Your Goal | Category | Top Picks |
|---|---|---|
| Make a helpful chat assistant | Instruction | Stanford Alpaca, ShareGPT 52K, OpenHermes 2.5 |
| Teach complex reasoning | Instruction | WizardLM Evol Instruct 70k |
| Small but powerful assistant | Instruction | LIMA (1k), Smoltalk |
| Build a coding assistant | Code | Magicoder-OSS-Instruct, CodeFeedback |
| Pretrain a code model | Code | The Stack v2 |
| Align with human preferences | Preference | Anthropic HH-RLHF, UltraFeedback |
| Quick local DPO experiment | Preference | DPO Mix 7K |
| Multi-turn conversation quality | Instruction | ShareGPT 52K |
| Benchmark your fine-tuned model | Evaluation | MMLU (knowledge), HumanEval (code), MT-Bench (chat) |
Not all datasets are equal. Here's how to evaluate one before committing:
Good signals:
Warning signs:
The LIMA paper (2023) made a striking discovery: a model fine-tuned on just 1,000 carefully curated examples was competitive with models trained on millions of examples from noisy datasets.
Key insight: surface alignment is shallow. Models already have world knowledge from pretraining. Fine-tuning teaches *format and style*, not new facts. A small, high-quality dataset (200–1,000 examples) often outperforms a large, noisy one (100k+).
Practical rule: Start with 500–2,000 high-quality examples specific to your task. Scale only if quality examples are exhausted.
Sometimes the best dataset doesn't exist yet. Here's a minimal viable approach:
1. Define your task precisely — 'customer support for a SaaS product' beats 'helpful assistant'
2. Collect 200–500 examples — real user questions + ideal answers, or synthesize with Claude/GPT-4
3. Use JSONL format:
{"messages": [{"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "Click 'Forgot Password' on the login page..."}]}
{"messages": [{"role": "user", "content": "Where can I find my invoice?"}, {"role": "assistant", "content": "Invoices are in Settings → Billing → Invoice History."}]}4. Iterate on failures — after one fine-tuning run, look at which types of queries the model gets wrong, add more examples for those cases.
5. Mix with general data — combine your custom data 50/50 with a general instruction dataset like OpenHermes 2.5 to prevent catastrophic forgetting.
→ Ready to fine-tune? See the full workflow guide | → Browse all datasets