Choosing the right dataset is the single biggest factor in a successful fine-tuning run. This guide cuts through the confusion with a practical decision framework.
The Five Dataset Categories
| Category | What It Is | When to Use |
|---|---|---|
| Pretraining | Raw text at massive scale (web, books, code) | Training a model from scratch — requires 100B+ tokens and serious GPU clusters |
| Instruction | Prompt + response pairs teaching a model to follow instructions | Most common fine-tuning goal — makes a base model into an assistant |
| Preference | Chosen/rejected response pairs for alignment (RLHF/DPO) | After SFT, when you want to make responses more helpful, safe, or stylistically aligned |
| Code | Source code, coding problems, and solutions | Building a coding assistant or improving a model's programming ability |
| Evaluation | Benchmarks to measure model capability (not for training) | Testing and comparing models — run these after fine-tuning to measure improvement |
Quick Decision Table: Goal → Dataset
| Your Goal | Category | Top Picks |
|---|---|---|
| Make a helpful chat assistant | Instruction | Stanford Alpaca, ShareGPT 52K, OpenHermes 2.5 |
| Teach complex reasoning | Instruction | WizardLM Evol Instruct 70k |
| Small but powerful assistant | Instruction | LIMA (1k), Smoltalk |
| Build a coding assistant | Code | Magicoder-OSS-Instruct, CodeFeedback |
| Pretrain a code model | Code | The Stack v2 |
| Align with human preferences | Preference | Anthropic HH-RLHF, UltraFeedback |
| Quick local DPO experiment | Preference | DPO Mix 7K |
| Multi-turn conversation quality | Instruction | ShareGPT 52K |
| Benchmark your fine-tuned model | Evaluation | MMLU (knowledge), HumanEval (code), MT-Bench (chat) |
Dataset Quality Signals
Not all datasets are equal. Here's how to evaluate one before committing:
Good signals:
- High monthly downloads (>500k suggests community trust)
- Backed by a published paper with ablation studies
- Clear license (Apache 2.0 or MIT for commercial use, CC-BY-NC for research)
- Explicit deduplication and quality filtering steps documented
- Transparent data source (human-written > GPT-3 > unknown LLM)
Warning signs:
- No license specified
- "Generated by GPT-4" with no quality filtering
- Very low download count for an old dataset
- No paper, blog post, or technical documentation
- Contains copyrighted text scraped without permission
Quality vs. Quantity: The LIMA Lesson
The LIMA paper (2023) made a striking discovery: a model fine-tuned on just 1,000 carefully curated examples was competitive with models trained on millions of examples from noisy datasets.
Key insight: surface alignment is shallow. Models already have world knowledge from pretraining. Fine-tuning teaches format and style, not new facts. A small, high-quality dataset (200–1,000 examples) often outperforms a large, noisy one (100k+).
Practical rule: Start with 500–2,000 high-quality examples specific to your task. Scale only if quality examples are exhausted.
Where to Find More Datasets
- Hugging Face Datasets — Search by task, language, license. Sort by downloads.
- Papers With Code — Datasets with linked papers and leaderboards.
- r/LocalLLaMA — Community shares new fine-tuning datasets weekly.
- DataComp — Systematic benchmark for data curation methods.
Building Your Own Dataset
Sometimes the best dataset doesn't exist yet. Here's a minimal viable approach:
1. Define your task precisely — 'customer support for a SaaS product' beats 'helpful assistant'
2. Collect 200–500 examples — real user questions + ideal answers, or synthesize with Claude/GPT-4
3. Use JSONL format:
{"messages": [{"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "Click 'Forgot Password' on the login page..."}]}
{"messages": [{"role": "user", "content": "Where can I find my invoice?"}, {"role": "assistant", "content": "Invoices are in Settings → Billing → Invoice History."}]}4. Iterate on failures — after one fine-tuning run, look at which types of queries the model gets wrong, add more examples for those cases.
5. Mix with general data — combine your custom data 50/50 with a general instruction dataset like OpenHermes 2.5 to prevent catastrophic forgetting.