SlimPajama
A heavily deduplicated and cleaned version of RedPajama. It removes low-quality and repetitive data to improve training efficiency.
Load it
from datasets import load_dataset
ds = load_dataset("cerebras/SlimPajama-627B")Preview a sample row
{"text": "The mitochondria is the powerhouse of the cell...", "meta": {"redpajama_set_name": "RedPajamaCommonCrawl"}}Frequently asked questions
Can I use SlimPajama commercially?
Yes — SlimPajama is released under Apache 2.0, a permissive license that allows commercial use, including training models you ship in a product. Check the dataset card for attribution requirements before release.
How much data does SlimPajama contain, and do I need all of it?
SlimPajama contains 627B Tokens. You rarely need all of it: for style and format fine-tuning, a few hundred to a few thousand examples are enough — load a slice (e.g. split="train[:1000]") and scale up only if quality plateaus.
What is SlimPajama best used for?
Efficient English pretraining on a heavily deduplicated corpus. It belongs to the Pretraining section of our dataset hub, where you'll find alternatives and complementary sets.