Tried and tested mixes for strong pretraining. Inspired by https://huggingface.co/blog/codelion/optimal-dataset-mixing
AI & ML interests
We release large pre-training datasets to accelerate open LLM development. Part of the Hugging Face Science team (hf.co/science)
Recent Activity
Papers
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Organization Card
🍷 FineData
This is the home of the 🍷 FineData team, a branch of the 🤗 Hugging Face Science Team releasing large scale pre-training datasets to accelerate open LLM development.
- 🍷 FineWeb: A 15T tokens English dataset for LLM pre-training. See the blogpost and paper.
- 📚 FineWeb-Edu: a filtered subset of the most educational content from FineWeb.
- 🥂 FineWeb2: an extension of FineWeb to over 1000 languages. See the paper.
- 📄 FinePDFs: 3T tokens of text data extracted from PDFs sourced from the Web. See the blogpost
- 🌐 FineWiki: an updated, better extracted version of Wikipedia in 300+ languages.
- 📄 FinePDFs-Edu: 350B+ highly educational tokens filtered from 📄 FinePDFs
- 💬 FineTranslations: 1+1T tokens of parallel text translated from 500+ 🥂 FineWeb2 languages
spaces 8
Running
2
The Synthetic Data Playbook: Generating Trillions of the Finest Tokens
📝
Visualize synthetic data experiments as an interactive bookshelf
Running
Featured
67
FinePDFs: Liberating 3T of the finest tokens from PDFs
📄
Running
11
FineWiki Viewer
🌐
Viewer to explore the finewiki dataset
Running
Featured
1.3k
FineWeb: decanting the web for the finest text data at scale
🍷
Generate a curated web‑text dataset for LLM training
Running
88
Scaling FineWeb to 1000+ languages: Step 1: finding signal in 100s of evaluation tasks
📝
Evaluate multilingual models using FineTasks
models 105
HuggingFaceFW/finepdfs_edu_classifier_eng_Latn
0.4B • Updated
• 32.4k • 2
HuggingFaceFW/finepdfs_dclm_classifier_eng_Latn
0.4B • Updated
• 113
HuggingFaceFW/finepdfs_edu_classifier_v2_eng_Latn
0.4B • Updated
• 36
HuggingFaceFW/finepdfs_ocr_quality_classifier_eng_Latn
0.4B • Updated
• 12
HuggingFaceFW/finepdfs_edu_classifier_guj_Gujr
0.3B • Updated
• 3
HuggingFaceFW/finepdfs_edu_classifier_nno_Latn
0.3B • Updated
• 11
HuggingFaceFW/finepdfs_edu_classifier_kaz_Cyrl
0.3B • Updated
• 6
HuggingFaceFW/finepdfs_edu_classifier_tam_Taml
0.3B • Updated
• 4
HuggingFaceFW/finepdfs_edu_classifier_azj_Latn
0.3B • Updated
• 5
HuggingFaceFW/finepdfs_edu_classifier_afr_Latn
0.3B • Updated
• 5
datasets 35
HuggingFaceFW/finephrase
Viewer
• Updated
• 2.71B • 2.99k • 2
HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
Viewer
• Updated
• 56.1M • 524
HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT
Viewer
• Updated
• 56.1M • 47.5k
HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
Viewer
• Updated
• 62.1M • 429 • 3
HuggingFaceFW/finepdfs_50BT-dclm_30BT-fineweb_edu_20BT
Viewer
• Updated
• 62.1M • 31.6k • 1
HuggingFaceFW/finepdfs_edu_100BT-shuffled
Viewer
• Updated
• 17.8M • 817
HuggingFaceFW/finepdfs_edu_100BT
Viewer
• Updated
• 17.8M • 5.25k
HuggingFaceFW/finepdfs_100BT-shuffled
Viewer
• Updated
• 14.6M • 609
HuggingFaceFW/finepdfs_100BT
Viewer
• Updated
• 29.9M • 6.78k
HuggingFaceFW/fineweb_edu_100BT-shuffled
Viewer
• Updated
• 102M • 415