Open Datasets for Machine Learning: The Essential Landscape
The open datasets that built modern machine learning — ImageNet to Common Crawl — where to find them, their licenses, and how to choose one for your project.
Modern machine learning was built on a handful of public datasets. ImageNet made computer vision leap forward; Common Crawl and Wikipedia seeded the language models; benchmark sets like GLUE turned model development into a comparable science. Knowing this landscape — and its licenses — is a practical skill for anyone training or evaluating models.
The canonical sets
- ImageNet (vision): 14M labeled images; the dataset whose 2015-era benchmark collapse produced today's convolutional neural networks.
- Common Crawl (text): petabytes of web crawl; the raw material behind most LLM pretraining after heavy filtering.
- Wikipedia / Gutenberg (text): curated, licensed, high-quality; the backbone of instruction tuning and classic NLP.
- MNIST / CIFAR (vision, teaching): small, canonical, the "hello world" sets.
- LibriSpeech / Common Voice (speech): Mozilla's crowd-sourced Common Voice is the leading openly licensed multilingual speech corpus.
- Hugging Face Hub: the current de facto index — hundreds of thousands of datasets with license tags.
- Google Dataset Search, Kaggle, data.gov: discovery layers for everything else.
Licenses are the product
A dataset's license determines what you may train on and ship. The spectrum matters: CC0/public domain (unrestricted), CC-BY (use with attribution), CC-BY-SA (share-alike obligations that can reach derivative models, depending on jurisdiction — contested), CC-BY-NC (no commercial use — a landmine for products), and research-only or custom licenses (Licensing is part of the dataset, not fine print). Models trained on NC data have been pulled from commercial products; checking the license is cheaper than rebuilding.
Choosing one for a project
- Define the task and the needed scale; a 10k well-curated set often beats a 10M noisy one (classification accuracy is not linear in raw size).
- Check license against your use, including the commercial question.
- Check provenance and known biases — every dataset encodes the world that made it (how to open AI training data covers the systemic issues).
- Check the eval set is separate and uncontaminated — benchmark leakage is the replication crisis of ML.
Going deeper. Open Data for AGI: Why Public, Free, and Open Datasets Matter by the author of this wiki surveys this landscape in depth — the landmark datasets, their histories, and the case for treating open data as critical infrastructure. Instant download at the author's bookstore.
Tags
bookshop datasets machine learning training data