Ask a product team what would most improve their model and the answer is usually more data. The instinct is inherited from pretraining, where scaling laws did reward every additional token, and it survives into fine-tuning where the economics are different.
So the annotation budget grows, the label queue grows, and the training set becomes something nobody on the team has read.
The discipline that pushes the other way is called data-centric AI: holding the model fixed and improving the dataset instead of the architecture. For most applied work that is where the remaining gains sit.
What volume costs in visibility
A dataset too large to inspect is one whose failure modes you learn about from production. When the model behaves strangely on a customer segment, the honest answer to “what is in the training data for that segment?” is usually a guess.
Volume buys coverage and costs visibility, and that trade usually runs the wrong way.
It also compounds. Noisy examples do not simply fail to help; they teach the model whatever the noise happens to be correlated with.
Duplicate rows quietly upweight the behaviour they encode, and mislabelled edge cases push the decision boundary in the one region where it matters most.
None of this shows up in the training loss, which is why a team can watch a curve converge nicely and still ship a model worse than the one it replaced.
What the published results show
The evidence against volume-first is unusually clear, because several groups have run the experiment directly.
Meta’s LIMA fine-tuned a 65B LLaMA on “only 1,000 carefully curated prompts and responses,” with no reinforcement learning and no human preference modelling, concluding that “only limited instruction tuning data is necessary to teach models to produce high-quality output.”
Reasoning made the point more sharply. LIMO reached 63.3% on AIME24 and 95.6% on MATH500 from a curated set of roughly 800 examples, against 6.5% and 59.2% for previous fine-tuned models, using “only 1% of the training data required by prior approaches.”
Stanford’s s1 built a 1,000-question set selected on three stated criteria: difficulty, diversity and quality, and the supervised fine-tuning run “took 26 minutes on 16 NVIDIA H100 GPUs.”
None of this is confined to fine-tuning. Hugging Face filtered its FineWeb corpus, 15 trillion tokens drawn from 96 Common Crawl snapshots, through an educational-quality classifier.
The surviving FineWeb-Edu subset holds 1.3 trillion tokens, which means more than 90% of the source was discarded, and the authors report that models pretrained on it “exhibit better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC.” Throwing away most of the corpus produced a better model on the same compute.
Size is not the variable that moves the number
It would be easy to read all this as permission to train on less, which is the wrong lesson. The s1 team ran the ablation that settles it, holding the dataset at 1,000 examples and changing only the selection rule.
On AIME24, random selection scored 36.7, selecting the longest reasoning traces scored 33.3, and selecting for diversity alone scored 26.7, against 50.0 for the combined difficulty, diversity and quality selection.
Same size, same compute, and a gap of 13 to 23 points produced entirely by how the examples were chosen.
The pattern holds at pretraining scale. Work presented at ICLR 2026 on data-efficient pretraining introduced ASK-LLM, which scores each candidate example by asking an instruction-tuned model whether it is informative.
The authors report that it “consistently outperforms full-data training, even when we sample only 10% of the original dataset,” and that at 60% retention it trained a T5-Large “both better and 70% faster.”
Those experiments run on T5-style encoder-decoder models trained on C4 and not on decoder-only LLMs, so the mechanism travels more confidently than the exact figures. Their contrast with cheaper methods is the useful part: coverage-based sampling “often recover[s] the performance of training on the entire dataset,” while quality scoring can exceed it.
The failure in the other direction is real, though. Small and curated is not the same as small and arbitrary, and below the threshold our discussion of parameter-efficient fine-tuning sets out, a fine-tune will learn noise instead of the behaviour you intended, with prompting or retrieval the better first move.
Every curated-small result above sits on top of a very capable base model and a very deliberate selection rule, and removing either one makes the effect disappear.
What curation costs
Teams reach for volume because curation is unglamorous and the work is real. Before treating a small dataset as an advantage, budget for the following:
- Deduplication, including near-duplicates, which quietly reweight whatever they repeat.
- Contamination checks against the evaluation set, run before any headline number is quoted.
- Manual review of a random sample, by someone who understands the domain and not only the annotation guidelines.
- Inter-annotator agreement on the labels that matter, and a decision about what to do where it is poor.
- Deliberate balance across the slices you will be judged on in production, instead of the ones that were easy to collect.
Automated filtering carries its own bill. ASK-LLM needs one language model inference call per candidate example, so scoring a corpus before training on a tenth of it is a compute line item of its own.
The authors make that case, reporting that their approach trains 800M models to the performance of 1.5B models on unfiltered data, which is the comparison worth running on your own numbers first.
Priced honestly, curation often costs more per example than acquisition did. What it buys is a training set whose contents you can describe and correct.
Build the evaluation set first
The sequencing follows from all of this. Define what good looks like, assemble a held-out set that measures it, and only then decide what to train on.
Without it, a curated dataset and a carelessly truncated one look identical from the outside, and the team has no way to tell whether the last round of filtering helped or removed the only examples covering a hard case.
That ordering is awkward, because the evaluation set is the part with no immediate payoff and it comes first.
It is also the only thing that converts filtering from a judgement call into a measurement, which is why the data layer and the validation phase are treated as linked stages in the guide to AI product development.
From there it becomes a loop: curate, evaluate, look at what the model still gets wrong, and add examples that target it. Run that way, a dataset tends to converge on something smaller than anyone expected and considerably more useful than the pile it came from.
Where the data work and evaluation loop are the constraint instead of the compute, that is a resourcing problem more than a research one.
Neurotechnology Cloud’s applied AI and custom model work covers that ground, from dataset construction and review through to training and production monitoring on dedicated GPU infrastructure.
If your model is waiting on data work and not on hardware, get in touch.
