Data Annotation: How to Get Training Data Right
Key takeaway
Your model cannot be more accurate than your labels are consistent. Measure agreement between annotators before scaling up — if two trained people disagree on 15% of items, that disagreement is your accuracy ceiling, and the fix is clearer guidelines, not more data.
Annotation is treated as the boring part of a machine learning project and is routinely the part that determines whether it works. Models learn the labels they're given, including the inconsistent ones.
Write guidelines that settle arguments
- Define each label with a positive example, a negative example, and the nearest confusing case.
- Give an explicit rule for ambiguity — a "cannot determine" option beats forcing a guess, because forced guesses become noise.
- Cover edge cases as you find them, and version the document so you know which rules applied to which batch.
- Have the guidelines written by someone who has personally labelled a few hundred items. Guidelines written from a desk fail contact with real data.
Measure agreement before you scale
Have several annotators label the same set independently and measure agreement. Low agreement means the task or the guidelines are unclear — not that the annotators are careless. Fix the definition and re-measure. Scaling a task with poor agreement simply buys a larger quantity of unreliable labels.
Quality control at scale
- Seed known-answer items through the work to monitor each annotator continuously.
- Overlap a percentage of items between annotators, and route disagreements to an adjudicator.
- Track per-annotator accuracy and give feedback quickly; quality drifts within days without it.
- Re-check old batches when the guidelines change — old labels under old rules will contradict the new ones.
Practical decisions
- Start with a pilot of a few hundred items. It exposes almost every guideline problem before you commit a budget.
- Subject-matter tasks — clinical, legal, or engineering — need expert annotators, and cost accordingly. Don't crowdsource a task your own staff would find hard.
- For rare classes, deliberately over-sample them; random sampling will produce almost none.
- Consider model-assisted labelling once you have a first model: annotators correct predictions rather than starting from scratch, which is faster and reduces drift — provided you still audit for the bias it introduces.
Hold back a clean test set
Reserve a carefully adjudicated set that is never used for training and never changes. It's the only stable way to tell whether a new model is genuinely better, and it's worth more care per item than any other data you label.