Evaluating LLM Features: Testing AI Output Before Users Do
Key takeaway
LLM features need an evaluation set of real inputs with known-good outcomes, run automatically on every prompt, model, or retrieval change. Combine deterministic checks, model-graded scoring calibrated against human judgement, and sampled production review; without that, every change is a guess.
Traditional software either passes a test or doesn't. LLM features are probabilistic: the same input can produce different wording, and a prompt tweak that fixes one case can quietly break five others. Teams that ship AI features without an evaluation process end up judging quality by anecdote — whichever example someone tried last.
An evaluation process replaces that with evidence. It doesn't need to be elaborate, but it does need to exist before launch.
Build an evaluation set first
- Collect 100–300 real or realistic inputs covering common cases, edge cases, and known failure modes.
- Record what a good answer must contain — key facts, the right action, the correct refusal — rather than one exact expected string.
- Include adversarial inputs: off-topic questions, prompt-injection attempts, and requests the feature should decline.
- Add every production failure you find to the set, so it can never quietly come back.
Choose checks that fit the task
Use the cheapest reliable check for each requirement. Deterministic checks — valid JSON, required fields present, a figure within range, no forbidden phrases — are fast and exact. For qualities that need judgement, such as whether an answer is grounded in the retrieved documents or whether a summary misses a key point, a second model can grade the output against a written rubric.
Model-graded scoring is useful but not self-validating. Have people grade a sample of the same outputs and compare: if the grader disagrees with humans often, fix the rubric before trusting its scores.
Metrics worth tracking
- Task success — did the output achieve what the user needed?
- Groundedness — for retrieval-augmented features, are claims supported by the sources supplied?
- Refusal accuracy — does it decline what it should, and only that?
- Format validity — can downstream code parse the output every time?
- Latency and cost per request — quality that arrives too slowly or too expensively still fails.
Make it part of release
Run the evaluation set in CI whenever the prompt, model version, retrieval settings, or tool definitions change, and block releases that fall below an agreed threshold. After launch, sample a percentage of real conversations for human review each week and feed what you learn back into the set. Model providers update their models; your evaluation suite is how you find out what that update did to your product before your users do.