Evaluating LLM Features: Testing AI Output Before Users Do
Key takeaway
LLM features need an evaluation set of real inputs with known-good outcomes, run automatically on every prompt, model, or retrieval change. Combine deterministic checks, model-graded scoring calibrated against human judgement, and sampled production review; without that, every change is a guess.
Traditional software either passes a test or doesn't. LLM features are probabilistic: the same input can produce different wording, and a prompt tweak that fixes one case can quietly break five others. Teams that ship AI features without an evaluation process end up judging quality by anecdote — whichever example someone tried last.
An evaluation process replaces that with evidence. It doesn't need to be elaborate, but it does need to exist before launch.
Build an evaluation set first
- Collect 100–300 real or realistic inputs covering common cases, edge cases, and known failure modes.
- Record what a good answer must contain — key facts, the right action, the correct refusal — rather than one exact expected string.
- Include adversarial inputs: off-topic questions, prompt-injection attempts, and requests the feature should decline.
- Add every production failure you find to the set, so it can never quietly come back.
Choose checks that fit the task
Use the cheapest reliable check for each requirement. Deterministic checks — valid JSON, required fields present, a figure within range, no forbidden phrases — are fast and exact. For qualities that need judgement, such as whether an answer is grounded in the retrieved documents or whether a summary misses a key point, a second model can grade the output against a written rubric.
Model-graded scoring is useful but not self-validating. Have people grade a sample of the same outputs and compare: if the grader disagrees with humans often, fix the rubric before trusting its scores.
Metrics worth tracking
- Task success — did the output achieve what the user needed?
- Groundedness — for retrieval-augmented features, are claims supported by the sources supplied?
- Refusal accuracy — does it decline what it should, and only that?
- Format validity — can downstream code parse the output every time?
- Latency and cost per request — quality that arrives too slowly or too expensively still fails.
Make it part of release
Run the evaluation set in CI whenever the prompt, model version, retrieval settings, or tool definitions change, and block releases that fall below an agreed threshold. After launch, sample a percentage of real conversations for human review each week and feed what you learn back into the set. Model providers update their models; your evaluation suite is how you find out what that update did to your product before your users do.
How CalliArc measures accuracy and reliability
On client projects we treat accuracy as something agreed and measured, not claimed. The same steps run on every AI build, whether it is a support assistant, a document-extraction pipeline, or an agent that acts on business systems.
- Agree what "correct" means first — in discovery we write down, with your domain experts, what a good answer must contain for each type of request and the minimum score the feature must reach before launch.
- Build the evaluation set from your data — real or realistic inputs drawn from your tickets, documents, and transactions, labelled by people who know the business, including the edge cases and requests the system must refuse.
- Measure a baseline before building — the simplest workable approach is scored first, so every later change to prompts, retrieval, or models is compared against a number rather than an impression.
- Report each metric separately — task success, groundedness, refusal accuracy, format validity, latency, and cost per request — because one blended score hides the failure that matters.
- Gate releases on the agreed threshold — evaluations run automatically on every change, and a release that drops below the threshold does not ship.
- Keep measuring in production — a sample of real interactions is reviewed by people each week, user feedback and escalations are tracked, and every new failure is added to the evaluation set.
Reliability is more than accuracy
A feature that is accurate in testing can still be unreliable in use. We also design for the failures you cannot evaluate away: timeouts and fallbacks when a model provider is slow or down, confidence thresholds that route uncertain cases to a person, logging that lets you trace any answer back to its inputs and sources, and alerts when quality metrics drift after a model or data change.