The fastest way to ship a disappointing AI feature is to test it by trying a few prompts, seeing good answers and calling it done. Large language models are very good at producing plausible output on the examples you happen to try. The failures hide in the cases you did not think of, and users find them within days.
Evaluation, or "evals", is how you find those failures first. It is less glamorous than prompt writing and far more important. I touched on it in adding an AI feature to your product; this post is the practical method.
Why normal testing is not enough
A normal unit test checks that a function returns exactly the expected value. LLM output is different:
- The same input can produce different output on different runs.
- There are many acceptable answers, worded differently.
- Small prompt or model changes can improve some cases and quietly break others.
- Failures are often subtle: a wrong number in an otherwise good summary, a missing caveat, a confident answer to a question that should have been declined.
So instead of exact assertions, you build a set of cases and a way to score outputs against what "good" means for your feature.
Step 1: Define what good means, in writing
Before writing any test, write a short rubric for your feature. For example, for a feature that summarizes technician notes on a job:
- Includes every action the technician reported.
- Includes parts used and any follow-up needed.
- Adds nothing that is not in the notes.
- Under a set length.
- In the language of the job record.
Each line should be something a person could check. If you cannot write down what good looks like, the feature is not ready to build.
Step 2: Build a test set from real cases
Collect real inputs, anonymized where needed: actual notes, actual emails, actual questions. Real data is messy in ways invented examples are not. Include:
- Typical cases, the bulk of what the feature will see.
- Hard cases: long inputs, very short inputs, mixed languages. For Arabic-speaking users, mixed Arabic, English and Arabizi text is a real case, not an edge case.
- Adversarial cases: inputs trying to make the model ignore instructions. See prompt injection risks.
- Cases that should be refused or flagged: missing information, out-of-scope questions.
Start with a few dozen. A small, well-chosen set run on every change beats a huge set nobody runs. Grow it every time a real failure appears: each bug becomes a test case.
Step 3: Automatic checks first
Many quality criteria can be checked with plain code, which is fast, cheap and deterministic:
- Output parses as valid JSON and matches the schema.
- Required fields are present.
- Length is within limits.
- Numbers, dates and names in the output appear in the input. This catches many hallucinations.
- Forbidden content does not appear: internal identifiers, other customers' data, phrases you never want shown.
- Language matches the expected language.
Asking the model for structured output, then validating it, makes most features far easier to test than free prose.
Step 4: Model-graded checks, used carefully
Some criteria need judgment: "is this summary faithful to the notes?" or "does this reply answer the question politely?" Here you can use another model call as a grader, with a clear rubric and the input and output in front of it.
Rules for making graders useful:
- Ask narrow, specific questions with yes or no answers, one criterion at a time, rather than "rate this 1 to 10".
- Ask for a short reason before the verdict, and read those reasons while setting it up.
- Check the grader against human judgment on a sample. If it disagrees with you often, fix the rubric before trusting it.
- Keep the grader's model and prompt fixed while you compare versions of the feature, so you are measuring one change at a time.
A grader is a tool for scale, not a replacement for looking at outputs yourself.
Step 5: Look at the outputs
Every time you change the prompt, the model or the retrieval, read a sample of outputs side by side with the previous version. Numbers summarize; reading shows you what changed. Many of the most important failures are things you did not know to check for until you saw them.
Step 6: Run evals on every change
Wire the test set into your workflow, just like other tests:
- Run it when the prompt, model, retrieval or surrounding code changes.
- Compare scores with the previous version, per criterion and per case.
- Block a release when an important criterion regresses.
Model providers update and retire models. When you switch models, even to a newer version from the same provider, run the full set. Behaviour can change in ways that matter to you specifically.
Cost and latency belong in the report too. A version that scores slightly better but costs much more or takes twice as long may not be the better choice.
Step 7: Keep evaluating in production
Offline tests do not cover everything users will send. In production:
- Log inputs and outputs, with care for privacy and retention.
- Give users an easy way to flag bad output, like a thumbs down or "report a problem".
- Review a sample of real outputs regularly.
- Watch for drift: new kinds of inputs, rising error rates, more user corrections.
Feed what you find back into the test set.
Design for failure anyway
No amount of evaluation makes an LLM perfect. The product should assume some outputs will be wrong:
- Keep a human in the loop for anything that matters: drafts that a person approves, suggestions rather than automatic actions.
- Show sources when answers are based on documents, so users can verify.
- Make it easy to edit or ignore the AI output.
- Fail safely: if validation fails, fall back to a simple message, not a broken screen.
A minimal setup to start
- Write a rubric.
- Collect 30 to 50 real cases, including hard and adversarial ones.
- Add automatic checks for structure and facts.
- Add one or two narrow model-graded checks, validated against your own judgment.
- Run the set on every change and read the diffs.
It takes a day or two to set up, and it turns AI work from guesswork into engineering.
If you are adding AI to a product and want help making it reliable, see my work and get in touch.