LLM Testing: What to Check Before You Ship

What LLM testing is

LLM testing means running a language-model feature against fixed example inputs and scoring the outputs against written criteria, so you know whether a change helped before users see it.

Ordinary software tests compare an output with one expected value. A language model gives different wording each time, and a wrong answer looks exactly like a right one. So the tests score properties of the answer: is it valid JSON, does it stick to the source document, does it refuse what it should refuse.

The vocabulary is contested. Some teams treat LLM evaluation as measuring quality in general and testing as the pass/fail gate on top. Others use the words interchangeably. This guide uses testing for the gate and evals for the scoring behind it, since that is how most release pipelines end up structured.

How LLM testing works

A test run feeds every case in a fixed dataset through the feature, scores each output, compares the scores with a baseline, and passes or fails the change. The same loop runs after any edit to the prompt, model, retrieval setup or tools.

Four pieces make it work:

• A test set of real inputs, each with an expected behaviour or a scoring rubric.
• Scorers, which are code checks, a second model acting as judge, or a person.
• A baseline, meaning the scores the current production version gets.
• A threshold agreed before launch, such as "no safety case may fail, and the overall score may not drop below baseline."

Both Anthropic's guidance on defining success criteria and building evals and OpenAI's evals guide start from the same premise: write down what good looks like in measurable terms, then test against it repeatedly.

Who needs LLM testing and who can skip it

Any team putting a language-model feature in front of customers or into a business process needs it. A throwaway internal prototype used by its author does not, and building a use for it is wasted effort.

The line is consequence. If a wrong answer reaches a customer, writes to a system of record, or feeds a decision, you need release criteria. If the worst outcome is that the author rolls their eyes and re-runs it, skip the use and keep a notes file of cases that broke.

Teams that benefit most are the ones with a working demo and no way to say whether the next prompt edit helped. At Viithiisys, with engineering in Mohali and a Canadian office in Markham, Ontario, that is the situation we are most often asked to untangle, for clients in the US, UK and Canada. The fix is almost always the same: write the criteria down, then automate them.

The four kinds of check and when each applies

Four check types cover most features: code assertions, model-graded scoring, human review and production monitoring. Use the cheapest one that can detect the failure you care about, and add a costlier one only where the cheaper one is blind.

Start with code assertions. If the feature must return JSON with five fields, a schema check catches a broken response in milliseconds. Many problems reported as "the AI got worse" turn out to be format problems.

Can an LLM grade another LLM?

Yes, and it is the only way to score meaning at volume, but a judge model is an instrument you calibrate. It is not an authority. Check it against human labels before you trust its numbers.

The MT-Bench paper reports that a strong judge matched human preferences over 80% of the time, about the level at which humans agreed with each other. The same paper documents position bias, where the judge favours an answer because of where it appears, and verbosity bias, where longer answers score higher.

Practical handling: write the rubric as explicit criteria with worked pass and fail examples. Have a person label 30 or so cases and compare. When the judge and the person disagree, tighten the rubric before you blame the judge. Re-run that comparison whenever you swap the judge model.

Building the test set from your own traffic

Build the set from real inputs, because public benchmarks measure someone else's task. Pull support tickets, search queries, documents or chat logs, remove personal data, and label what a good answer looks like.

For a new feature with no traffic yet, have domain staff write realistic requests. Sort cases into three buckets: typical requests, edge cases such as empty or very long inputs, and adversarial inputs such as prompt injection.

For retrieval features, store the source passages next to each case so you can score whether the answer is grounded in them, not just whether it reads well.

Keep a held-out slice that you never tune prompts against. If scores climb on the tuning slice and stay flat on the held-out one, you are fitting the prompt to your own tests. Our LLM development work treats this dataset as a deliverable, versioned next to the code.

LLM regression testing for prompts

LLM regression testing means treating a prompt like source code: keep it in version control, run the test set on every change, and block the merge when a gate fails. It sounds simple, and the upkeep is the hard part.

Run a fast subset on each pull request: all code assertions plus a sampled slice of judge-scored cases. Report the change against the main branch per metric and per case. A single average hides the case that flipped from pass to fail while ten others improved slightly.

Flaky scores need a policy. Use low temperature for test runs where your product allows it, run borderline cases several times, and compare against a tolerance band instead of exact equality. If a case flips with no change underneath, the case or its rubric is ambiguous, so fix it. Our QA testing practice brings the same discipline of fixtures, baselines and triage to this work.

A real case: an agent that scored well and still failed users

One of our production agents shipped with a good evaluation score, then quietly got worse for real users. The test set no longer matched the traffic. We have not named the client, and the account below is as we described it in our write-up.

Usage had shifted toward a request type that made up almost none of the original test set. When we scored that slice separately, it performed clearly worse than the headline number implied, and the headline had been hiding it. The full account sits in our post on nine AI agent failure modes and how we fixed them, under eval rot.

What we changed:

• Sample real production runs every week and label the failures.
• Add those failures to the test set as permanent regression cases.
• Run the full set before every prompt or model change is released.

Checking a model update before you switch

Run the whole test set against the new model version next to the old one, using the same judge and settings, then compare scores and the list of cases that flipped. Do it before moving traffic and again on a schedule.

Drift is documented. Researchers at Stanford and UC Berkeley, in How Is ChatGPT's Behavior Changing over Time?, compared March and June 2023 versions of GPT-4 and reported that accuracy on identifying prime numbers fell from 84% to 51%, while other tasks improved. That is one study of one provider in one period, so treat it as proof that drift happens and not as a rate to plan around.

Pin model versions where the provider allows it. Where it does not, schedule the run. Check format compliance first, because a small shift in output format breaks downstream parsers before anyone notices a quality change.

What LLM testing costs to run

The running cost is mostly token spend on judge calls, multiplied by case count and run frequency. Tiering keeps it low: cheap checks always, expensive checks on a schedule, and batch pricing for anything that can wait.

Run code assertions on every commit, a sampled judge subset on every pull request, the full suite nightly or before release, and a sampled job on live traffic. Anthropic's Message Batches documentation lists a 50% price reduction for batch requests, which suits nightly full-suite runs. Check the current pricing page before you budget, because those terms change.

The larger cost is people. Someone has to write rubrics and label the calibration sample. Budget their time as well as the tokens, and compare the total with a rollback or a wrong answer sent to a customer. We scope this work per project after a discovery call, since it depends on how many features and how much traffic you have.

How to tell whether your tests are any good

Good tests fail when the feature is broken. Prove it by introducing a known defect and watching the suite catch it. If the suite stays green on a deliberately worse prompt, it is measuring something other than quality.

Four checks to run on the suite itself:

• Break the prompt on purpose, for example by removing a constraint, and confirm the score drops on the cases that depend on it.
• Look for cases that every version passes. They carry no signal and can be retired.
• Compare slice scores against the overall score, so a small segment cannot hide inside an average.
• Read twenty failures by hand each month. If the judge's reasons do not match yours, recalibrate.

This is also where LLM evals earn their keep as a team habit. A suite nobody reads decays into a green tick that means nothing, and that is worse than having none.

What typically goes wrong in testing LLM applications

The common failures are a stale test set, a judge nobody calibrated, a single average that hides a bad segment, and tests that only cover the model while the tools around it fail silently.

The tools matter more than teams expect. In another of our production agents, a search tool returned an empty list when its login expired. The model read that as "nothing matched" and told the user so. Nothing errored, so nothing was logged. A test that only scores the final answer would have passed it.

So test the tool layer separately. An empty result from a tool that is rarely empty should raise a flag, and every tool should report failure in a way the code can see. Add the failures you find to the same test set, and score the whole run, not only the last message.

Getting a first set of checks in place

Start small: write down three pass/fail criteria for your riskiest feature, collect 50 real inputs, add code assertions, and wire them into the pull request. You can add judges and monitoring once that loop runs.

Viithiisys has shipped software since 2007, across 500+ projects, and the pattern holds on every one: the teams that can ship a prompt change on a Tuesday afternoon are the ones with a test set they trust. Setting up the tracking and release side belongs with MLOps, and the test set is part of that.

If you have an LLM feature that works in the demo and that nobody will sign off, book a broken workflow assessment. We read through your prompts, tools and failure cases and tell you which checks to build first. If you want to talk it through before that, use the contact form.