LLM Evaluation Framework: What to Measure Pre-Ship
Build an LLM evaluation framework before you ship: ship gates, four eval types, golden sets, prompt regression tests, drift checks and eval costs.
AI Engineer, Viithiisys
Ship gates, not vibes
A ship gate is a numeric pass/fail condition an LLM feature must meet before release. Without one, "it looked good in the demo" becomes your release criterion, and nobody can say whether the next prompt edit made things better or worse.
Why does a demo not count as evidence?
LLM output is non-deterministic and the failures are quiet. A chatbot that answers 19 of 20 questions well still gives a confident wrong answer to the twentieth, and a person clicking through a demo sees the nineteen.
Both Anthropic's guidance on defining success criteria and building evals and OpenAI's evals guide start from the same premise: write down what good looks like in measurable terms, then test against it repeatedly.
What does a gate look like in practice?
A gate has three parts: a metric, a threshold and a comparison point. For example: "groundedness score on the golden set must not fall below the current production baseline, and zero cases in the safety subset may fail."
Pick the threshold from your baseline, not from a number you saw in a blog post. Zero tolerance suits safety and compliance cases. A tolerance band suits fuzzy quality scores, because judge scores wobble between runs.
What are the four eval types in an LLM evaluation framework, and when does each apply?
Four eval types cover most LLM features: code-based assertions, model-graded scoring, human review and production monitoring. Use the cheapest one that can detect the failure you care about, and add costlier ones only where it cannot.
| Eval type | Best for | Speed and cost | Main weakness |
|---|---|---|---|
| Code-based assertions | Format, schema validity, required fields, forbidden strings, exact-match answers | Milliseconds, near zero | Cannot judge meaning or tone |
| Model-graded (LLM-as-judge) | Relevance, groundedness, tone, rubric scoring at scale | Seconds, token cost per case | Judge bias; needs calibration |
| Human review | Subjective quality, new failure modes, judge calibration | Minutes per case, highest cost | Slow, inconsistent without a rubric |
| Production monitoring | Real-traffic drift, user-reported failures, latency | Continuous, sampled | Finds problems after users see them |
Where do code-based checks fit?
Start here. If your feature must return JSON with five fields, a schema check catches that in milliseconds, for free, deterministically. Many "AI quality" problems turn out to be format problems.
When is an LLM judge worth using?
When the property is semantic and you need volume. The MT-Bench paper reports that a strong LLM judge matched human preferences over 80% of the time, the same level of agreement humans had with each other, while also documenting position bias and verbosity bias. Treat a judge as an instrument you calibrate, not an oracle.
Why keep humans and live monitoring in the loop?
Humans find the failure categories you did not think of, and they label the samples you use to calibrate the judge. Production monitoring then catches what no offline set predicted. Skipping either leaves the other two eval types blind to new problems.
Building a golden set from your own data
A golden set is a fixed collection of inputs paired with expected outputs or grading rubrics, drawn from real usage. Build it from your own data, because public benchmarks measure someone else's task, not yours.
Where do the first 50 cases come from?
Pull real inputs: support tickets, search queries, documents, chat logs, after removing personal data. If the feature is new, have domain staff write realistic inputs, and cover three buckets: typical requests, edge cases such as empty or very long inputs, and adversarial attempts such as prompt injection.
For retrieval features, store the source passages with each case so you can score groundedness, not just fluency. Our LLM development work treats this dataset as a deliverable, versioned alongside the code.
How do you keep the set honest over time?
Every production failure becomes a new case, with its fixed expected behaviour. This is the same habit as adding a regression test after a bug. Review labels with two people on a sample, and resolve disagreements by tightening the rubric, not by averaging.
Keep a held-out slice that you never tune prompts against. If scores climb on the tuning slice and not the held-out slice, you are overfitting to your own tests.
Regression testing prompts like code
Prompts are code with a probabilistic runtime. Store them in version control, run the golden set on every change, and block merges when a gate fails. That is the whole idea, and it is easy to say and hard to maintain.
What goes in the pull-request check?
Run a fast subset: the code-based assertions plus a sampled slice of judge-scored cases. Report the diff against the main branch baseline per metric, and per case where scores moved. A single aggregate number hides the case that flipped from pass to fail while ten others improved slightly.
Teams building these checks in CI will find the discipline familiar from our QA testing practice: fixtures, baselines, flaky-test policy and failure triage all apply.
How do you handle flaky scores?
Set temperature low for eval runs where your product allows it, and run borderline cases several times. Compare against a tolerance band rather than exact equality. If a case flips between runs without any change, the case or the rubric is ambiguous, so fix that instead of retrying until green.
A prompt change that nobody can test is a production deployment that nobody reviewed.
How do you measure drift after a model update?
Run your full golden set against the new model version side by side with the old one, and compare per-metric scores and per-case flips. Do this before switching traffic, and again on a schedule, because hosted models change.
Is drift real or folklore?
It is documented. Researchers at Stanford and UC Berkeley, in How Is ChatGPT's Behavior Changing over Time?, compared March and June 2023 versions of GPT-4 and reported that its accuracy on identifying prime numbers fell from 84% to 51%, while other tasks improved. Same model name, different behaviour.
That is one study of one provider in one period, not a rate to plan around. The lesson is to pin model versions where you can and re-run evals when you cannot.
What should a drift check compare?
Compare the same golden set, same judge, same settings, two model versions. Look at three things: aggregate metric change, the list of cases that flipped from pass to fail, and format compliance, since small output-format shifts break downstream parsers first.
Then check live traffic: sample production outputs, score them with the same judge, and alert when the rolling score leaves its tolerance band. Our MLOps work wires this into the same dashboards teams already use for latency and error rates.
What does it cost to run evals continuously?
Eval cost is mostly token spend on judge calls, multiplied by cases and run frequency. It stays manageable if you tier the suite: cheap checks always, expensive checks on a schedule, and batch pricing for anything that is not urgent.
How do you keep the bill down?
Tier by cost. Code assertions run on every commit. A sampled judge subset runs on every pull request. The full suite runs nightly or before release, and a sampled scoring job covers live traffic.
Both major providers discount asynchronous work: Anthropic's Message Batches documentation and OpenAI's Batch API guide each list a 50% price reduction for batch requests. Nightly full-suite runs are an obvious fit. Check current pricing pages before budgeting, since these terms change.
What is the cost of not running them?
The comparison that matters is eval spend against the cost of a bad release: a rollback, a support spike, or a wrong answer sent to a customer. Judge calls on a few hundred cases are usually small next to engineer-hours spent debugging a regression found by users.
The honest limit: evals do not remove the need for monitoring, and a judge that is never re-calibrated can give false comfort. Budget for the human labelling time as well as the tokens.
What is in our LLM evaluation framework template?
Our template is a starting structure, not a benchmark: a golden-set schema, a rubric format, a gate definition file and a CI job. It reflects how Viithiisys has approached delivery since 2007, across 500+ projects.
What does the template contain?
- Dataset schema: input, expected behaviour or rubric, source passages, tags for bucket and severity.
- Rubrics: written scoring criteria for each judge-scored metric, with worked examples of pass and fail.
- Gate file: metric, threshold and comparison point, reviewed like any other config.
- CI job: runs the tiered suite and posts a per-case diff on the pull request.
- Drift job: a scheduled run against the pinned model and the candidate.
Who is it for?
Teams that have a working prototype and no release criteria. Our engineers in Mohali and our Canadian office in Markham, Ontario, work with clients across the US, UK and Canada, from founders to enterprise teams. Clients have included Paytm, Snapdeal, IKEA, Nestle, Shiprocket and Vikram Solar.
For a new product, Moonship delivers a working MVP in 30 days against a fixed scope, and the eval set is part of that scope. Get the eval template and we will walk through it against your feature.
If your LLM feature is stuck at "looks fine, nobody will sign it off", the broken workflow assessment is the place to start. For build support, see our AI development services.
FAQ
- What is an LLM evaluation framework?
- It is the combination of a test dataset, scoring methods and pass/fail thresholds that tells you whether an LLM feature is good enough to release. It runs on every prompt, model or retrieval change, the same way a test suite runs on every code change.
- How many examples does a golden set need?
- Start with 50 to 100 real cases covering normal traffic, edge cases and known failures, then grow it from production incidents. Coverage of distinct failure modes matters more than raw count. A small set you trust beats a large one nobody has reviewed.
- Can I use an LLM to grade another LLM's output?
- Yes, with checks. The MT-Bench paper found a strong judge model agreed with human raters over 80% of the time, but judges show position and verbosity bias. Calibrate the judge against a sample of human labels and re-check it whenever you change the judge model.
- How often should LLM evals run?
- Run a fast, cheap subset on every pull request, the full suite nightly or before each release, and a sampled scoring job on live traffic. Also run everything whenever the provider announces a model update or deprecation, since hosted models change underneath you.