AI Evals for PMs: How to Know Your AI Feature Works
AI features fail in ways A/B tests miss. What evals are, three grading methods compared, and how PMs set the pass rate that gates a release.
Running the same prompt through the same model twice can produce different answers. An eval tests those outputs repeatedly against defined quality criteria. Engineers build the test harness, while the PM defines what the output needs to do for the user.
Why your A/B testing instincts don't transfer directly
If you have internalized A/B testing, you already hold most of the statistical intuition evals require. Sampling error, confidence intervals, the temptation to stop early when the numbers look good: all of it carries over. What changes is the thing being measured.
A conversion funnel may have a defined event at each step, such as whether the user clicked. An AI answer can require several judgments at once: factuality, audience fit, latency, and safety-policy compliance. A single output can pass some criteria and fail others, which a single conversion metric may not reveal.
The second shift is that there is no fixed variant. In a classic test, variant B is the same pixels for every user. In an AI feature, "variant B" is a prompt-and-model combination that produces a different output for every input, and sometimes a different output for the same input. You are comparing two probability distributions of behavior rather than two fixed screens, and you need enough graded samples from each to say anything about the difference.
A/B tests assess whether a change affects retention or task completion in production. Evals help determine whether a version is ready to show to users in the first place.
Three ways to grade an output
Every eval reduces to the same loop: take an input, generate an output, grade it. The grading method is the design decision, and there are three.
| Grading method | Cost per output | Speed | Reproducible | Catches | Misses |
|---|---|---|---|---|---|
| Code-based checks (exact match, regex, schema validation, latency assertions) | Near zero | Milliseconds | Fully | Broken JSON, banned phrases, missing citations, blown latency budgets | Anything needing judgment: tone, relevance, subtle wrongness |
| LLM-as-judge (a model grades the output against a rubric) | Cents | Seconds | Mostly, with a pinned judge model and rubric | Relevance, instruction-following, faithfulness to a source document | Failure modes the judge model shares with the model being graded |
| Human review | Dollars, plus reviewer time | Hours to days | Low, since reviewers disagree | Domain correctness, real harm, the failure nobody predicted | Scale. Humans cannot grade every output of every run |
Code-based checks provide a baseline. They are efficient enough to run on every commit; Hamel Husain notes that teams often under-invest in this layer before adopting more complex methods1.
LLM-as-judge is used for tasks that code cannot validate. Research by Zheng et al. shows that GPT-4 judgments agree with human raters more than 80 percent of the time on chat-quality comparisons, a rate similar to human-to-human agreement2. The same paper documents the judge's weaknesses: position bias (preferring the first answer shown), verbosity bias (preferring longer answers), and self-preference when grading outputs from its own model family2.
Calibrate the judge before you trust it. Take 50 outputs your team has hand-graded, run the judge on the same 50, and measure agreement. If the judge and your reviewers disagree on more than roughly one in five, fix the rubric before you automate anything. An uncalibrated judge gives you a precise pass rate that happens to be wrong.
Human review is the most expensive method but can provide the ground truth used to calibrate other methods and discover new failure modes. A common stack uses code checks on every commit, judge runs on release candidates, and human review for a scheduled sample and incidents.
Offline gates the release, online finds what you missed
An offline eval uses a fixed dataset, a scoring method, and a pass rate to assess a change before release. It serves as a release check, similar to a regression suite, without exposing users to the new version.
An online eval monitors live traffic after release. It can sample outputs for judge review and track thumbs-up and thumbs-down feedback, escalation rates, and the edits between an AI draft and what the user keeps. These checks can reveal failures missing from the offline dataset. Choose a sampling rate that balances the expected failure frequency, privacy requirements, and grading cost.3
When an online eval reveals a failure, add the example to the offline dataset as you fix it. Otherwise, a later prompt change can reintroduce the problem without the release checks detecting it.
Add triaged failures to the golden dataset when they represent a reproducible behavior the release check should catch. Review the dataset regularly so it reflects the failure modes that matter now.
If your product chains multiple model calls, the offline story gets more granular. Step-level and handoff evals for agent pipelines are covered in building agentic products.
Building a golden dataset
A golden dataset is the fixed set of inputs (and expected behaviors) your offline evals run against. Two decisions matter: what goes in it, and how big it is. Most of the attention goes to size, but the contents decide whether the set catches anything.
Start from real failures: support tickets where the AI answer was wrong, thumbs-down events with the output attached, and internal reports with enough context to reproduce the issue. Synthetic happy-path examples can confirm intended behavior, while failure-sourced examples cover known limits. Practitioners often recommend defining evals from observed failure states as well as planned requirements.4
A useful first golden set can include support escalations, thumbs-down events, and known edge cases such as empty input, hostile input, or a question outside the product's scope. Start with examples that cover distinct failure modes, then grow the set from production findings.
Coverage comes first, but size is not optional, because your sample-size intuition still applies. Suppose your assistant passes 90 percent of a 100-example set, an engineer changes the prompt, and the rate reads 87 percent. Regression or noise? On 100 examples, the 95 percent confidence interval around a 90 percent pass rate is roughly plus or minus 6 points, so 87 is statistically indistinguishable from 90. To reliably detect a true drop from 90 to 85 percent at the standard 95 percent confidence and 80 percent power, you need around 700 examples per version. It is the same binomial math you use to size an A/B test, so run your own scenario here.
The practical resolution: grow the set over time via the online loop, and treat small-set comparisons as smoke tests rather than verdicts. A 30-example set can tell you a prompt change broke JSON formatting, but it cannot distinguish 87 from 90.
The PM's actual job
You will not write the harness. Here is what is yours.
Define success in terms of the user's task. A model can score well on public benchmarks and still fail your users, because MMLU does not contain your refund policy. Write the rubric the judge uses, in plain language, anchored to what a user came to do: "the answer cites the correct plan tier," "the summary preserves every action item," "the reply never promises a timeline support cannot honor." If you cannot write that sentence, no eval can measure it for you.
Decide what pass rate ships, by severity. A flat threshold ("95 percent overall") can hide the distinction between a formatting miss and a hallucinated refund commitment. Tier the failures. A starting policy might block a release for any safety or policy violation, set a separate threshold for task correctness, and track style misses without blocking. The thresholds are product decisions about acceptable harm and should be agreed by the people responsible for the product, engineering, safety, and relevant domain expertise.
Connect eval scores to product metrics. An offline pass rate that climbs quarter after quarter while retention and task completion sit flat means the eval is measuring something users do not feel. Treat the eval suite itself as a hypothesis ("outputs that pass this rubric produce more completed tasks") and check it the way you would any other hypothesis. When the correlation breaks, rewrite the rubric.
Put one line in every AI feature PRD: "This ships when the golden set passes at X percent, with zero severity-1 failures, measured on dataset version Y." That sentence forces the threshold conversation before launch week, when it is still cheap to have.
FAQ
What is an AI eval? A repeatable test that scores an AI feature's outputs against a defined standard of quality. Inputs come from a fixed dataset, outputs get graded by code, by a judge model, or by humans, and the resulting pass rate gates releases the way a regression suite gates deterministic software.
Can I just A/B test my AI feature instead of building evals? Usually no. An A/B test measures whether a shipped version moves a product metric. An eval checks whether a version meets defined quality requirements before or alongside release, including failure modes that may be too rare for a live test to expose. Many AI features need both.
Do PMs write evals themselves? PMs often write or co-own the rubric and thresholds, while engineers build the harness and pipelines. The ownership model should include the people accountable for product outcomes, technical operation, and any domain or safety requirements.
How big should a golden dataset be? Start around 30 failure-sourced examples and grow it with every triaged production failure. Once you start comparing versions, size becomes a statistical question: distinguishing a 90 percent pass rate from 85 percent takes roughly 700 examples per version, which is why small sets are smoke tests rather than verdicts.
Is LLM-as-judge reliable enough to trust? Within limits. Published results show strong judges agreeing with human raters more than 80 percent of the time, comparable to human-to-human agreement2. Judges also carry documented biases toward longer answers, first-position answers, and their own model family. Calibrate against a hand-graded sample before trusting the number, and keep a weekly human review in the loop.
Sources
Footnotes
-
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023, arXiv:2306.05685). The over-80-percent agreement figure, matching human-to-human agreement, and the position, verbosity, and self-enhancement bias findings all come from this paper. ↩ ↩2 ↩3
-
AI Evals for Product Managers: A Beginner's Guide (Amplitude) ↩