AI Evals for PMs: How to Know Your AI Feature Works

Running the same prompt through the same model twice can produce different answers. An eval tests those outputs repeatedly against defined quality criteria. Engineers build the test harness, while the PM defines what the output needs to do for the user.

Why your A/B testing instincts don't transfer directly

If you have internalized A/B testing, you already hold most of the statistical intuition evals require. Sampling error, confidence intervals, the temptation to stop early when the numbers look good: all of it carries over. What changes is the thing being measured.

A conversion funnel has one question per step: did the user click or not. An AI answer has several questions at once. Is it factually correct? Is the tone right for the audience? Did it arrive within the latency budget? Did it stay inside safety policy? A single output can pass two of those and fail the others, and a single conversion metric collapses the distinction. Your checkout test never had to decide whether a purchase was "rude."

The second shift is that there is no fixed variant. In a classic test, variant B is the same pixels for every user. In an AI feature, "variant B" is a prompt-and-model combination that produces a different output for every input, and sometimes a different output for the same input. You are comparing two probability distributions of behavior rather than two fixed screens, and you need enough graded samples from each to say anything about the difference.

A/B tests assess whether a change affects retention or task completion in production. Evals help determine whether a version is ready to show to users in the first place.

Three ways to grade an output

Every eval reduces to the same loop: take an input, generate an output, grade it. The grading method is the design decision, and there are three.

Code-based checks provide a baseline. They are efficient enough to run on every commit; Hamel Husain notes that teams often under-invest in this layer before adopting more complex methods¹.

LLM-as-judge is used for tasks that code cannot validate. Research by Zheng et al. shows that GPT-4 judgments agree with human raters more than 80 percent of the time on chat-quality comparisons, a rate similar to human-to-human agreement². The same paper documents the judge's weaknesses: position bias (preferring the first answer shown), verbosity bias (preferring longer answers), and self-preference when grading outputs from its own model family².

Human review is the most expensive method but provides the ground truth used to calibrate other methods and discover new failure modes. Most working setups use all three in a stack: code checks on every commit, judge runs on every release candidate, humans on a weekly sample and on every incident.

Offline gates the release, online finds what you missed

An offline eval uses a fixed dataset, a scoring method, and a pass rate to assess a change before release. It serves as a release check, similar to a regression suite, without exposing users to the new version.

An online eval monitors live traffic after release. It can sample outputs for judge review and track thumbs-up and thumbs-down feedback, escalation rates, and the edits between an AI draft and what the user keeps. These checks can reveal failures missing from the offline dataset. Most production teams run both, and sample a slice of live traffic (5 to 15 percent is typical) for the judge to keep grading costs sane³.

When an online eval reveals a failure, add the example to the offline dataset as you fix it. Otherwise, a later prompt change can reintroduce the problem without the release checks detecting it.

If your product chains multiple model calls, the offline story gets more granular. Step-level and handoff evals for agent pipelines are covered in building agentic products.

Building a golden dataset

A golden dataset is the fixed set of inputs (and expected behaviors) your offline evals run against. Two decisions matter: what goes in it, and how big it is. Most of the attention goes to size, but the contents decide whether the set catches anything.

Start from real failures. Support tickets where the AI answer was wrong. Thumbs-down events with the output attached. The screenshots people post in your internal Slack channel. Synthetic happy-path examples confirm the model can do what it already does. Failure-sourced examples test the boundary, and the boundary is where the next regression lives. Practitioners keep landing on the same advice: define evals from the failure states you find, not from a list you brainstorm in a room, or every eval passes while customers stay unhappy⁴.

Coverage comes first, but size is not optional, because your sample-size intuition still applies. Suppose your assistant passes 90 percent of a 100-example set, an engineer changes the prompt, and the rate reads 87 percent. Regression or noise? On 100 examples, the 95 percent confidence interval around a 90 percent pass rate is roughly plus or minus 6 points, so 87 is statistically indistinguishable from 90. To reliably detect a true drop from 90 to 85 percent at the standard 95 percent confidence and 80 percent power, you need around 700 examples per version. It is the same binomial math you use to size an A/B test, so run your own scenario here.

The practical resolution: grow the set over time via the online loop, and treat small-set comparisons as smoke tests rather than verdicts. A 30-example set can tell you a prompt change broke JSON formatting, but it cannot distinguish 87 from 90.

The PM's actual job

You will not write the harness. Here is what is yours.

Define success in terms of the user's task. A model can score well on public benchmarks and still fail your users, because MMLU does not contain your refund policy. Write the rubric the judge uses, in plain language, anchored to what a user came to do: "the answer cites the correct plan tier," "the summary preserves every action item," "the reply never promises a timeline support cannot honor." If you cannot write that sentence, no eval can measure it for you.

Decide what pass rate ships, by severity. A flat threshold ("95 percent overall") hides the distinction between a formatting miss and a hallucinated refund commitment. Tier the failures. A workable starting policy: safety and policy violations gate at 100 percent (one failure blocks the release), task-correctness failures gate at a negotiated rate, style misses get tracked but never block. The thresholds are product decisions about acceptable harm, which is exactly why an engineer should not be setting them alone.

Connect eval scores to product metrics. An offline pass rate that climbs quarter after quarter while retention and task completion sit flat means the eval is measuring something users do not feel. Treat the eval suite itself as a hypothesis ("outputs that pass this rubric produce more completed tasks") and check it the way you would any other hypothesis. When the correlation breaks, rewrite the rubric.

FAQ

What is an AI eval? A repeatable test that scores an AI feature's outputs against a defined standard of quality. Inputs come from a fixed dataset, outputs get graded by code, by a judge model, or by humans, and the resulting pass rate gates releases the way a regression suite gates deterministic software.

Can I just A/B test my AI feature instead of building evals? No, and you also cannot replace A/B tests with evals. An A/B test measures whether a shipped version moves a product metric. An eval measures whether a version is safe and correct enough to ship at all, across failure modes too rare for any live test to surface. You need both, in that order.

Do PMs write evals themselves? PMs write the rubric and own the thresholds; engineers build the harness and the pipelines. The dividing line: anything that defines "good" or decides "ship" is PM work. Anything that executes the grading at scale is engineering work.

How big should a golden dataset be? Start around 30 failure-sourced examples and grow it with every triaged production failure. Once you start comparing versions, size becomes a statistical question: distinguishing a 90 percent pass rate from 85 percent takes roughly 700 examples per version, which is why small sets are smoke tests rather than verdicts.

Is LLM-as-judge reliable enough to trust? Within limits. Published results show strong judges agreeing with human raters more than 80 percent of the time, comparable to human-to-human agreement². Judges also carry documented biases toward longer answers, first-position answers, and their own model family. Calibrate against a hand-graded sample before trusting the number, and keep a weekly human review in the loop.

Sources

Footnotes

1. Your AI Product Needs Evals (Hamel Husain) ↩

2. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023, arXiv:2306.05685). The over-80-percent agreement figure, matching human-to-human agreement, and the position, verbosity, and self-enhancement bias findings all come from this paper. ↩ ↩² ↩³

3. AI Evals for Product Managers: A Beginner's Guide (Amplitude) ↩

4. AI Evals for Product Managers (Productboard) ↩