Observation·
AI evals are closer to product validation than testing
An AI eval tells you something useful only when it checks the answer against the job the customer needs done. Developers pick up eval tools as if they were unit testing: a fixed input, an expected output, a pass or a fail.
My research on where the market is going with AI evaluation shows the same split. About four times in five, the goal is to score answers for accuracy or to train a model. About one time in five, the check is tied to what a product has to do for its customers.
A unit test checks that code does what the developer wrote. The developer knows the right answer before the test runs. An agent's answer has no right answer written down until someone decides what the customer needed from it. An answer can be accurate and still useless, when it gets the facts right for the wrong problem.
So an eval needs a purpose before it needs a test set. Five things are set first: the job the agent should do, what it may do on its own, what counts as done, who answers when it is wrong, and what the work is worth against its cost. A score for accuracy covers one of the five.
That makes evals product work as much as engineering work. The test set comes from the customer's job, and a passing score means the product delivered what the customer pays for.
Reel transcript · 1:09
Your AI feature passes every test, and customers still don't use it. I see this every week.
How teams test AI today
Most teams test AI the way they test code: a fixed question, an expected answer, pass or fail. But an AI answer is only right if it helps the customer.
- A fixed question
- An expected answer
- Pass or fail
- But does it help the customer?
Correct. Still useless.
In my research, about four in five AI evaluations only check if the answer is correct. About one in five checks if it does the customer's job. An answer can be correct, and still useless.
Decide first
So before you test, decide what good looks like. Five things.
First, the job it does for the customer. Second, what it may do on its own. Third, when the work counts as done. Fourth, who answers when it's wrong. And fifth, what the work is worth. Correct answers cover just one of the five.
- The job it does
- What it may do alone
- When the work is done
- Who answers when it's wrong
- What it's worth
The takeaway
Testing AI is product work. Start from your customer's job, and a passing score finally means something.