Quality
Test your AI on real cases before customers see its work
An AI agent that sounds sure of itself can still be wrong. The fix is a test built from your own past work. It takes an afternoon to start and tells you more than any demo.
Foundation
Build the test set first
How do you test an AI agent?
Run it on real past cases where you already know the right answer, then compare. Score whether the reply is supported by your documents, whether it does the job, whether it ever does something it must not, and how long and how much it takes. Write down the pass mark before you run the test.
- Size
- Start with 30 to 50 cases. Add one every time you find a new kind of mistake.
- Source
- Real questions and requests from the past year, not ones you invent. Invented questions test your assumptions.
- Mix
- Include easy cases, awkward ones, and a few where the right answer is that you do not know or the owner must decide.
- Right answers
- The person who does the work writes each correct answer and notes the document it comes from.
- Disagreement
- If two staff write different answers, your own process is unclear. Settle that first.
- Privacy
- Real cases hold real customer details. Store the test set the way you store customer records.
Four scores
What to score
- Is it supported by your documents?
- Check each claim in the reply against the document it cites. One invented price makes the whole reply wrong.
- Does it do the job?
- Could a staff member send it with small edits, or would they rewrite it?
- Does it ever do something it must not?
- Show another customer's details, promise a credit you do not offer, follow instructions hidden in an incoming email. A single failure is a fail, not a score to average.
- How long and how much?
- Time to a usable reply, and cost per reply that staff used.
Knowing its limits
Does it say "I don't know" at the right time?
An agent that answers everything is less trustworthy than one that knows when to stop. Include cases where the right move is to hand the question to a person. Then score both mistakes: answering when it should have asked, and handing over when your documents had the answer.
A good hand-over says what the agent searched, why it could not settle the question and what the person should do next. "I cannot answer that" is a poor hand-over.
Two languages
Test in both
If you serve customers in French and English, write the French test cases yourself rather than translating the English ones. Translated questions carry English phrasing and hide the failures you care about. Score each language on its own, because a blended number hides a service that works in one and not the other.
After launch
Run it again after every change
Run the whole test set again when the AI model changes, when your instructions change, when the tools it can use change and when your documents change. A new price list can break answers as surely as a new model.
Agree the pass mark in writing before launch. Do not lower it because a launch date is close.
Honest limits
What a test cannot tell you
- Whether your test set matches what customers ask. A kind of question you missed stays hidden behind a good score.
- Whether staff keep reading drafts closely once the drafts look reliable. Watch how often they edit.
- Whether the job should exist at all.
- Whether it is still true next month. Models and documents change, so test on a schedule.
Testing is part of Consulting: each skill we build has an acceptance test that you sign, and the guarantee rests on it.
Next step
Start with ten cases this week
Pull ten real customer questions and the answers you gave. That is the start of your test set.