Most AI chatbots look good in a demo. Someone asks five sensible questions, the bot answers well, and everyone agrees it is ready. Then real customers arrive, ask the sixth question, and the bot confidently gives a wrong answer about a refund policy or a price.
The difference between a demo and a product is testing. Not “we tried it and it seemed fine”, but a fixed set of questions with known correct answers, run every time anything changes. This is how we test AI chatbots before launch, and why it matters more than the choice of model.
Why a chatbot needs a different kind of testing
Ordinary software gives the same output for the same input, so a test either passes or fails. A chatbot built on a large language model does not work like that. The same question can be answered in different words each time, and a small change, a new prompt, a newer model, an updated document, can quietly change answers that used to be right.
So the question is not “does it work?” but “how often is it right, on the questions that matter, and does that stay true after every change?” That needs an evaluation set.
Step 1: build the evaluation set from real questions
An evaluation set is a list of questions, each with the correct answer and, where the bot uses your documents, the source that answer comes from. The best questions come from real customers: support tickets, live chat transcripts, emails, and the questions your sales team hears. Invented questions are too tidy.
We make sure the set covers five kinds of question:
- Answerable questions. The answer is in your documents. The bot should give it, correctly, and point to the source.
- Unanswerable questions. The answer is not in your documents. The bot should say it does not know and offer a person, not make something up. This is the category that catches most problems.
- Personal and account questions. “Where is my order?” or “Why was I charged twice?” The bot should either look it up through a secure integration or hand over, never guess.
- Out of scope and tricky questions. Questions about competitors, legal advice, attempts to make the bot ignore its instructions, rude or confusing messages.
- Conversations, not single questions. Follow-ups such as “and what about for Germany?” that only make sense with the earlier message.
For a typical customer support bot, a few hundred questions is a good start. The set grows after launch, as described below.
Step 2: test the search before the answer
Most business chatbots answer from your own documents: help articles, policies, product data. Before the model writes anything, a search step finds the passages it should use. If that search returns the wrong passage, even the best model will give a wrong answer.
So we test the search on its own first. For each answerable question we know which document holds the answer, and we check whether it is among the passages returned. When it is not, the fix is usually in how documents are split up, labelled or kept current, not in the prompt. For example, a returns policy that differs per country needs each country’s rules kept apart, or the bot will mix them.
Step 3: grade the answers
Each answer is checked against a short list of rules:
- Are the facts the same as in the correct answer?
- Does it cite or link the right source?
- Does it add anything that is not in the source?
- For unanswerable questions: did it decline and offer a person?
- Is the tone right for your brand?
Grading hundreds of answers by hand after every change is not realistic, so most of it is automated: a second model compares each answer with the correct one using the rules above. Automated grading is not perfect, so a person reviews a sample of the grades, and every answer graded as failed. The aim is a score you can trust and compare from one version to the next.
Step 4: set a bar before launch, and run the set on every change
Agree in advance what “ready” means with the people who own the answers, usually customer service and sometimes legal. For example: correct answers on the answerable questions above an agreed level, no confident wrong answers on the unanswerable ones, and every personal question handed over. Then the launch decision is based on results, not on how the demo felt.
After that, the evaluation set runs automatically whenever something changes: a prompt, the model or its version, the search settings, or the documents. If a change makes answers worse, it is caught before customers see it. This is the single most useful habit in running an AI chatbot.
Step 5: keep learning after launch
Real customers will always find questions nobody thought of. After launch we watch:
- Conversations that were handed to a person, and why.
- Answers that customers marked as unhelpful.
- Questions the bot could not answer because the information does not exist yet. These are often a list of gaps in your help content.
Every real failure becomes a new question in the evaluation set, with its correct answer. Over time the set becomes a precise description of what your customers ask and what a good answer looks like.
What this means if you are buying a chatbot
When you talk to anyone about building an AI chatbot, ask three questions: How will you test it before launch? What happens when it does not know the answer? How will we know if a change makes it worse? A good partner has clear answers to all three. If the answer is “the model is very good”, be careful.
We build AI chatbots and AI agents for customer support, sales and internal knowledge, with an evaluation set agreed with you before launch. If you want a second opinion on a chatbot you already run, we can build an evaluation set for it and show you where it goes wrong. See also our AI and machine learning development services.