Building a GenAI knowledge base that answers correctly: load, extract, test, publish

When an AI assistant gives a wrong answer about your refund policy, the instinct is to blame the model. Nine times out of ten the model did exactly what it was told; the problem is that the refund policy it was given was a 2023 PDF, a 2025 email thread that changed it, and a FAQ page that contradicts both.

A knowledge base for AI is not a folder of documents. It is a curated, tested set of answers with sources. Here is a method that produces one, in four steps that happen in order.

Step 1 — Load: everything, then less

Start wide. Policies, SOPs, FAQ pages, the terms and conditions, the price list, the timetable, the internal "how we handle X" documents that only exist in someone's inbox. PDFs, Word files, spreadsheets, web pages. Load all of it into the studio, tagged by source and date.

Then cut. For every topic, decide which document is the source of truth and retire the rest. Two documents on the same topic is not "more context" — it is a coin toss on every answer. The loading step ends with one source per topic and a list of the topics that have no source at all, which is its own useful finding.

Step 2 — Extract: from documents to answerable units

A policy document is written for a lawyer or a manager. A knowledge base is read by a model answering a customer at 11pm. The extraction step turns the first into the second.

Good extraction produces question-and-answer units, each pointing back to the passage it came from:

Q: How long does a refund take to arrive? A: Refunds are initiated the same day and reach your bank in 5–7 working days. UPI refunds are usually faster. Source: Refund policy §3, updated June 2026

Let the tool draft these, then have someone who actually handles refunds read them. They will catch the exception the policy forgot to mention — the one that generates half the tickets. Add it. That correction is the most valuable thing in the knowledge base, and it only comes from a human who does the job.

Step 3 — Test: ask it the questions customers actually ask

Before anything is published, test it against real questions. Not the ones you think customers ask — the ones they wrote. Export the last 200 support conversations, take the customer's first message from each, and run them through.

For each answer, look at three things:

• Is it right? Against the source, not against your memory of the policy.
• Is it cited? Every answer should show which unit it came from. An answer with no citation is a guess and should be blocked.
• What is the confidence? Low-confidence answers to common questions mean the extraction is thin on that topic. Go back to step 2.

Group the failures by topic. Typically three or four topics account for most of them — refunds with conditions, anything involving a date calculation, and whatever changed most recently. Fix those units, retest, repeat until the failure rate on real questions is one you would accept from a new hire.

Step 4 — Publish: with a scope and an owner

Publishing is not a switch; it is a decision about scope. A knowledge base can serve:

• the agent assistant inside the ticket (safest — a person reviews every answer before it goes out),
• the website chatbot (customers see answers directly, above a confidence threshold),
• the WhatsApp bot (same, on a channel where the customer is one tap from a screenshot).

Publish to the assistant first. Watch the correction rate. Widen to the chatbot when agents stop rewriting. Widen to WhatsApp last.

Every published unit needs an owner and a review date. Policies change; the knowledge base should have a "reviewed by" column that a manager can sort by, and a rule that anything older than the last policy change gets re-tested.

What good looks like after a month

• One source of truth per topic, and a short list of topics with none.
• A few hundred Q&A units, each with a citation, most with a real-question test behind them.
• A confidence threshold set on evidence rather than on a vendor's default.
• A correction rate from the agent assistant that is falling week over week.
• A published scope that is deliberately narrower than what the tool could do.

That last point is the one teams get wrong most. The knowledge base can answer everything about your business; it should answer the things it has been tested on. Narrow and correct beats wide and occasionally embarrassing.

Request a demo of Conversational AI →

The question that reveals the vendor

Ask any GenAI knowledge tool one question: show me the test step. If the product goes straight from "upload your documents" to "your bot is live", the testing is going to happen on your customers. If there is a place to load real questions, see the answers with their sources and confidence, and fix the units that failed — before anything is published — you are looking at a knowledge base built by people who have run one.