AI automation buying guides
AI Knowledge Base Setup: A Buyer Acceptance Checklist
An AI knowledge base should be accepted as an operating system for evidence, not as a polished chatbot demo. Buyers need to prove that the right sources are indexed, permissions survive retrieval, changes appear on time, answers cite usable evidence and the system knows when not to answer.
Prepared with AI assistance and checked against the linked sources. Numerical comparisons are illustrative unless expressly sourced.
Accept the knowledge base only after a source inventory, permission tests, freshness and deletion checks, retrieval inspection, grounded-answer tests and a named operating owner all pass. A fluent answer without traceable evidence is not an acceptance result.
Accept repeatable behavior, not the best demo answer
A useful demonstration proves that one prepared question can produce a convincing answer. Acceptance must show that the deployed system behaves appropriately across ordinary, ambiguous, sensitive, outdated and unanswerable questions.
Begin with the decision the knowledge base will support. A public product FAQ, an internal policy assistant and an agent guiding support staff have different requirements. Write the intended users, channels, topics, prohibited topics and allowed actions before discussing models or databases.
Retrieval-augmented generation, commonly called RAG, finds information from connected sources and supplies retrieved material to a model. AWS describes searching a knowledge base and generating a response, and documents citations as a way to reference the original source. [1] That architecture can help ground an answer, but it does not prove the material is current, permitted or sufficient.
Define acceptance in observable terms: which source should be found, which source must never be exposed, how recent an update must be, what evidence must be displayed and when the assistant should decline or escalate.
Build an approved source register before ingestion
Do not start by connecting every drive, website and workspace. Create a register of approved source collections. For each collection, name the owner, location, content type, audience, confidentiality level, update method, refresh interval and removal process.
Review documents for ownership and fitness. Drafts, duplicates, superseded policies, image-only scans and pages without effective dates can create retrieval problems. Decide which version is authoritative when several documents describe the same rule. Preserve metadata that supports filtering and useful citations.
| Register field | Acceptance evidence |
|---|---|
| Authority | Named owner confirms the collection and current version |
| Audience | Public, team, role or restricted group is explicit |
| Lifecycle | Create, update, archive and delete paths are documented |
| Structure | Headings, tables and attachments survive ingestion as intended |
| Coverage | Known gaps and excluded collections are visible |
If a proposal says “all company knowledge” without listing repositories and owners, comparison and acceptance will be unreliable. Ask what will not be ingested as clearly as what will.
Inspect retrieval before judging the final answer
Test retrieval independently. For each question, inspect returned passages, source identifiers, order and metadata filters before reading the generated answer. A correct answer from irrelevant passages may be accidental.
Chunk size, overlap, search method, filters, result count and reranking can change what reaches the model. AWS documents retrieval and reranking as configurable parts of a knowledge-base query. [1] These are choices to test, not guarantees of relevance.
Create variants using customer language, abbreviations, misspellings and multi-part requests. Include near-duplicate documents and conflicting versions. Record whether the expected passage appears, whether an obsolete passage outranks it and whether the citation points to a section a user can inspect.
For each failure, record the question, expected source, retrieved source, relevant passage, filter and proposed fix. Content, metadata and search-configuration fixes are different. Do not tune only the answer prompt when the evidence never reached the model.
For high-stakes topics, require a supporting passage or route the user to an authorized human. Retrieval is not professional verification.
Test freshness, replacement and deletion as separate events
“Connected” does not mean current. Define a freshness target for each source class, then measure the time from an approved source change to the changed passage becoming retrievable. A product specification may tolerate a scheduled refresh, while an incident instruction may need a faster controlled update path.
Run three lifecycle tests. First, update a known sentence and confirm the new version appears within the agreed window. Second, replace a document and confirm the old version no longer competes with it. Third, delete or revoke a document and confirm it is no longer retrieved, cited or available through a cached path.
Microsoft notes that query-time authorization relies on permission metadata already synchronized into the index and that source permission changes are reflected only after the relevant synchronization mechanism runs. [2] The same operational lesson applies more broadly: test the actual synchronization path rather than assuming the source system and index change together.
Ask for failed-sync visibility, retry behavior, last-success timestamps and an owner for stalled connectors. Keep an emergency removal procedure for materially wrong or sensitive content. Acceptance should include evidence that the team can identify what was indexed, when it changed and how to remove it, without rebuilding the whole service blindly.
Prove permissions at retrieval time
A login screen is not enough. The system must prevent an unauthorized document from entering the retrieved context. Hiding a citation or redacting the final sentence after retrieval leaves a weaker control boundary and can still expose information through summaries, prompts, logs or tool traces.
Map the source identity model to the search index. Microsoft documents document-level approaches that carry permission metadata into the index and apply user or group information when querying. It also warns that several native ACL and label features are preview capabilities, so buyers should verify the production status and limitations of the exact service and API they plan to use. [2]
Create positive and negative tests for at least three identities: an authorized user, an unauthorized user and a user whose access was just revoked. Test direct questions, paraphrases and questions that combine allowed and restricted material. Include shared folders, inherited permissions and a document moved between groups.
The acceptance record should show the caller identity, expected accessible collections, retrieved document identifiers and result. Do not put real secrets or sensitive personal data into a test set. Use controlled fixtures that prove the boundary safely. If the platform cannot preserve source permissions, reduce the scope to a separately approved collection or choose an architecture that can enforce the required boundary.
Define a grounded-answer contract
Write down what every acceptable answer must do. A practical contract can require: answer only from retrieved approved material, cite the supporting source and section, distinguish a direct statement from an inference, state material uncertainty, refuse when evidence is insufficient and route defined topics to a human.
Citations must be usable, not decorative. Open them during testing. Confirm that the cited passage supports the claim, the user is allowed to view it and the label helps distinguish similarly named documents. A response can contain a valid link and still misrepresent the source.
Include adversarial cases such as a user asking the assistant to ignore policy, reveal restricted content or invent an answer when the source is silent. NIST's Generative AI Profile is voluntary risk-management guidance spanning governance, mapping, measurement and management across the AI lifecycle. It specifically frames pre-deployment testing as part of managing generative AI risk. [3] Use that principle proportionately to the business impact, rather than treating a one-time launch review as permanent assurance.
Define escalation output too. “I don't know” is useful only if the user learns what to do next. State which channel, queue or owner receives unresolved questions and what context is passed. Avoid allowing the assistant to improvise legal, medical, financial or safety conclusions merely because a related document was retrieved.
Create a representative test set and scorecard
Build the test set from real task patterns, anonymized where necessary, before final tuning. Include easy facts, paraphrases, comparisons, multi-step questions, conflicting sources, no-answer cases, restricted content, fresh updates and deleted content. Give each item an expected source, expected behavior and severity if it fails.
OpenAI's evaluation guidance describes test data that represents the work a prompt is expected to handle, with human-provided reference outputs and explicit testing criteria. [4] For a knowledge base, do not reduce everything to exact wording. Score several dimensions independently.
| Dimension | What to record | Launch blocker example |
|---|---|---|
| Retrieval | Expected evidence appears in inspected results | Authoritative source is consistently absent |
| Grounding | Material claims are supported by cited passages | Answer adds an unsupported policy rule |
| Permissions | Only authorized fixtures are retrieved | Restricted fixture appears for the wrong identity |
| Abstention | System declines when evidence is missing | Confident answer is invented |
| Freshness | Update and removal meet the agreed window | Revoked content remains retrievable |
Set thresholds by risk and test category, not only one blended percentage. Any permission leak can block launch even if hundreds of simple FAQ answers pass. Keep the test set as a regression suite after changes to sources, models, prompts, retrieval settings or connectors.
Worked example: accept a support-policy assistant
Consider an invented internal assistant for a software support team. Its approved sources are the current support policy, plan matrix and escalation runbook. The buyer sets a four-hour synchronization target and requires source title, section and effective date in every policy answer.
The acceptance set contains 60 questions: 25 common questions, 10 paraphrases, 5 conflicting-version cases, 5 no-answer cases, 5 restricted-document attempts, 5 permission-change checks and 5 update or deletion checks. These numbers are educational, not a universal benchmark.
During testing, 58 answers sound reasonable. However, an unauthorized identity retrieves a restricted renewal exception, and one deleted policy remains available after four hours. The system fails acceptance. The overall count does not average away those control failures.
The supplier corrects permission metadata propagation and a stalled deletion job, then reruns the regression set. The buyer records the configuration, evidence, exceptions and owner. Only then is the deployment considered for a limited monitored pilot.
Lower-severity answer issues may remain documented during a pilot, but permission exposure, unsafe escalation or materially stale controlled content should not be waived through a polished demonstration.
Compare proposals by operating scope and evidence
Ask every supplier to price the same scope: named repositories, document volume and formats, identity integration, refresh targets, channels, test-set size, environments, monitoring, support period and handover. Separate one-time discovery and ingestion from recurring infrastructure, model use, connector maintenance, evaluation and content operations.
A bounded AI knowledge-base training scope should leave behind the source register, ingestion rules, permission design, test set, failure log, release configuration and runbook. If the system will answer customers, review the related chatbot scope. If it can change records or call tools, the chatbot versus agent buying guide explains why permissions and action approval need a wider evaluation.
Do not buy RAG only because the source volume is large. A curated search page or maintained FAQ may be cheaper and safer when questions are few, rules are stable or users need exact document navigation rather than generated summaries. Do not connect restricted repositories until identity and deletion behavior can be tested.
TechLithic offers AI knowledge-base implementation and related automation services. That commercial relationship does not change the acceptance standard. Use this checklist with TechLithic or any other provider, require evidence for each gate and define who owns the system after handover.
Run the six-gate launch checklist
Use the infographic as a release path. A failed gate returns the system to remediation rather than being hidden inside an average score.
- Approved sources: owners, audiences, versions, exclusions and known gaps are recorded.
- Sync and deletion: updates, replacements, removals, failures and last-success times behave within agreed targets.
- Permission-aware retrieval: positive, negative and revoked-access fixtures pass for the deployed identity path.
- Relevant evidence: expected passages appear for representative wording, conflicts and metadata filters.
- Grounded response: claims are supported, citations open correctly, uncertainty is visible and insufficient evidence triggers abstention or escalation.
- Operations: a named owner can monitor failures, run regressions, manage incidents and approve source changes.
Archive the acceptance set, results, system configuration and unresolved issues with the release date. Define which changes require a full rerun and which need a focused regression. Track user-reported wrong answers back to retrieval, source, permission or generation causes.
For a wider deployment combining a knowledge base, shared inbox and human handoff, compare the acceptance evidence against the intended customer-support transformation scope. Launch only the smallest audience and content set that has passed. Expand after observed usage confirms that the controls and operating ownership work in practice.
Frequently asked questions
Is a successful chatbot demo enough to accept an AI knowledge base?
No. Acceptance should cover representative and adverse questions, source retrieval, permissions, freshness, deletion, citations, abstention and operating ownership in the deployed environment.
Should every company document be added to the knowledge base?
No. Start with approved collections that have clear owners, audiences and lifecycle rules. Exclude duplicate, obsolete, restricted or ungoverned material until it can be handled safely.
What is the most important launch-blocking test?
There is no universal single test, but unauthorized retrieval, unsafe escalation and failure to remove materially sensitive or wrong content are reasonable blockers regardless of an average quality score.
How often should the test set be rerun?
Run it before launch and after material changes to sources, permissions, connectors, retrieval settings, prompts or models. Add focused regression cases whenever a real failure is found.
Sources and research notes
Sources checked on 2026-10-21. Pricing and policy documents can change; verify current terms before approving expenditure.
- Amazon Bedrock documentation: Knowledge Bases for retrieval-augmented generation
Primary provider documentation on retrieving source information, generated answers, citations and reranking. Product capabilities vary by architecture and configuration.
- Microsoft Learn: Document-level access control in Azure AI Search
Primary provider documentation on carrying document permissions into retrieval, query-time enforcement, synchronization behavior and preview limitations.
- NIST AI 600-1: Generative AI Profile
Voluntary cross-sector risk-management guidance for generative AI across governance, mapping, measurement and management, including pre-deployment testing considerations.
- OpenAI API documentation: Working with evals
Primary provider documentation illustrating representative test data, human-provided reference outputs and explicit evaluation criteria.
Official AWS, Microsoft, NIST and OpenAI sources were checked on October 21, 2026. Provider documentation is used to explain retrieval, permissions, synchronization, citations and evaluation patterns, not to recommend one vendor. The six-gate framework, scorecard and 60-question example are original educational devices, not measured client results or universal compliance thresholds. Buyers should set controls to their use case, data and applicable obligations.