Preparing Documents for an Internal AI Knowledge Assistant
An internal AI knowledge assistant can only answer reliably if its source documents are authoritative, owned, current and filtered by the access rules of the company. Before building, choose the sources, assign an owner to each one, set a freshness rule, and map who may read which content. Test the assistant with a fixed set of evaluation questions and check that it cites the right source.
Which company sources should the assistant use?
Start with the questions staff ask most often, then identify the documents that answer them. Typical sources include standard operating procedures, product and price documents, HR policies, onboarding material, technical manuals and approved templates. Each source should have a clear purpose, so the assistant does not mix a draft proposal with an official policy.
Exclude sources that are personal, unfinished or unreliable, such as private notes, old copies saved in email attachments and documents marked as drafts. If two sources conflict, decide which one is authoritative and either retire the other or label it clearly.
Who owns each document and keeps it current?
Every source needs a named owner who is responsible for accuracy. The owner is usually the team that writes and approves the document, not the IT team that stores it. Without an owner, outdated content stays in the index and the assistant will answer from it with confidence.
Record the owner, the approval date and the review cycle as metadata for each document. This information helps you decide which sources to trust and gives the business a clear person to contact when an answer looks wrong.
How do you keep the knowledge base fresh?
Freshness rules depend on how often each source changes. A price list may need updating whenever it changes, while a technical manual may be reviewed once a year. Agree a refresh rule for each source type and decide how removed or superseded documents will be taken out of the index.
Version control matters as well. Keep only the current approved version active, and make sure the assistant returns the document date with each answer so that users can judge whether the information is recent enough for their purpose.
How should access control work for an AI assistant?
The assistant must never show a user content that the user could not open in the original system. Apply the existing permissions of each source, such as department folders, customer records or confidential HR files, and check them at the moment a question is answered, not only when the index is built.
Do not rely on the instructions given to the model to hide restricted content. If restricted text reaches the model, it can appear in an answer. Filter the search results by the user’s permissions before any text is passed on, and test this with users in different roles.
How should you prepare documents for indexing?
Retrieval works best when documents are clear and well structured. Use meaningful headings, avoid text inside images where possible, and convert scanned pages with reliable text recognition before indexing. Check how tables, forms and attachments are read, because they often lose meaning when converted to plain text.
Where your company works in Vietnamese, Japanese and English, check that each language is extracted and searched correctly. Add metadata such as language, product, department and effective date, so that the assistant can narrow its search to the right content.
What evaluation questions should you prepare?
Before launch, write a fixed set of evaluation questions with the answer you expect and the source that should be cited. Include questions that require combining two documents, questions about recent changes and questions the documents cannot answer. For the last group, the correct response is to say that the information is not available.
Add access tests as well. Ask the same question as users in different roles and confirm that restricted content is not shown. Run the same set after each change to the sources, the index or the model, and record which answers changed. Measure accuracy by checking answers against the expected sources, not by the fluency of the response.
Frequently asked questions
Does the assistant need every company document?
No. Start with the sources that answer the most common questions and that have a clear owner. Adding documents later is easier than cleaning up a large, unreviewed collection.
Do we need to train or fine-tune a model on our documents?
Not necessarily. A retrieval approach searches your approved documents at question time, so the content can be updated without retraining. The preparation work described here matters in either case.
How do we know the assistant is ready for staff use?
It is ready when it passes the agreed evaluation questions, including the access tests, and when the owners of the main sources have reviewed the results. Keep the evaluation set and repeat it after every major change.
