Building an AI Agent Knowledge Base and Testing It on My Own Medical Tests
A backend developer at Iskra tested his company's AI agent knowledge base by uploading his own medical test results, because he could immediately spot errors in answers about his own lab data. The system uses a file-system-like approach with document mirrors, grep, and graph links instead of classic RAG, which works well but has pitfalls like the agent forgetting to access the knowledge base.
The author, a backend developer at Iskra, built a knowledge base for an AI agent and tested it with his own medical checkup results, which he knew by heart, making errors obvious. The system works like a prepared file system rather than a search index: each file gets a mirrored .md with a YAML header, and an agent navigates via ls, grep, read, and link-following. This approach outperforms classic RAG for multi-document answers and source citation, as the agent reads entire documents instead of chunks. However, the agent sometimes forgets to access the knowledge base, as seen when it claimed not to see the attached files, then on a retry read all seven and provided a correct summary. Additional issues include quirky file renaming (some titles in Latin), paraphrase inaccuracies in citations, and higher token cost—about 10–15 tool calls per complex query. The author also notes that the system is not foolproof: the model can cite sources but still hallucinate, so a server-side source list is crucial.
- Abbreviations
- RAG = Retrieval-Augmented Generation — генерация с дополнением поиском
- BM25 = Best Match 25 — алгоритм ранжирования BM25
Source: Habr — хаб NLP —
original