Turning the Boss's 73 Review Comments into a Dataset: Creating a Digital Twin of a Reviewer

An employee at a large organization writes technical feasibility reports for AI projects, a task he had automated with an LLM agent that gathers project documents, extracts text, fills a structured JSON, and generates a DOCX from a template. After submitting nine reports, his manager returned them with 73 comments, many repetitive and related to missing sources, unsupported claims, and contradictions. He used Python to extract comments from the DOCX files, then had an LLM cluster the 73 comments into 16 patterns across 6 groups, including adherence to the brief, evidential support, competence boundaries, architectural choices, depth of questions, and stage-gating. He identified that almost all comments were epistemological — demanding clear status of claims, since the LLM produces uniformly confident text. He made three pipeline changes: marking each claim as [B] from brief, [E] expert estimate, or [D] assumption; adding a checklist as a separate pipeline step; and storing the checklist in the agent's persistent memory. The next four documents passed without repeated comments, and only new substantive issues remained. The author acknowledges the small sample size and that the effect partly comes from the source marking itself, not using them to bypass the reviewer but to formalize the reviewer's implicit quality standards.