Building an Honest RAG Eval Set: How to Collect the First 100-300 Cases Without Deceiving Yourself

The author argues that public benchmarks like MTEB and BEIR do not guarantee good retrieval on domain-specific tasks, and synthetic data can lead to overly neat formulations. Instead, they recommend building a manual eval set of 100-300 cases, with a practical start of 120 cases sourced from logs, documentation, expert interviews, and synthetic candidates after review. Each case should include fields such as query, scenario, expected document IDs, gold evidence spans, expected facts, answerability, and why it matters. The set should be divided into 6-8 slices covering exact entities, paraphrasing, long sections, tables, versions, no-answer cases, poor phrasing, and sensitive scenarios. Retrieval and answer generation should be evaluated separately, with retrieval metrics including Recall@k, MRR@k, and nDCG@k, and answer metrics including groundedness, completeness, answer relevance, and correct abstention. The author emphasizes that for no-answer slices, standard recall is meaningless and suggests a separate release gate. They also warn against using training data in the eval set and suggest validating LLM-as-a-judge on a manual subset.