6 Lessons for Creating Benchmarks from NeurIPS Tutorial
Google DeepMind
The authors of SWE-bench and GPQA at NeurIPS discussed key mistakes in creating benchmarks: tyranny of metrics, subjectivity of crowdsourcing, danger of using LLMs as evaluators, and the need for dynamic tests. Main advice — avoid saturated benchmarks, check multimodality, and publish results with confidence intervals.
At the NeurIPS tutorial "The Art of Benchmarking" featuring authors of SWE-bench, GPQA, and researchers from Google DeepMind, NYU, and Berkeley, key issues and pitfalls in benchmark creation were discussed. These include metric tyranny, where metrics have honey spots that allow models to "hack" the test, with Pearson correlation failing to detect these points. It was also noted that using LLMs as evaluation sources is dangerous: they correlate with humans only on the calibration set, and self-bias leads to inflated own results. The only correct approach is to generate examples on which the model fails. Among the six major mistakes made by benchmark creators: launching a benchmark with 70-80% accuracy on the top model (one should aim for 0-1%), lack of verification for true multimodality of the task, reliance on one's own theory of intelligence, failure to filter crowd spam (up to 25% random answers), concentration of work on a single crowd worker, and publishing results without p-values and confidence intervals. In the future, we need dynamic benchmarks, longitudinal studies of LLM impact on humans, calibration of model knowledge, and tasks that a cat can solve but GPT cannot.
Source: Хабр — Data Mining —
original