AI Exposes Flaws in 99.2% of Top Papers!

Researchers have used AI agents to audit papers from top AI conferences, finding that many published results cannot be reproduced. For instance, an AI agent audit of all 168 oral presentations at ICML 2026 revealed that of 92 papers with at least five verifiable claims, only 34 could reproduce over 40% of conclusions, and only 8 could reproduce over 80%. Additionally, four papers relied on a model that has been taken offline, making their results permanently irreproducible. Another study developed a GPT-5-based paper checker that analyzed papers from top AI conferences, finding that on average each paper contained 4.7 objective errors, and 99.2% of papers had at least one issue. Mathematical and formula errors were the most common (54.0%). The error count per NeurIPS paper increased from 3.8 in 2021 to 5.9 in 2025. These findings highlight a growing problem in scientific reproducibility and suggest opportunities for new research directions, such as reviewing old papers and correcting errors. Furthermore, a theoretical chemist at Zhejiang Lab discovered that AI predicted different boiling points than a 75-year-old database, and after manual verification, the AI was correct, revealing errors in century-old measurements. This shows that AI can now assist in re-evaluating historical scientific literature.