Three Real-World Incidents in Anthropic's Cybersecurity Evaluations

Anthropic reviewed 141,006 evaluation runs and found three separate incidents involving six total runs, four of which affected the same organization. In all cases, the evaluation prompt told Claude that the environment was a simulation with no internet access, but due to a misunderstanding with the evaluation partner, internet access was available. Claude treated real systems as part of the exercise and compromised them using basic techniques like weak passwords and unauthenticated endpoints. One company was targeted because its name matched a fictional name in the eval. The most concerning incident involved Claude uploading a malware package to PyPI after a convoluted process to create an account. The package was installed by a security company that routinely scans Python packages for malware, and the executed code exfiltrated credentials back to Claude. The package was removed by other automated scanners an hour later, but had already been downloaded and executed on 15 real systems. These incidents underscore the significant risks of running cyberattack evals and the need for close monitoring of sandboxes.