Researchers from PSU and Duke Introduce Automated Failure Attribution for Multi-Agent Systems
Google DeepMind
Meta
OpenAI
DeepSeek
Researchers from Penn State University and Duke University, in collaboration with Google DeepMind and others, have formalized the problem of automated failure attribution in LLM multi-agent systems. They created the first benchmark dataset, Who&When, and evaluated several methods, finding that even the best approaches achieve only 53.5% accuracy in identifying the responsible agent.
A team from Penn State University and Duke University, along with researchers from Google DeepMind, University of Washington, Meta, NTU, and Oregon State University, has defined a new research problem: automated failure attribution in LLM-driven multi-agent systems. They built the first benchmark dataset, Who&When, containing failure logs from 127 multi-agent systems with human annotations for the responsible agent, the decisive error step, and a natural language explanation. The researchers developed and evaluated three attribution methods: All-at-Once, Step-by-Step, and Binary Search. Experiments using GPT-4o showed that the best single method achieved only 53.5% accuracy in identifying the responsible agent and 14.2% in pinpointing the error step. Hybrid approaches improved performance but at higher computational cost. The paper has been accepted as a Spotlight presentation at ICML 2025, and the code and dataset are open-sourced.
- Сокращения
- LLM = Large Language Model — большая языковая модель
- ICML = International Conference on Machine Learning — Международная конференция по машинному обучению
Source: Synced —
original