AI Agents: Parallelize Work Without Parallelizing Errors

Researchers have developed a benchmark named Horizon and analyzed more than 3,100 agent trajectories in various environments, including the web, operating systems, databases, and embodied settings. The experiments involved agents using models from OpenAI and Anthropic. The findings reveal that performance does not degrade gradually as tasks lengthen but can remain stable and then abruptly collapse beyond a certain threshold. Seven major categories of failures were identified, grouped into two families: 72.5% of failures associated with process-related risks and 27.5% with agent design risks. Anthropic highlights a phenomenon called 'context pollution,' where accumulated intermediate information clutters the agent's context and impairs judgment. The study suggests that splitting a task into independent subtasks handled by separate agents works, whereas splitting phases like design, implementation, and testing, which share too much context, leads to information loss akin to a game of telephone. For executives, four questions are key: who made the decision, based on what information, at what cost, and whether a single failed step can be replayed independently. Without replayability, incidents restart everything, and improper splitting can cost 3 to 10 times more than a single agent.