Autonomous AI Agent Monitoring Production Logs for 4 Months: 113 Alerts, 2 Hallucinations, and a Fix in 3.5 Hours

The AI agent monitors production logs for the Creatorry platform, which generates music, photos, and videos using AI. The system comprises 18 services, logging approximately 300,000 lines daily, of which around 12,500 are errors or warnings. The agent runs every 30 minutes, querying Loki for recent errors, and only sends alerts to Telegram if it detects critical issues or a significant spike in error rates. Over 84 days, the agent issued 113 alerts, with 69 errors, 41 criticals, and three early entries missing severity labels. The most impactful alert detected a critical error early on June 1, leading to a fix that was merged within 3.5 hours. The agent also caught a wave of 503 errors that revealed a fallback provider was not being used. However, the agent hallucinated twice: once by inventing a bot token and another time by reporting inflated error counts. To counter this, the team implemented a verification step where a separate model recounts the errors using different methods, and they set a strict ratio threshold of 2.0 to reject unverified alerts. The agent initially used Claude Opus but switched back to MiniMax-M3 due to Opus verbosity and high cost. The agent consumes about 435 million tokens per month on a $10 subscription, with most tokens coming from cached reads. The team emphasizes prompt injection safety, as logs are untrusted input, but the agent has read-only access and sends to their own Telegram channel only. In total, the system has been running for 135 days, with about a month of downtime due to various issues.