Supabase Open Sources Evals: Benchmarking AI Coding Agents on Real Tasks

Supabase has open sourced Supabase Evals, a benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks such as building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, then scores the result. The benchmark powers a public leaderboard at supabase.com/evals and an internal regression suite monitored daily. The framework is available under Apache-2.0 and runs locally via pnpm. Scenarios cover dimensions like products (database, auth, storage) and topics (RLS, security), and each runs against a real containerized Supabase stack. Scoring combines deterministic checks with LLM-as-a-judge, and agents get one retry. Key findings show that top models like Opus 5 and Kimi K3 scored 100% unaided, while skills helped smaller models close the gap. Weaknesses include hand-written migrations instead of declarative schemas, manual auth verification, and inconsistent docs usage among agents.