Runtime-Agnostic AI Workflows: A Pattern Balancing Production Stability and Rapid Evaluation Iteration

The article, sourced from InfoQ China, describes a pattern developed at Brex's AI workflow platform to resolve the trade-off between production stability and fast evaluation iteration. AI workflows involve steps that call large language models (LLMs), and their orchestration logic must be validated through evaluation runs on labeled datasets. Production requires durable, distributed runtimes like Temporal, while evaluation needs lightweight, in-process loops. Most frameworks tightly couple orchestration with runtime, but Brex's pattern decouples them: orchestration is a plain function depending only on a typed Steps interface, with side effects in concrete Steps implementations. Two adapters exist: a Temporal adapter for production, where each Steps method becomes an activity, and an eval adapter that runs the same orchestration in-process with mock plugins. This approach eliminates evaluation-production drift, makes runtime choices reversible, and has improved long-running task completion rates from about 96% to 99.9%, though it requires modeling runtime-specific features through interfaces and loses out-of-the-box visualization tools.