RouteLLM

Microsoft Unveils AI Agent LLM Routing Architecture on AKS, Cutting Costs by Up to 85%

Microsoft has published a reference architecture for routing agent traffic on Azure Kubernetes Service (AKS), combining RouteLLM for semantic routing, agentgateway for governance, and the Kubernetes Gateway API Inference Extension for GPU-aware load balancing. The design targets agentic workloads, where many LLM calls don't require frontier models, potentially saving up to 85% on costs. However, the savings depend on model pair calibration and the maturity of components.

Microsoft OpenAI RouteLLM