Microsoft Unveils AI Agent LLM Routing Architecture on AKS, Cutting Costs by Up to 85%

Microsoft has published a reference architecture for routing agent traffic on Azure Kubernetes Service (AKS), breaking down the problem into three key choices: which model responds to a call, how the call is managed, and which GPU replica handles it. The design combines the Kubernetes Gateway API Inference Extension for load balancing, agentgateway as an AI proxy, and RouteLLM for semantic routing, all connected to an OpenAI-compatible endpoint. This is tailored for agentic workloads rather than simple chat, as a single agent task can trigger hundreds of LLM calls in a planning-acting-observing loop, most of which (filling tool arguments, yes/no decisions, summaries) do not require a frontier model. RouteLLM inspects the prompt and predicts if a cheaper model can match the quality of a stronger model, using a matrix factorization router trained on human preference data. Agentgateway, an open-source proxy compatible with OpenAI, manages authentication, per-agent rate limits, cost tracking, and guardrails without inspecting prompt content. The Gateway API Inference Extension's Endpoint Picker checks real-time GPU status, including vLLM's KV cache usage and queue depth, to decide which replica of the selected model handles the request. For self-hosted paths, agentgateway calls Endpoint Picker via ext-proc, bypassing a separate Gateway API gateway. KAITO provides GPU node pools on demand and runs vLLM, exposing metrics like vllm:num_requests_waiting and vllm:kv_cache_usage_perc. The strong model path goes through agentgateway's AI backend to Azure OpenAI, while the weak model path routes via a service backend to KAITO-served pods. Azure-managed Prometheus and Grafana scrape routing, cost, and GPU metrics for a unified view. A key number is RouteLLM's upgrade threshold: on tested model pairs, the mf router achieved about 95% of GPT-4's quality on MT-Bench while sending only about 26% of calls to GPT-4, saving up to 85% in costs. Microsoft warns this figure is not automatic and depends on the model pair used for training RouteLLM; users must calibrate thresholds based on actual traffic. Prompt caching complicates token costs, as cache hits get discounts, and switching models cools caches on both sides. The article cautions that components are young; fields change between versions—Inference Extension renamed and restructured CRDs before v1. All was verified end-to-end in mid-2026 on AKS with Inference Extension v1.0.0 and agentgateway v1.3.1. Teams can adopt the architecture in phases: for managed models with few agents, agentgateway governance suffices; for self-hosted single model, KAITO and Inference Extension are enough without semantic routing; RouteLLM is worthwhile when there's a clear price gap between strong and weak models and considerable simple traffic, which applies to almost all looping agents.