PDD Architecture Breaks Cross-Cluster Heterogeneous Inference Barrier: 51.5% Lower First-Token Latency, 37.5% Cost Reduction

At WAIC 2026, InfiniteG presented PDD (Prefill-RelayDecode-MainDecode), a novel inference architecture that decomposes the traditional PD separation into three stages. The key insight is that agent workloads have highly skewed cache hit rates (70-80% of requests >95% hit rate), causing only a few low-hit requests to suffer long KV Cache transfer delays over WAN Ethernet. PDD inserts a RelayDecode (RLD) instance in the same cluster as Prefill, which receives KV Cache via fast RDMA and starts decoding immediately, masking the slow Ethernet transfer to the remote MainDecode (MD) instance. Once MD receives the full KV Cache, a handoff mechanism transfers only small token IDs (few KB) to MD for local recomputation, avoiding large KV transfers. In tests on DeepSeek-V4-pro with real agent traces, PDD achieved P90 TTFT reduction from 18.3s to 9.8s (46% lower), P99 from 29.7s to 14.4s, while RLD handled only 6.2% of token generation. Total infrastructure cost dropped 3.5-7.1% while effective throughput (RPS goodput) increased 27.8%, yielding a 37.5% improvement in benefit-cost ratio. The work enables heterogeneous inference across low-cost WAN-linked homogeneous clusters, maximizing utilization of scattered compute resources.