Accelerating inference of an encoder guard model: TensorRT, Triton, vLLM, Ray Serve

The article describes an attempt to accelerate a guard model that sits between an LLM and the user, checking for dangerous content. The model is a zero-shot PII-encoder: it takes entity types as text input along with the document, extracting entities and classifying safety in one forward pass. Due to its architecture, it poses seven engineering constraints: variable input shape, non-tensor output requiring CPU span decoding, specific operations like gather and bilinear scoring, potentially non-standard attention, batch cost determined by the longest element, serving infrastructure optimized for autoregressive LLMs, and the importance of tail latency (P95, P99) because the guard is called twice per dialogue turn. The experiment compares five tools: TensorRT runtime, NVIDIA Triton, vLLM, Ray Serve, and a Flash DeBERTa backbone, with baselines using LitServe. The runtime track with TensorRT used a short load profile (one worker, 50 users, 60 seconds), while serving tracks used a longer profile (four workers, 100 users, 15 minutes), so numbers are only comparable within each track. The TensorRT FP16 variant achieves 130.72 RPS versus 106.49 RPS for PyTorch FP16, a 23% improvement. INT8 quantization attempts failed due to lack of support for custom operations, resulting in very low RPS and high latency.