Ray

Accelerating inference of an encoder guard model: TensorRT, Triton, vLLM, Ray Serve

The article compares tools for accelerating a zero-shot PII-encoder guard model that checks input and output of an LLM app. The author tests TensorRT, NVIDIA Triton, vLLM, Ray Serve, and a Flash DeBERTa backbone, measuring RPS and latency. TensorRT FP16 gives a 23% boost over PyTorch FP16, but INT8 quantization fails due to custom operations.

NVIDIA vLLM Ray