NVIDIA TensorRT

NVIDIA's inference compiler that squeezes maximum performance out of a GPU in production.

TensorRT compiles a network, usually imported from ONNX, into an engine optimized for one specific NVIDIA GPU: layer and tensor fusion, automatic selection of the fastest kernels for the target architecture, and reduced precision in FP16, calibrated INT8 or FP8. The engine runs server-side through Triton or embedded on Jetson, and TensorRT-LLM applies the same approach to large language models with in-flight batching and a paged KV cache. It's our choice when the target is a fixed NVIDIA GPU and every millisecond of latency counts.