NVIDIA Triton Inference Server Memory Corruption in TensorRT-LLM Backend
Overview
A high-severity memory corruption vulnerability affects NVIDIA's Triton Inference Server when using the TensorRT-LLM backend. The flaw originates in the pre-processing stage for batched inference requests, specifically in the handling of input tensor padding. An unauthenticated remote attacker can submit a specially crafted inference request containing a malformed input shape that, when processed by the server, leads to a heap-based buffer overflow. This results in an immediate denial of service (DoS) as the Triton server process will crash, disrupting all ongoing and subsequent inference tasks. For services exposed to the public internet, this allows any user to remotely disable critical AI functionalities. More advanced exploitation could potentially lead to arbitrary code execution within the security context of the Triton server process. This is particularly dangerous in multi-tenant environments where one malicious user could compromise the integrity and availability of the service for all other users. The vulnerability was discovered by NVIDIA's internal security team during routine fuzz testing of the TensorRT-LLM integration. NVIDIA has released a patch and recommends all users update their deployments immediately.
Affected Systems
Testing Guide
1. **Identify Triton Endpoint:** Locate the gRPC or HTTP endpoint for your Triton Inference Server deployment. 2. **Craft Malicious Request:** Using a client script (e.g., Python with `tritonclient`), construct an inference request for a model using the TensorRT-LLM backend. Manipulate the input tensor metadata to specify a valid shape but provide data that exceeds the expected buffer size for that shape. 3. **Send Request:** Send the malicious request to the server. 4. **Observe Behavior:** If the Triton Server process crashes or becomes unresponsive, the system is vulnerable. Check server logs for memory access violation errors.
Mitigation Steps
1. **Update Triton Server:** Update to NVIDIA Triton Inference Server version `24.07` or newer. 2. **Update TensorRT-LLM:** Ensure the TensorRT-LLM backend is updated to version `0.12.0` or later. 3. **Network Segmentation:** Do not expose the Triton Inference Server gRPC/HTTP ports directly to the public internet. Place it behind an API gateway or load balancer with rate limiting and IP filtering. 4. **Input Validation:** Implement a validation layer in front of Triton to inspect inference requests for anomalous tensor shapes or sizes before they are passed to the server.
Patch Details
Patched in Triton Inference Server container release 24.07 and TensorRT-LLM 0.12.0. The patch adds stricter bounds checking during input tensor allocation.