Denial of Service in NVIDIA Triton Inference Server via Malformed Tensor Shape in Request
Overview
A high-severity denial-of-service (DoS) vulnerability was discovered in the NVIDIA Triton Inference Server, a widely used platform for deploying and scaling AI models in production. The vulnerability, present in the server's request handling logic, could be triggered by an unauthenticated remote attacker. The flaw lies in the pre-processing stage where the server allocates memory for incoming inference requests. By sending a gRPC or HTTP request containing a tensor with a malformed or excessively large shape definition (e.g., specifying dimensions that result in an integer overflow when calculating total memory size), an attacker could cause the Triton server to attempt a faulty memory allocation. This triggers a segmentation fault or an unhandled exception, leading to the immediate crash of the Triton server process. Since a single Triton instance often serves multiple models for various applications in a multi-tenant environment, exploiting this vulnerability would render all hosted models unavailable until the server is manually restarted. This poses a significant operational risk for critical AI-powered services relying on the platform for real-time inference. The attack is low-complexity, requiring only the ability to send a crafted request to the server's public-facing endpoint.
Affected Systems
Testing Guide
1. **Check Triton Version:** Access your Triton server's metrics endpoint or check its startup logs to determine the running version. If it is older than 2.45.0, you are affected. 2. **Send Test Request (CAUTION):** In a non-production environment, use a client script to send an inference request with an intentionally invalid tensor shape (e.g., a negative dimension or dimensions that multiply to a value exceeding system memory limits). Observe if the server process crashes. **Do not perform this on a production system.**
Mitigation Steps
1. **Upgrade Triton Server:** Update all instances of NVIDIA Triton Inference Server to version 2.45.0 or later. 2. **Use a WAF:** Place a Web Application Firewall (WAF) or an API gateway in front of the Triton server to inspect incoming requests. Configure rules to block requests with anomalous or excessively large tensor shape definitions. 3. **Rate Limiting:** Implement strict rate limiting on the inference endpoint to reduce the impact of repeated exploit attempts. 4. **High Availability:** Run Triton servers in a high-availability cluster (e.g., on Kubernetes with multiple replicas) so that if one instance crashes, traffic can be automatically rerouted to healthy instances.
Patch Details
NVIDIA released a patch in NGC container version 24.02, corresponding to Triton Inference Server 2.45.0. The update adds improved input validation and sanity checks on tensor shape metadata before memory allocation.