NVIDIA Triton Inference Server Path Traversal via Model Repository API
Overview
A high-severity path traversal vulnerability was discovered in the NVIDIA Triton Inference Server. The vulnerability, tracked as CVE-2023-31041, allows an attacker with access to Triton's management HTTP/gRPC API to read and write arbitrary files on the server's filesystem. The flaw exists in how the server handles model repository paths when loading or unloading models. An attacker can use directory traversal sequences (e.g., `../..`) in the model repository path provided in an API request. This enables them to navigate outside of the intended model repository directory. A successful exploit could allow an attacker to read sensitive files, such as configuration files, SSH keys, or application source code. More critically, an attacker could write files, such as a web shell into a web server's root directory or a malicious cron job, leading to full Remote Code Execution (RCE) on the inference server. This vulnerability is particularly dangerous in MLOps environments where different teams may have access to the Triton API to manage their models, as a compromised user could escalate privileges and take over the entire inference infrastructure.
Affected Systems
Testing Guide
1. **Check Version**: Run `docker inspect <triton_container_id>` or check your deployment configuration to determine the version of the Triton Inference Server image being used. If the version tag is earlier than `23.04`, you are likely affected. 2. **Send Malicious Request**: Using a gRPC or HTTP client, send a `RepositoryModelLoad` request to the Triton server. In the `model_name` or repository path parameter, include a path traversal payload. For example, attempt to load a model from `/path/to/models/../../../../etc/passwd`. 3. **Check Response**: If the server returns an error indicating the file could not be parsed as a model, but does not return a 'path not allowed' error, it suggests the traversal was successful. If the server logs show an attempt to access `/etc/passwd`, the vulnerability is confirmed.
Mitigation Steps
1. **Upgrade Triton**: Update the NVIDIA Triton Inference Server to version 23.04 or later, where the path validation logic has been fixed. 2. **Restrict API Access**: Limit access to the Triton management API. Use a firewall or network security group to ensure only trusted IP addresses can connect. 3. **Use API Authentication**: If Triton is exposed, enable and enforce authentication and authorization for the management API. 4. **Run as Non-Root**: Run the Triton server process as a dedicated, low-privilege user. This will limit the impact of a successful file write exploit, preventing modification of critical system files.
Patch Details
Patched in NVIDIA Triton Inference Server container release 23.04 and later.