Path Traversal in NVIDIA Triton Inference Server Allows Unauthorized Model Access and Overwrite
Overview
A high-severity path traversal vulnerability was discovered in the NVIDIA Triton Inference Server, a widely used platform for deploying AI models at scale. The vulnerability exists in the server's model repository API, which is responsible for loading and unloading models. Due to improper input validation on model names or paths provided via the API, an authenticated attacker with permissions to manage models could use 'dot-dot-slash' (`../`) sequences to traverse the file system. This allows the attacker to read or write files outside of the designated model repository directory. The impact is significant: an attacker could read sensitive configuration files from the server, exfiltrate proprietary models by accessing their raw file paths, or worse, overwrite existing models with malicious versions to carry out model poisoning attacks. In some configurations, this could also lead to a denial of service by overwriting or deleting critical system files. This vulnerability underscores the importance of securing MLOps infrastructure, as a compromise in the model serving layer can undermine the integrity and confidentiality of the entire AI system.
Affected Systems
Testing Guide
1. **Check Version**: Verify your NVIDIA Triton Inference Server version. You can typically find this in startup logs or by querying the server's metadata endpoint. 2. **Attempt Traversal (Non-destructive)**: If you have a non-production test environment, an authenticated user can attempt to load a model using a path traversal payload in the model name, such as `../some-other-dir/model`. Observe if the server returns an error indicating it tried to access a path outside the repository. **Do not perform write tests on production systems.**
Mitigation Steps
1. **Update Triton Server**: Upgrade to NVIDIA Triton Inference Server version 2.42.0 (for the 24.01 branch) or newer versions as specified in the NVIDIA security bulletin. 2. **Restrict API Access**: Limit access to the Triton server's management and model control APIs. Use network firewalls and authentication mechanisms to ensure only trusted administrators and MLOps pipelines can interact with these endpoints. 3. **File System Permissions**: Run the Triton server process under a least-privilege user account that has restrictive read/write permissions, limited only to the model repository and necessary log directories. 4. **Enable Audit Logging**: Enable and regularly review Triton's audit logs for any unusual model load or unload requests, especially those containing suspicious path-like strings.
Patch Details
Patched in versions released after February 2024, including 2.42.0. The patch adds stricter validation and sanitization of model repository paths.