Path Traversal in NVIDIA Triton Inference Server Enables Model and Data Tampering
Overview
NVIDIA Triton Inference Server was found to contain a path traversal vulnerability in its model repository loading mechanism. An attacker with privileges to add or modify model configurations, but not necessarily access the server's filesystem directly, could exploit this flaw. By crafting a malicious model configuration file (`config.pbtxt`) with specially formatted paths containing `../` sequences for model or version locations, an attacker can force Triton to load model files from arbitrary locations on the server's filesystem. This could lead to severe security breaches, including information disclosure by loading sensitive files (e.g., `/etc/passwd`, private keys) as if they were model weights, and data tampering by overwriting legitimate model files with malicious versions. The impact is critical in multi-tenant environments where different users manage their own models. One tenant could potentially read or overwrite another tenant's proprietary models, or escalate their privileges by loading a crafted model that executes arbitrary code when processed by a backend like PyTorch that can load pickled objects. The vulnerability was discovered by internal security auditing and affects both Linux and Windows deployments of the Triton server.
Affected Systems
Testing Guide
1. In your model repository, create a new model directory (e.g., `malicious_model`). 2. Inside, create a `config.pbtxt` file. 3. In the configuration, define a model version that points to a sensitive file outside the repository, e.g., `platform: "onnxruntime_onnx", version_policy: { specific: { versions: ["../../../../etc/passwd"] } }`. 4. Attempt to load the `malicious_model` via the Triton API. 5. If the server does not return an error and attempts to load the file (which will likely fail parsing but proves the read attempt), the system is vulnerable.
Mitigation Steps
1. **Upgrade Triton Server**: Immediately upgrade to NVIDIA Triton Inference Server version 23.08 or later. 2. **Strict Access Control**: Restrict permissions to modify the model repository to only highly trusted administrators. Use filesystem permissions to ensure the Triton server process cannot write outside of its designated directories. 3. **Input Validation on Paths**: If using a custom model management front-end, implement strict validation to sanitize and normalize all file paths, explicitly disallowing directory traversal sequences. 4. **Use Minimal Container Images**: Run Triton in a minimal, hardened container with a non-root user to limit the potential impact of a successful exploit.
Patch Details
Patched in Triton Inference Server release 23.08. See the NVIDIA security bulletin for details.