Remote Code Execution in NVIDIA Triton Inference Server via Malformed Model Configuration
Overview
A high-severity remote code execution (RCE) vulnerability was discovered in the NVIDIA Triton Inference Server. The vulnerability, a heap-based buffer overflow, exists in the server's C++ backend responsible for parsing model configuration files (`config.pbtxt`). An attacker with permissions to upload models to the server can exploit this by crafting a configuration file with an unusually long string in a specific field, such as a tensor name within an `input` or `output` block. When the Triton server parses this malformed configuration, it fails to perform adequate bounds checking while copying the string into a fixed-size buffer on the heap. This overwrite can corrupt adjacent memory structures, including function pointers or object vtables. By carefully structuring the overflow, an attacker can hijack the program's control flow and divert execution to a shellcode payload embedded elsewhere in the model repository. Successful exploitation allows the attacker to execute arbitrary code with the same privileges as the Triton server process, which often runs as a privileged user. This can lead to a complete compromise of the inference server, exfiltration of all models and sensitive data processed by it, and lateral movement within the hosting infrastructure.
Affected Systems
Testing Guide
1. In a controlled test environment, deploy a vulnerable version of the Triton Inference Server. 2. Create a model repository and a `config.pbtxt` file for a dummy model. 3. In the configuration file, modify one of the input tensor names to be an extremely long string (e.g., 2048 'A' characters). 4. Load the model into the server using an admin client or by placing it in the model directory. 5. Monitor the server process. If the server crashes, it is likely vulnerable to the buffer overflow. Advanced testing would involve crafting a specific payload to achieve RCE instead of just a crash.
Mitigation Steps
1. **Upgrade Triton Server:** Update to NVIDIA Triton Inference Server version 2.59.0 or newer, which contains a patch for this vulnerability. 2. **Restrict Model Uploads:** Limit permissions to upload or modify models in the model repository to only trusted, authenticated users. 3. **Input Validation:** Implement a pre-validation step or proxy that scans `config.pbtxt` files for abnormally long strings or other structural anomalies before they are passed to the Triton server. 4. **Run as Low-Privilege User:** Run the Triton Inference Server process as a dedicated, low-privilege user with restricted file system and network access to limit the impact of a potential compromise.
Patch Details
Patched in NVIDIA Triton Inference Server version 2.59.0. The fix involves adding proper bounds checks during the parsing of model configuration files.