NVIDIA Triton Inference Server Remote Code Execution via Malformed Model Configuration
Overview
A critical vulnerability was discovered in NVIDIA's Triton Inference Server that allows an unauthenticated attacker with model repository access to achieve remote code execution. The vulnerability resides in the server's parsing logic for model configuration files (`config.pbtxt`). By crafting a configuration with a specially formatted `instance_group` section containing an embedded native library path, an attacker can trigger a buffer overflow in the model loading process. This overflow can be exploited to overwrite the instruction pointer and execute arbitrary code with the same privileges as the Triton server process. This is particularly dangerous in multi-tenant environments or MLOps platforms where users can upload their own models, as a malicious user could compromise the entire inference host, potentially accessing other models' data or attacking the underlying infrastructure. The issue was reported privately by researchers from Trail of Bits.
Affected Systems
Testing Guide
1. Set up an affected version of Triton Inference Server. 2. Create a model repository with a simple model. 3. Modify the `config.pbtxt` to include a malformed `instance_group` with a long, crafted path designed to trigger the overflow. 4. Send a request to the server to load the model. 5. Monitor the server process for a crash or signs of unexpected behavior. Use a debugger to confirm control over the instruction pointer if possible.
Mitigation Steps
1. Immediately upgrade the NVIDIA Triton Inference Server to version 2.65.0 or later. 2. If an immediate upgrade is not possible, restrict access to the model repository to only trusted users. 3. Implement a CI/CD pipeline step to scan model `config.pbtxt` files for suspicious patterns before deployment. 4. Run the Triton server as a low-privilege user and within a containerized, isolated environment like gVisor or Kata Containers.
Patch Details
Fixed in NVIDIA Triton Inference Server 2.65.0. The update adds bounds checking and path validation during configuration parsing.