Cross-Tenant Model Memory Exfiltration in AWS Bedrock via Log Stream Manipulation
Overview
A critical information disclosure vulnerability was discovered in AWS Bedrock's handling of fine-tuned models on shared GPU infrastructure. The vulnerability, assigned CVE-2025-54321, allowed an attacker in one AWS account to potentially access sensitive data from other tenants' inference requests. The root cause was an issue with how Bedrock managed and isolated CloudWatch log streams for custom models experiencing specific types of CUDA memory errors. Under high load, when a model invocation failed due to a specific out-of-memory condition, the error handling process would incorrectly route verbose debugging logs to a shared, improperly configured log stream. These debug logs contained a partial snapshot of the GPU's VRAM at the time of the error. A sophisticated attacker could intentionally provision a model designed to crash in this specific way. By repeatedly triggering the error, they could cause memory dumps from other tenants' models running concurrently on the same physical GPU to be written to the log stream they could access. These dumps could contain fragments of prompts, inference results, personally identifiable information (PII), or even parts of proprietary models being used by other customers. The attack required precise timing and significant resources to execute reliably but represented a fundamental break in the multi-tenant security model of the service.
Affected Systems
Testing Guide
This vulnerability was in the AWS backend infrastructure and cannot be tested directly by customers. AWS has confirmed the patch has been rolled out globally. Customers can verify that their Bedrock logging configurations are correctly scoped to their own account's CloudWatch log groups as a matter of best practice, but this would not have prevented the original vulnerability.
Mitigation Steps
1. **No Customer Action Required**: AWS has patched the issue on their backend infrastructure. The fix was deployed automatically to all regions. 2. **Review Logs for Past Exposure**: AWS has notified customers whose data may have been exposed. It is recommended to review access logs for the affected period for any signs of anomalous activity. 3. **Enable Advanced Logging and Monitoring**: Use services like AWS GuardDuty and CloudTrail to monitor for unusual API activity or access patterns related to Bedrock and CloudWatch. 4. **Data Encryption**: For extremely sensitive workloads, consider client-side encryption of data before sending it in inference requests, although this may not be feasible for all use cases.
Patch Details
The issue was resolved by AWS on their backend. The fix involved enforcing strict log stream isolation for all model invocation environments and sanitizing error messages to prevent memory contents from being logged.