Cross-Tenant Data Leakage in AWS Bedrock Custom Model Import Logging
Overview
A high-severity data leakage vulnerability was discovered in the AWS Bedrock service, specifically within the custom model import and fine-tuning workflow. The vulnerability arose from a race condition in the internal logging system that processed model validation jobs. When a user submitted a custom model from an S3 bucket for fine-tuning, the service would run a series of validation steps. If these steps failed due to data formatting or other errors, detailed error messages were generated. These messages could include snippets of the user's proprietary training data, model configuration parameters, or S3 bucket names. Due to a resource isolation flaw in the logging pipeline, a small fraction of these error logs could be incorrectly associated with and written to the AWS CloudTrail event log of a different AWS account operating in the same region. An attacker could potentially discover fragments of a victim's sensitive data by simply monitoring their own CloudTrail logs for anomalous entries. While the conditions for successful exploitation were rare and the leaked data was partial, the vulnerability posed a significant risk of intellectual property and confidential data exposure for organizations leveraging custom models on Bedrock. AWS has since patched the issue by enforcing stricter tenancy boundaries within their internal logging infrastructure.
Affected Systems
Testing Guide
This vulnerability was in the AWS backend and has been patched. It is no longer possible for customers to test for or reproduce this issue. The primary method of investigation is to review historical CloudTrail logs from before the patch date (approximately December 2025) for any unusual or unrecognized log entries.
Mitigation Steps
1. **Patch Applied by AWS**: AWS has deployed a fix for this vulnerability across all regions, and no customer action is required to be protected. 2. **Review Historical Logs**: Organizations concerned about past exposure should review their historical AWS CloudTrail logs for any anomalous entries that do not correspond to actions taken by their own users or roles, particularly entries containing keywords related to machine learning or model training. 3. **Encrypt Training Data**: As a best practice, always encrypt training data at rest in S3 using customer-managed keys (CMK) via AWS KMS. This ensures that even if data snippets are leaked, they remain encrypted. 4. **Minimize Data in Logs**: When creating training datasets, avoid placing highly sensitive information (e.g., PII) in fields or file paths that are likely to be logged in case of an error.
Patch Details
The vulnerability was resolved on the AWS service side. No client-side patch is required. AWS published a security bulletin for affected customers.