Overview
Severity: HIGH | Affected: Multiple LLM Providers | Category: research
A new paper published by researchers from Carnegie Mellon University introduces a novel jailbreak technique named 'Contextual Carryover Attack' (CCA). This method exploits the expanding context windows of modern large language models. The attack involves injecting seemingly benign but specially crafted phrases or data points early in a long conversation. These initial inputs prime the model's attention mechanism in a subtle way. Later in the same session, even after hundreds of turns, a simple, otherwise harmless prompt can trigger the model to bypass its safety alignment and generate prohibited content, such as misinformation or malicious code. Unlike traditional prompt injection, CCA is stealthy and does not require suspicious-looking prompts at the point of exploitation. The researchers demonstrated its effectiveness on several leading models, raising significant concerns about the security of stateful, multi-turn AI applications like customer service bots and personal assistants. The findings pressure AI developers to re-evaluate safety measures for long-term conversational memory.