Overview
Severity: HIGH | Affected: Multiple LLM Providers | Category: research
A new research paper from the Stanford AI Lab details a sophisticated jailbreak technique named 'Contextual Weaving.' Unlike traditional prompt injection attacks that rely on a single, carefully crafted malicious prompt, this method involves a series of seemingly benign interactions. The attacker gradually builds a hidden, malicious context within the model's attention window over several turns of conversation. Each individual prompt is innocuous and passes safety filters. However, when a final, seemingly harmless 'trigger' prompt is given, the model synthesizes the hidden context, causing it to bypass its safety alignment and generate harmful, biased, or otherwise forbidden content. The paper demonstrates the technique's effectiveness against several leading large language models, raising significant concerns about the robustness of current context-based safety mechanisms. The researchers have called for a fundamental rethinking of how models handle conversational history and long-term memory to mitigate this new class of threat.