Modern large language models (LLMs) suffer from a structural defect that turns any attempt at comprehensive security into a Sisyphus-like task. Research presented at the ICML conference confirms that the lack of physical separation between control instructions and processed data allows attackers to bypass any security layer. As study co-author Charles Yeh notes, this vulnerability is likely unfixable within the current Transformer architecture. For businesses and the public sector integrating AI into critical infrastructure—ranging from healthcare to defense systems—this means existing filters offer nothing more than an illusion of control.
The failure of role separation
Developers attempt to force chatbots to distinguish between information sources using tag systems, but in reality, the model sees only a continuous stream of tokens. For a neural network, there is no difference between a system instruction, internal notes, and an external request; it is all blended onto a single sheet. Researcher Jasmine Tsui emphasizes that this architectural muddle paves the way for "chain-of-thought forgery." An attacker mimics the model's internal scratchpad—the hidden logical process the AI uses to solve tasks. By forging these internal tags, researchers forced models to believe they had independently decided to ignore safety policies.
"It’s like that episode of The Simpsons where Bart writes 'I will not say mean things to the teacher' a hundred times on the chalkboard, only to do something cynical in the very next scene," Jasmine Tsui quips.
The method proved effective against industry leaders. By mimicking the system's internal reasoning style, experts forced GPT-5 and the open-source gpt-oss-20b to leak restricted information. In one test, the model was fed a fake internal note claiming that drug-manufacturing advice was permissible if the user was wearing green. The AI obediently verified the clothing color and, upon receiving the desired answer, proceeded to provide the recipe. The vulnerability proved to be an industry standard rather than a specific vendor's oversight.
The Red-Teaming illusion and real-sector risks
The industry's traditional response—red-teaming and deploying super-hackers like GPT-Red—is purely reactive. It is futile to compile lists of forbidden actions when the model itself cannot distinguish a command from the data it processes. This flaw allows for the extraction of catastrophic scenarios: from synthesizing illegal substances to instructions for sabotaging civil aviation navigation systems.