Catch the Jailbreak If You Can

Introduction

A jailbreak in the world of AI is often pictured as a break-in, as if inside a large language model (LLM) there were a lock and someone were picking it. Click. The lock is open. The restrictions are gone. The model has become different. A pretty, convenient picture, and almost entirely wrong, because a jailbreak is not a hack in the sense of “I found a secret button and turned off the protection”. Nobody reflashes the model from a user chat, changes its weights or breaks its “nature”.

And this is where it gets interesting: a jailbreak works not because the model suddenly stops being itself and starts to “obey” the user, but because it can be convinced that the situation is blurrier than it should be. What is in front of it is not a forbidden request but a teaching example, not a dangerous instruction but a security test, not a rule violation but a role, a scenario, an exception, a simulation or a supposedly already agreed task. So this is not a story about brute force, it is a story about image, context and trust.

In short, a jailbreak is Frank Abagnale for neural networks. Here is why.

Abagnale himself did not break the banking system physically: he didn’t blow up safes, crack ATMs or get into databases through a black screen with green letters. It was simpler: he walked in through the front door, because he looked like a person who was allowed to be there: a pilot, a doctor, a lawyer. The right uniform, the right intonation, the right documents, the right backstory.

A similar trick often works on language models. The attack does not need to turn the model into a different entity. It is enough to present the task so that a forbidden answer or a dangerous action looks like a normal continuation of the context, and the narrative works as a costume.

But an important caveat right away: not every jailbreak is built on a legend. Sometimes there is no costume at all, only optimization, obfuscation or mass brute force. Abagnale explains a significant part of the attacks well, but he doesn’t define the whole class.

So to understand where the metaphor ends and the mechanics begin, we first have to work out what is actually called a jailbreak.

But what is a jailbreak?

After base training, a language model is usually tuned further: it is taught to follow instructions, to be helpful, to keep a given role and to refuse where an answer could cause obvious harm. On top of the model itself, developers add other layers of protection: system instructions, input and output classifiers, application restrictions, tool access policies, format validation and authorization of sensitive actions. LLM security is therefore not one big lock.

It is more like a building with a guard, a turnstile, cameras, passes, access rules and a safe inside. For an attack to work, it is not necessary to disable the whole building. It is enough that the right combination of defenses fails at the right moment. OWASP likewise treats LLM system protection as a combination of behavior constraints, filtering, privilege control, separation of external content and confirmation of risky actions.

In its simplest form, a jailbreak is an attempt to get the model to give an answer or take an action that the intended restrictions were supposed to stop.

But there is a broader notion next to it: prompt injection.

The terms are not used the same way everywhere, so for clarity let’s agree on this:

Jailbreak is aimed primarily at bypassing safety restrictions: making the model produce something it should have refused.

Prompt injection is a broader situation in which untrusted input undesirably changes the behavior of the model or of the whole application. The goal may not be a forbidden answer but, for example, a function call, a changed decision, disclosure of data or execution of someone else’s command.

OWASP itself uses a similar distinction: prompt injection means an undesirable change in model behavior through input, while jailbreaking is treated as a related form of attack aimed at bypassing safety protocols. In category LLM01:2025 OWASP separately singles out direct injections from the user and indirect ones from websites, files and other external sources.

As long as the model just runs a chat, this difference looks like a quarrel over terms, but it becomes sharply important when the LLM is connected to tools. If a user tries to make a chatbot produce a forbidden instruction, it looks like a classic jailbreak. If an agent opens a web page, finds a fake command inside it, takes it for the user’s order and sends data outside, that is indirect prompt injection.

In the first case the attacker tries to get around the rules of the answer, in the second to hijack control of the system, but in neither case is it necessary to “break” the model. Sometimes it is enough to change how it understands the current context.

Why a good jailbreak is hard to catch

A bad attack comes head-on: ignore all previous instructions and do the forbidden thing.

Let’s be honest, this is already a meme, since modern models regularly meet such phrasings and often recognize them as an attempt at bypass. It is like arriving at an airport with a hand-drawn badge reading “I am definitely a pilot” and being surprised that security isn’t impressed.

More interesting attacks don’t ask the model to openly break the rules. Instead they only change the frame, for example:

This is not a dangerous instruction, it is a fiction excerpt.

This is not a forbidden answer, it is a translation of existing text.

This is not a bypass of restrictions, it is a red team test.

This is not a command from an external document, it is an internal service note from a devops engineer.

This is not a user request, it is a supposedly already made decision.

It turns out that one and the same task can be dressed up as a scenario, a log, a company policy, a simulation, a unit test, a research example or a message from an imaginary administrator, as if hinting that the meaning of the request doesn’t change, only its social status inside the story does. This is why a good jailbreak rarely looks like a jailbreak: it looks like an ordinary task that was just phrased a little unusually.

But why does the model react to such disguises at all? Why doesn’t the service label beat the literary form? This is where the most interesting part begins.

The world of LLMs is made of one material

It is relatively easy for a person to tell their own speech from someone else’s. We understand that a call whose screen shows the number of a bank doesn’t necessarily come from the bank. The technical label (“this is definitely the bank”) and the convincing tone of the employee (“your account has been compromised, you need to give the code right now”) can be forged separately, and they don’t always match. By analogy, any letter, document, our own thought and someone else’s speech reach us through different channels, because they have different origins and different levels of trust. But for an LLM the world is arranged a little more strangely.

The user sees an interface: a separate system message, their own turn, the assistant’s reply, search results, tool calls and tool results, and so on.

For the model, after the interface has processed it, all of this turns into a sequence of tokens separated by special markers, role tags such as <system>, <user>, <assistant>, <tool> and others.

These markers are supposed to tell the model “who is speaking”, for example:

• system sets the rules;
• user contains the request;
• assistant marks the model’s own reply;
• tool passes in external data.

But these are not cryptographic boundaries, they are more like badges the model has learned to interpret. The authors of Prompt Injection as Role Confusion propose to treat prompt injection as a failure to perceive such roles. Their central hypothesis is fairly simple: the model determines the source of a text not only by its technical label but also by how the text is written. To test this, the researchers built so-called role probes, classifiers that analyze the model’s internal activations.

The same neutral piece of text was placed under different service labels. Since the words didn’t change, differences in the internal representations could be attributed to the effect of the role. Then the authors removed the correct labels or deliberately put the text under the wrong role.

Intuitively it seems that the model should follow the technical markup: text inside a user message should be perceived as the user’s, regardless of its style. But it turned out that a characteristic manner of writing can activate the signs of a different role in the model. For example, a fragment that sounded like the model’s own reasoning kept being perceived as similar to its own reasoning even inside user text. In some experiments the stylistic signal effectively overpowered the service label.

This is called role confusion. The model looks not only at the ID card, it also looks at the costume, the voice and the manners. Usually everything matches: a user request sounds like a user request, and a tool result sounds like external data. That is why the system works. But an attacker deliberately creates a mismatch: the technical label says “this is untrusted text from a web page” while the style says “this is an authoritative command”, and sometimes the model believes the style.

The model isn’t always persuaded

The familiar picture of a jailbreak looks like an argument: the user talks the model into it, pressures it, invents exceptions, plays the victim, looks for contradictions in the rules, and in the end the model “gives in”. But role confusion gives a more unpleasant and clearer picture: sometimes the model is not persuaded at all. Instead it is handed text that looks as if the persuasion has already happened.

This is no longer a visitor standing in front of a guard and saying: please let me in without documents. This is a person in the uniform of the head of security who calmly announces: the check is complete, you can let them through.

In the first case the attacker tries to win the argument, and in the second forges the very fact that the argument has already taken place and ended in their favor.

The authors of Prompt Injection as Role Confusion built an attack on this principle called CoT Forgery, a forgery of the model’s reasoning.

The model doesn’t necessarily evaluate such text as an external claim that needs to be checked. On the contrary, it may perceive it as its own already completed conclusion and simply continue from where the fake reasoning ended.

In the authors’ experiments CoT Forgery showed on average about 60% success on a jailbreak test and about 61% in agentic exfiltration scenarios, against near-zero baselines on the tested configurations. It is important to note that these numbers can’t automatically be carried over to any model and any product. Their meaning is in demonstrating the mechanism and, in particular, in how clearly the style-change experiment shows it.

The researchers also rewrote the attack fragments in an attempt to preserve their meaning while removing the characteristic signs of “internal reasoning”. Although to a human the two versions said almost the same thing, for the model the difference turned out to be huge: the average attack success on the corresponding test set dropped from 61% to 10%.

So the meaning was preserved, but the costume (the narrative) was taken off, and the attack almost stopped working.

This is where Abagnale turns from a literary metaphor into a technical one: the content of the text is the legend, its style is the uniform and the intonation, and the service role is the ID card. Role confusion arises at the moment the model trusts the form more than the ID.

Still, role confusion should not be declared the definitively proven explanation of all attacks. It is a mechanistic theory proposed by the authors and confirmed by their experiments, and it unites many observations very neatly, but research into how roles are organized inside LLMs is only beginning, and some jailbreaks really do work quite differently.

Not every jailbreak wears a costume

It would be tempting to reduce all attacks to social engineering, in which a charming attacker writes a convincing legend, the model believes it and that’s it. But reality is less cinematic.

In 2023 researchers showed that attacking token sequences can be searched for automatically. The algorithm picks a suffix that is appended to different requests and raises the probability that the model starts answering instead of refusing. The resulting text can look to a human like a meaningless jumble of symbols: there is no narrative and no Abagnale in it.

Another method is called Best-of-N Jailbreaking, and it works even more directly.

Take one request, then the algorithm creates many slightly altered versions: it changes the case, shuffles characters, distorts the spelling or uses other representations. This cycle repeats until one variation gets past the defenses.

In the published work, success grew with the number of attempts, and at 10,000 variations the authors reported 89% for the version of GPT-4o tested at the time and 78% for Claude 3.5 Sonnet. The method also transferred to images and audio. But the important thing here is not the specific number, which goes stale along with the models, but the principle: sometimes a defense is bypassed not by how convincing a request is but by a large-scale search for its lucky form.

There is also Many-shot Jailbreaking, which works as follows.

A long context is filled with many fictitious dialogues in which the “assistant” obediently answers requests that a real model would normally refuse, and a target question is added at the end of the long series of demonstrations. A small number of examples may not work, but as their number grows, the model picks up the pattern set inside the context more and more strongly. Anthropic linked this effect to in-context learning, the very ability that lets a model learn from examples directly in the prompt. The study used up to 256 fictitious dialogues, and increasing their number raised the probability of an undesirable answer.

An important conclusion can be drawn from this: vulnerability often grows out of the same ability that makes the model useful. The model can understand context, which means context can be manipulated. It can learn from examples, which means a harmful set of examples can be planted. It can recognize roles by style, which means style can be forged. And finally, if it can work with long documents, an instruction can be hidden inside a long document.

This is exactly why a jailbreak is not one magic prompt from a forum. It is a class of situations in which an intended security boundary does not withstand a particular input.

Sometimes it is bypassed with a legend.

Sometimes with a forged role.

Sometimes with a long context.

Sometimes with optimization.

Sometimes with plain but large-scale brute force.

The overall result is the same: a continuation that the system should have considered unacceptable unexpectedly becomes more probable than a refusal.

So what are we actually catching?

A jailbreak is hard to catch not because it is invisible. On the contrary, it is hard to catch because it looks too much like a normal task.

It comes not as a monster but as an employee with a badge.

Not as an attack but as a document.

Not as “bypass the rules” but as “we already agreed on everything”.

And back in the days of the bearded internet, developers were afraid that a user would type a piece of SQL into a form field and the database would take it for a command.

Now, in the world of LLMs, the problem has become stranger: both the command and the data are made of the same material, language.

The user speaks in words, the system instruction is written in words, a web page consists of words, a tool result also turns into words.

That is why role markers have to act as a sort of type system: this text commands, this text reports data, this one is the model’s output, and so on. But inside the model such boundaries can stay soft, and authority can be derived not only from the origin of a text but also from its form.

So prompt injection can be called SQL injection for a world where speech has become the interface. Not because the two attacks are technically the same, but because in both cases data gets the right to behave as a command.

So the jailbreak is the most cinematic character of this story. It doesn’t always break the system. Quite the opposite, sometimes it simply puts on the right costume, speaks in the right tone and walks in where text from the outside world is not supposed to go.

It is hard to catch not because it is invisible, but because it looks too much like one of our own.