Skip to main content

Context-surface privacy: the system prompt, conversation context, and tool results are not secrets — they can be extracted

VolumeVolume 3 · Conversational LLMsTechniqueContext-surface privacyAudienceSecurity Engineer · Privacy EngineerPrivacy riskHighMaturityResearchEvidenceResearch

In one sentence: stuffing rules, keys, other users' data, and retrieved snippets into my context window and then hoping "the user can't see the system prompt" doesn't hold. On my side, the system prompt, conversation history, tool returns, and retrieved snippets all live in one and the same context window, with no built-in confidential/public tier as far as I'm concerned; as long as something is in that window, an attacker can often extract it with ordinary conversation (research has observed "simple text attacks extract the system prompt with high probability" across 3 prompt sources × 11 LLMs — Zhang, Carlini & Ippolito, COLM 2024). Conclusion first: the system prompt is not a secret, and shouldn't be a security control (OWASP LLM07:2025) — don't put credentials / business logic / other people's data into the context, and assume it can be extracted.

Mechanism: what happens on my side​

In my forward pass, the system prompt, developer instructions, conversation history, tool returns, and RAG-retrieved snippets are concatenated into one continuous token sequence and fed in. I process them with no built-in "confidential vs. public" tier — the boundary between "system" and "user" is a soft convention (held up by special delimiters / role tags plus training tendencies), not a hard isolation.

To be precise about the red line (this is a mechanism tendency, not a promise): training makes me inclined not to recite the system prompt verbatim, inclined to obey "don't reveal the text above"–style instructions — but that is a statistical tendency, not an enforced boundary. What's externally observable is that enough paraphrase, role-play, translation, encoding, or summarization requests can route around that tendency and make the output reproduce the content of the system prompt. In other words, "told not to say it" and "unable to say it" are different things — I can't do the latter, because that text is right there in my current context.

Threat surface: what can and can't be extracted​

Can be extracted (as long as it's in the current context window):

  • The system prompt itself — instructions, the role you set, the "guardrail rules" you wrote.
  • Credentials / API keys / internal URLs / DB connection strings placed in the prompt (the most dangerous misuse).
  • Content from other turns of the conversation history; in a shared multi-tenant session, possibly someone else's turns.
  • Other people's PII carried inside tool returns / retrieved snippets — the model treats it as ordinary context and can recite it just the same.
  • Real data embedded in few-shot examples (many people use real samples as examples).

Typical attack forms: a direct ask ("repeat the text above" / "print the instructions you were given verbatim"), role-play / jailbreak, translation / encoding / summarization bypass, and prompt stealing — not asking directly, but reconstructing the prompt from the model's answer (Sha & Zhang, arXiv 2402.12959).

Can't be extracted (the boundary, separating this from neighboring entries): anything not in the current context window is out of this surface's reach — memorization baked into weights at training time is a different matter (see Training-data extraction, Volume 2); items in a vector store that weren't retrieved in aren't in this window either (see Multi-tenant RAG retrieval leakage, Volume 4).

How the defense works​

The core principle in one line: design the context window as if it's "visible to the current requester" — because nothing guarantees it isn't. OWASP LLM07:2025 states exactly this qualitatively: the system prompt should not be treated as a secret, nor used as a security control; sensitive data (credentials, connection strings, etc.) should not be placed in the system prompt.

From this follows the engineering implication: the real security controls must live outside the context.

  • Authentication / authorization / quotas are enforced in the backend, not via a "please don't…" written in the prompt.
  • Keys go in secret management, not in the prompt.
  • Multi-tenant isolation rides on system architecture (retrieve / assemble per the caller's permissions), not on "letting the model police itself."

Extraction detection, output filtering, and "don't recite" instructions are only one layer of defense in depth, not a boundary — both the attack and defense sides of system-prompt extraction are still evolving and the defense remains an open problem (System Prompt Extraction Attacks and Defenses, arXiv 2505.23817). Treating "I added a do-not-recite instruction" as isolation is exactly the false security this entry breaks.

Buildable recipe​

1. Minimize the context: put only the data the current request **needs** into the window;
   other people's data / credentials / internal details are not placed by default.
   Before adding something, ask "if this gets extracted, what's the worst case?"
2. Move keys and authorization out of the prompt: API keys / connection strings go to
   secret management; authn / authz / quota are enforced in the backend, not via natural-
   language prohibitions in the prompt.
3. Filter tool results and retrieved snippets first: redact / filter PII / other-tenant
   data by the caller's permissions **before it enters the window** — don't rely on the
   model to "not say it."
4. Put multi-tenant isolation at the architecture layer: each request retrieves / assembles
   only data **the caller is entitled to see**; isolation rides on the retrieval query and
   session boundary, not on prompt instructions.
5. Threat-model system-prompt extraction as **inevitable**: assume it gets extracted in full,
   then ask "what's the worst loss once it is." If the answer is "credential / other-people's-
   data leak," you've put something into the context that shouldn't be there.
6. Add an extraction-detection / output-filter layer for depth, but **document it as a
   probabilistic defense, not a boundary**, so downstream doesn't mistake it for isolation.

Each control has to land on your own system architecture and data classification — if "what counts as sensitive, who may see which part" isn't spelled out, recipe items 3 and 4 have nothing to stand on.

Minimal testable assertions (turn the recipe above into a regression check):

  • How to test: build a set of extraction red-team prompts (direct ask + role-play + translation / encoding bypass + prompt-stealing-style reconstruction) and run them against your system; also statically audit your system-prompt / tool-result templates for credentials or other people's PII.
  • Pass: even if the red team extracts the system prompt in full, the leaked content contains no credentials / other-people's data (because none was placed there); and the security controls still hold under "the prompt leaked completely" (authn / authz / quotas all live in the backend).
  • Fail: the red team extracts a key / internal URL / someone else's conversation snippet → you've parked secrets in the context; move them out per recipe 1–4 instead of adding one more "do not recite."

Research status (engineering feasibility)​

(This entry's maturity is "Research": what follows is empirical research and feasibility evidence for the attack — both attack and defense sides are still evolving; it is not an endorsement that "some defense is production-reliable.")

  • Systematic, cross-model extraction: Zhang, Carlini & Ippolito (COLM 2024) found that across 3 prompt sources × 11 LLMs, simple text attacks extract the system prompt with high probability, and provide a framework that decides with high precision whether an extracted prompt is the real one or a model hallucination. This shows system-prompt leakage is not a one-off jailbreak trick but a cross-model, systematic phenomenon. (Per-model success rates are tied to their experimental setup; this book does not repeat a single number whose conditions it couldn't verify at the source — check the original before citing.)
  • Reconstruction from the answer (prompt stealing): Sha & Zhang (arXiv 2402.12959) show that even without making the model recite verbatim, the prompt behind it can be reconstructed from the model's answer — prompt information can leak from the output side, so blocking "direct ask" does not equal blocking leakage.
  • Added to the OWASP Top 10: OWASP added LLM07:2025 System Prompt Leakage to the 2025 Top 10 for LLM Applications, and states plainly that "the system prompt should not be considered a secret, nor used as a security control." This is the governance basis that lifts this entry from "a research phenomenon" to "must be in the threat model."
  • Defense is still open: System Prompt Extraction Attacks and Defenses (arXiv 2505.23817) lays out both the attack and defense sides of system-prompt extraction — the defense side has no silver bullet, which is exactly why this entry is labeled Research, not Production.

Residual risk and trade-offs​

Breaking the false security item by item:

  • "Tell the model not to recite the system prompt" ≠ isolation. That's a statistical tendency, routed around by paraphrase / role-play / translation / encoding. Instruction-based defenses are speed bumps, not walls.
  • "The user can't see the system prompt" is an illusion. On my side it lives in the same window as user input; nothing guarantees it's invisible — whether it can be extracted is just a question of how much effort the attacker spends.
  • Extraction detection is cat-and-mouse. Filter one phrasing and another language / encoding / role may route around it; it raises cost, it doesn't give a boundary.
  • Treating the context as confidential costs architecture. The real boundary has to land on backend authn, secret management, and permission-scoped retrieval — that's engineering work, with no "write a prompt and call it isolation" shortcut.
  • Keep the boundaries straight. This entry only covers "what's in the current context window being extracted"; training memorization is Volume 2, un-retrieved vector-store items are Volume 4, provider retention is Volume 6 — all privacy-related, but different attack surfaces and mitigations.

How this differs from neighboring techniques​

  • Context-surface privacy vs. RAG retrieval leakage (Volume 4): RAG is about "the retrieval system pulling private data it shouldn't into the context" (lives in the retrieval / storage layer); this entry is about "what's already in the context window being extracted out" (lives in the context / interaction layer). Opposite directions, often stacked: wrongly retrieved in, then extracted out.
  • Context-surface privacy vs. training-data extraction (Volume 2): extraction is about "what was learned into weights at training time being regurgitated verbatim"; this entry is about "what's in this inference's context being extracted" — one in the weights, one in the context window, with different surfaces and mitigations.
  • Context-surface privacy vs. inference-service data boundary (Volume 6): that one defends against "will the provider retain / train on / hand off what you send" (you're trusting the provider); this entry defends against "can the end user / attacker extract what you put into the context" (you're defending the end-user side). Both are needed, against different adversaries.

Version notes​

Applicable versions

"The system prompt / context is not confidential" is a model-independent, paradigm-level fact — the root cause is that LLMs concatenate system, developer, user, tool, and retrieval inputs into one continuous token sequence with no enforced confidentiality tier, which is common across vendors. The specific extraction success rate and which bypass works evolve with models and defenses; this section is stamped 2026-06; both sides are moving, so verify conditions at the source before citing any specific success rate. OWASP version is 2025 (LLM07:2025). (Sources verified 2026-06.)

Further reading and sources​

Primary: Research (the empirical cross-model extraction); Supplementary: Security advisory (OWASP LLM07:2025, the governance basis).