Skip to main content

AI coding security: a threat-model index

This is the fourth browsing axis, after browse by role, the tool matrix, and the mechanism index: every **security- and governance-**related pitfall in the book, regrouped by threat dimension into a single threat model.

Security pitfalls are naturally scattered across the lifecycle phases (setup, implementation, acceptance…), and read one at a time each looks like an isolated trap; placed in a threat model, you'll see they're really different slices of the same attack surface. A useful starting point is Simon Willison's the lethal trifecta — private data access + exposure to untrusted content + the ability to communicate externally — an agent with all three is exploitable; most of the dimensions below are about locking down one of those three legs. At the mechanism level, most of these land in the mechanism index's "Tool & permission overreach" bucket.

This page is a curated index only (one line per entry, linking into the existing pitfall / case), not new per-entry metadata. Scan down by dimension, then click through for "why this happens" and "how to defend against it."

Minimum security baseline​

The sections below are organized by attack surface; if you just want a checklist you can stand up right now, configure these 8 first — they compress the defenses from every dimension below into one "minimum viable baseline," tool-neutral, each linking to the full version:

  1. Least privilege by default: read-only where you can, scoped to a directory rather than the whole disk — see over-permissioning, permissions config template.
  2. Confirm writes, allow reads: set "edit a file / run a command" to require confirmation and let read-only queries through, so approval fatigue doesn't push you into clicking yes all the way down — see permissions config template.
  3. Force human confirmation for delete / push / install-dependency / outbound requests: these irreversible or spillover actions never go on autopilot — see destructive / irreversible actions, autonomy risk-tier checklist.
  4. Treat unfamiliar / untrusted repos as read-only first: read them as potentially-poisoned input before granting execution rights — see hijacked by poisoned content at dev time.
  5. No outbound network while private data is in hand: cut the "external communication" leg of the lethal trifecta — see unbounded network egress, no data-boundary policy.
  6. Keep production credentials out of the dev agent: use stand-in / read-only credentials in development, keep the real ones where it can't reach — see no data-boundary policy, shipping vulnerabilities / leaking sensitive data.
  7. Plan high-risk tasks before executing: have me lay out what I'm about to do for your review instead of improvising as I go — see letting go without plan mode, missing autonomy / approval boundaries.
  8. Append-only audit logs, escape before writing: so you can reconstruct "what I actually did" afterward, and the log itself can't be poisoned or tampered with — see log injection / audit-trail integrity.

These 8 are a floor, not a ceiling: meeting them only means you haven't skipped the most basic guardrails; your specific blast radius will need more on top (see the dimensions below). For checklists you can tick off, see the companion artifacts at the bottom.

Permission & autonomy boundaries​

Core threat: I'm granted permissions that are too broad, or high-risk actions go unapproved, and so destructive, irreversible operations happen before you can react.

Positive defense: least privilege + risk-tiered approval gates + plan/review gates, keeping irreversible actions on the far side of execution.

Isolation & sandboxing​

Core threat: only "soft permissions" hold me back, with no OS-level wall — once breached, I can escape to the whole machine's filesystem, credentials, and network.

Positive defense: fence the filesystem with kernel-level isolation (containers / bubblewrap / seccomp) and fence the network egress with an allowlist — the two come as a pair.

Untrusted content & injection​

Core threat: I read external content (READMEs, issues, web pages, tool returns, logs), and malicious instructions hidden inside can hijack my behavior.

Positive defense: keep untrusted input on the far side of execution, and pair it with isolation and egress control to limit the blast radius once an injection lands.

Data boundaries & exfiltration​

Core threat: I use data that should never leave the company as ordinary context, or write code that leaks sensitive data — once it's past the boundary, you can't pull it back.

Positive defense: set data classification first (what must never reach an external AI), gate it with a mechanism at the boundary, and on the code side keep sensitive data out of logs / responses.

Audit, accountability & logging​

Core threat: logs can be poisoned into an injection surface, and they can also be written dirty / written incomplete / tampered with by me, leaving you unable to reconstruct "what the AI actually did" after the fact.

Positive defense: escape before writing, keep the audit trail append-only and tamper-resistant, and force structured records for critical actions.

Iteration & governance accountability​

Core threat: keep asking me to change things and vulnerabilities pile up; and with the ownership, liability, and copyright status of AI-generated code left unsettled, it's "all liability, no protection."

Positive defense: treat security as a continuous gate rather than a one-time check, and establish a written ownership / liability / licensing policy for AI code.

Real incidents (case library)​

Core threat: the mechanisms above aren't hypothetical — below are public, verifiable real incidents, each one tripping over one of the threat dimensions.

Positive defense: study them like "crime-scene photos," frame by frame: understand the mechanism in the entry first, then come here to see how it crashed in the real world and what it cost.

Companion artifacts​

These compress the defenses above into checklists you can use directly: Autonomy risk-tier checklist, Data classification policy template, Pre-release security checklist, Permissions config template.