Chapter 21: Incidents & Lessons Learned

On June 4, 2025, Asana found a bug in its MCP server that could expose data from one organization to MCP users in other organizations. It took the server offline while it fixed the code, and notified affected customers; follow-up reporting estimated that roughly 1,000 organizations were potentially affected. Public reporting described an internal logic flaw, not an external compromise. Nobody broke the authentication. The tenant boundary inside an already-authorized system was wrong, and connected AI workflows could traverse projects, tasks and people quickly once it was.

That is what MCP incidents look like: the same things that fail everywhere else in software, with a sharper edge. MCP concentrates capability. A single connection can expose tools, resources, approval prompts, OAuth state, tenant-scoped data, and sometimes a bridge into developer machines or production systems. Ordinary bugs become unusually high-impact incidents.

The rule for this chapter: every incident is fixed twice: once in the code, and once in the interface that made the bug expensive.

The server I run for Atelier customers, which lets them edit their own live websites through ChatGPT, Codex or Claude, is built around the second fix. The site a call may touch never comes from the model. It comes from a chain: the Clerk access token, its org_id claim, a crm_accounts row for that organization, the repository slug on that row, one directory under /home/john/sites/, and a policy that decides which relative paths inside it are editable. A token without an organization gets nothing; the code fails closed instead of falling back to the user. Every customer gets the safe tier, which can write only Markdown, YAML and CSS files, and only under content/ or static/. None of that stops a bug. It decides how much one bug can reach. Each incident below shows the pattern, then the control.

Four Incidents, Four Boundaries

Asana: the tenant boundary

Facts. A tenant-scoping flaw, found June 4, 2025; server offline until fixed; roughly 1,000 organizations potentially exposed.

Pattern. Authority boundaries break when a request arrives with a valid identity and the system still does work for the wrong tenant, workspace or environment. That usually happens in one of four ways: a scope lookup that depends on client-supplied identifiers, a cache entry not partitioned by tenant, a background job that loses the original principal, or an admin convenience path that bleeds into ordinary execution. In a human UI the bug leaks one screen. In an agent loop it leaks everything the model can follow references to.

Control. An MCP server cannot inherit tenant isolation from the application behind it. Derive the tenant once, from the token, and carry it into every cache key, query, cursor and job; Chapter 25 lists the layers. My Atelier server resolves the slug a customer asks for against the slugs their organization owns before it touches the filesystem, and rejects a slug containing /, \ or .. before resolving it.

mcp-remote: remote metadata as executable input

Facts. In July 2025, JFrog disclosed CVE-2025-6514 in mcp-remote: OS command injection when connecting to an untrusted MCP server, through a crafted authorization_endpoint URL in the server’s OAuth metadata.

Pattern. Remote metadata was treated as descriptive when it was executable input. An MCP client or bridge has to discover endpoints, launch browser flows and hand data between processes; if it feeds a field that should be a URL into a shell, a remote server turns protocol metadata into code execution on the operator’s machine. Any place your stack turns a remote protocol field into local process behavior is a code-execution boundary: shelling out with interpolated arguments, opening arbitrary schemes in helper applications, trusting redirect URIs loosely, feeding remote content into templated commands.

Control. Parse URLs with a real URL parser and reject every scheme except https:. Launch browsers and helpers with structured argument lists, never a shell string. Treat an unknown MCP server like any untrusted internet service. If you build a client, bridge, proxy or test harness, this incident should permanently change how you handle discovery metadata.

MCP Inspector: the workstation is in the threat model

Facts. CVE-2025-49596, fixed in MCP Inspector v0.14.1, was a critical remote code execution chain through the Inspector’s proxy server and its browser-facing behavior, reachable when developers ran the tool with unsafe local assumptions.

Pattern. A safe default that is only documented is not a control. If the safe mode needs the developer to notice a warning, bind the right address, or understand browser-to-localhost edge cases, many installs will eventually be wrong. And a locked-down production environment does not cover this path: it starts on a laptop, before production exists. When a product depends on “people should know better,” an incident report is usually just waiting for a date.

Control. Make local tools fail closed: bind to loopback by default, require an explicit flag to widen exposure, generate ephemeral tokens for local UIs, and make insecure modes loud. Review onboarding docs with the same skepticism as runtime code.

The same boundary covers every local server a developer installs. A config entry of "command": "npx", "args": ["-y", "some-server"] downloads the latest published version of a package and runs it as the user, with their environment variables, their SSH keys and their home directory. Chapter 18 tells stdio servers to read credentials from the environment, so the environment is exactly where the credentials are. The spec’s security guidance says a client offering one-click local install must show the exact command, untruncated, and get explicit approval before running it. That is the client’s job. Yours, as an operator, is to not depend on it.

Pin local servers to an exact version ([email protected]), install them from a lockfile, and never use a floating latest. Give each server only the environment variables it needs instead of the whole shell. Run servers that touch untrusted content in a container or sandbox with no home-directory mount. Review a version bump like a dependency upgrade, because it is one. As a publisher, ship a lockfile and a signed release so the people running your server can do all of that.

Filesystem reference server: paths are not strings

Facts. Two 2025 advisories hit the Filesystem reference server. CVE-2025-53110 bypassed path validation through colliding prefixes. CVE-2025-53109 escaped the allowed directories through symlinks inside them. The intent, confining access to allowed directories, was sound. The details were not.

Pattern. The developer knows a file tool needs guardrails, adds an allowlist, compares strings, and feels done. Then /allowed turns out to be a prefix of /allowed-but-not-really, a symlink inside the root points outside it, and normalization differs by platform. Pathname security fails when it is treated as a string problem.

Control. Resolve the real path, then compare path components against the real root:

from pathlib import Path, PurePosixPath
from urllib.parse import unquote

class PathEscape(ValueError):
    pass

def resolve_inside(root: Path, requested: str) -> Path:
    """Return the real path for `requested`, or raise if it leaves `root`."""
    if unquote(requested) != requested:
        raise PathEscape(f"percent-encoded path: {requested!r}")
    rel = PurePosixPath(requested)
    if rel.is_absolute() or ".." in rel.parts:
        raise PathEscape(f"must be relative, without '..': {requested!r}")
    base = root.resolve(strict=True)
    target = (base / rel).resolve()          # follows every symlink
    if not target.is_relative_to(base):      # compares components, not characters
        raise PathEscape(f"escapes the root: {requested!r}")
    return target

Then keep a hostile table and run it on every change. With root = /srv/allowed, a sibling directory /srv/allowed-but-not-really, and a symlink /srv/allowed/link pointing to that sibling:

My Atelier server’s safe_path does the same resolve-then-compare, and its tests include the absolute path, the .. escapes and the symlink escape. The second fix is the tier. Even a path that resolves correctly inside the site must still be under content/ or static/ with an allowed extension, and a Markdown file containing a <script> tag, an inline on...= handler or a javascript: URI is refused. Reads are narrower too: the safe tier cannot read .env files, keys, .git or deploy/. The contrast with a run_anything(command) tool is the point of Chapter 20’s minimization rules: a narrow tool with a bug leaks a little; a generic one with the same bug leaks everything.

AgentSmith in LangSmith: configuration is attack input

Not an MCP incident, but the same shape. Noma Security reported that a shared agent in LangSmith’s Prompt Hub could carry a proxy setting that routed a user’s traffic through an attacker, exposing prompts, API keys and uploads. LangChain deployed a fix on November 6, 2024, eight days after the report; Noma published the research on June 17, 2025. For MCP the lesson is that server cards, imported agent definitions, proxy settings and tool descriptions steer behavior as much as user input does. Validate them as untrusted input, and hash them as Chapter 20 describes.

Turning Incidents Into Design Rules

Appendix H catalogs these incidents with four more classes, each with its detection signal and first containment move.

Bind identity early and carry it explicitly

Derive one context near the edge and pass it everywhere. Use the same AuthContext as Chapter 19, and add a request ID beside it:

interface AuthContext {
  subject: string;
  clientId: string;
  tenantId: string;
  scopes: string[];
  roles: string[];
}

interface RequestContext {
  auth: AuthContext;
  requestId: string;
}

function buildContext(auth: AuthContext): RequestContext {
  return { auth, requestId: crypto.randomUUID() };
}

There is no approval field. Whether a call needs a human depends on the tool and its arguments, not on who is calling, so it is decided per call: needsApproval(ctx, tool, args). Every cache key, query, queue message and audit record carries enough of this context to prove which tenant and principal it belongs to.

Normalize dangerous inputs before policy checks

If the input is a path, resolve it. If it is a URL, parse it. If it is a hostname, canonicalize it. If it is a redirect URI, compare exact values. If a parameter later reaches a subprocess, turn it into a fixed argument list. The bugs above were not deep. They were failures to normalize before a trust decision.

Keep three questions separate

1. Is this caller authenticated?
2. Is this principal authorized for this tenant, workspace, or record?
3. Does this action need an explicit human confirmation step?

A prompt saying “allow this tool call?” does not repair bad tenant isolation or an over-broad backend token. Collapse the three into one fuzzy control and incident analysis becomes impossible.

Make containment a first-class feature

Operators need to narrow blast radius without an outage: kill switches per tool family, tenant-level isolation toggles, fast grant revocation, feature flags for risky integrations, and dashboards that slice activity by tool, tenant and principal. If your only emergency control is “take down the whole service,” you are not ready for a serious incident.

Keep a hostile-path suite

The highest-value security tooling is usually your own integration suite, run on every change: cross-tenant attempts, malformed OAuth discovery metadata, symlink and prefix path tricks, prompt-injection-driven tool chains, and approval bypass attempts. The table above is one page of it.

Tabletop: A Cross-Tenant Leak on the tickets Server

Run this exercise before you need it. The tickets server caches search_tickets results for thirty seconds to spare the ticket database. A refactor builds the cache key from (query, status) and drops auth.tenantId. An Acme engineer searches for “refund” and gets back two tickets from Globex, cached a few seconds earlier.

1. Detection

The first signal is usually a support report (“I can see another customer’s tickets”), not an alert. Other early signals worth wiring up:

• a spike in one high-risk tool’s invocations,
• repeated 401 or 403 errors right after an auth rollout,
• an unusual pattern of file reads or shell-like actions,
• a downstream team reporting actions that users deny initiating.

Detection improves with a log line for every tool call carrying timestamp, request ID, subject, client ID, tenant, tool, outcome, and a scrubbed list of affected object IDs. A useful cross-check here: any result row whose tenant differs from auth.tenantId is a bug by definition, so log it and alert on it. The common logging failure is too much raw data and too little decision context: privacy risk plus poor forensics. Chapter 20’s per-tool allowlist avoids it.

2. Containment

Prefer the smallest cut that stops the harm: disable search_tickets and flush its cache, rather than the whole server. An identity-aware proxy such as Pomerium can block one route per identity without a redeploy. MCP adds one step that web incidents do not have. The leaked rows are already in the model’s context window, and the host may keep acting on them for the rest of the session. Revoke the affected grants so those sessions end, and tell customers that answers given during the window may have used another tenant’s data.

3. Repair

Fix the code: put tenantId back in the key. Then fix the interface. Build every cache key through one function that requires an AuthContext, so a key without a tenant does not compile. Add the Acme-then-Globex search to the hostile-path suite. If the fix is only the missing field, the review is not done.

4. Communication

Tell affected customers what happened, which data may have been exposed, what is confirmed and what is still under investigation, what protections are in place, and what they should do now. For an MCP incident that often means concrete steps: reconnect the integration, review what the assistant told them during the window, and re-approve access after the fix.

5. Post-incident review

Answer which trust boundary failed, which control should have caught it, why detection was early or late, how containment performed, and what engineering or product change reduces repeat risk. For this tabletop the answers are short: the tenant boundary failed in the cache layer; the typed cache-key function should have caught it and did not exist yet; detection depended on a customer noticing; containment was one tool, not the server; and the permanent change is the typed key plus a test that fails if any result row’s tenant differs from the caller’s. If the review ends with “engineers need to be more careful,” you did not finish the job.

Implementation Checklist

Appendix F is the full pre-release list; these are the items the incidents above prove.

• Every request builds one AuthContext from the token and carries it through caches, queries, cursors and jobs.
• Remote metadata is parsed with structured libraries; non-https: schemes and shell interpolation are rejected by design.
• File and repository tools resolve real paths and compare components, and a hostile-path table runs in CI.
• Local servers are pinned to exact versions and run with only the environment they need.
• Each high-risk tool family has a kill switch that works without a full shutdown.
• Grant revocation for one client or one tenant is a tested runbook step.
• A tabletop exercise has been run for one cross-tenant leak and one over-broad execution scenario.

What Comes Next

Every incident here was fixed twice: once in the code, and once in the interface that made the bug expensive. That is why serious MCP engineering looks so conventional in the best sense. Strong identity binding. Narrow tools. Canonicalized inputs. Default-deny posture. Clear approval semantics. Good logs. Fast containment.

None of it helps if the server cannot be deployed, found and health-checked like any other service. Chapter 22 moves from incident lessons to deployment: packaging, discovery, health checks, and rollout discipline for a remote MCP server.