Chapter 24: Operations Runbooks

Runbooks are where architecture turns into repeatable behavior.

If your team can describe how the server should behave but cannot handle a bad day without improvising the basics, you do not yet have a production platform. A runbook exists to make the first ten minutes less chaotic and the next hour more disciplined.

The operator rule for this chapter: when the model misbehaves, fix the surface it reads before you touch the code it calls. Half the runbooks below exist because a plain API never has that problem.

How These Runbooks Are Laid Out

Every runbook uses the same five headings, because a reader under pressure depends on the same layout every time:

• Confirm: prove the incident is real and find its scope, with commands.
• Mitigate: the safe moves, in the order to try them.
• Do Not: the tempting moves that make it worse.
• Escalate: when this stops being a one-person problem.
• Follow-up: the durable fix and the evidence to keep.

For a symptom that doesn’t yet map to a runbook, start with Appendix E’s triage table.

The commands run against the tickets server at https://mcp.example.com/mcp. Log queries use the tool_calls table built from Chapter 23’s event. The kill switch is a per-tool flag in your flag service; the shape used throughout is:

# Disable one tool family. Callers that have used it get a tool error naming the fallback.
curl -sf -X PATCH https://flags.internal.example.com/v1/flags/tickets.bulk_update_tickets \
  -H "Authorization: Bearer $FLAGS_TOKEN" -d '{"enabled": false}'

A Simple Severity Model

Runbook: Server Unresponsive

Confirm.

curl -sf -o /dev/null -w '%{http_code}\n' https://mcp.example.com/healthz
curl -s  -o /dev/null -w '%{http_code}\n' https://mcp.example.com/readyz

200/200 with failing clients points at auth or routing. 200/503 means the process is up but a dependency or warm-up is not. No answer from /healthz means routing, TLS, or the process itself. Then check the current deploy version and recent config or secret changes.

Mitigate. Roll back the most recent suspect change; remove unhealthy instances; shift traffic away from a bad region; disable the worst tool family if the rest can serve safely.

Do Not. Restart everything repeatedly without preserving evidence; declare it fixed on liveness alone; widen access controls to get traffic moving.

Escalate when rollback does not restore service quickly, or more than one environment is affected.

Follow-up. Record which signal fired first and whether /readyz told the truth. If /readyz said 200 while clients were failing, the readiness check is missing a dependency; add it before closing the incident.

Runbook: High Error Rate or Schema Drift

Confirm.

SELECT tool, outcome, error_code, count(*)
FROM tool_calls WHERE ts > now() - interval '15 minutes' AND outcome <> 'success'
GROUP BY 1, 2, 3 ORDER BY 4 DESC LIMIT 10;

One tool with validation_error after a deploy is a contract change; dependency_error across tools is a downstream failure.

Mitigate. Rate-limit or disable the failing tool family; restore compatibility shims if a contract changed; return a safer fallback where possible; tell affected owners early.

Do Not. Raise retry limits to push through a dependency failure; loosen schema validation in production to make errors stop.

Escalate when the failing dependency belongs to another team, or errors span tenants.

Follow-up. Add the breaking case to Chapter 12’s contract tests.

Runbook: Auth or Discovery Failure

Confirm.

curl -s https://mcp.example.com/.well-known/oauth-protected-resource/mcp | jq '{resource, authorization_servers}'

SELECT token_aud, client_name, count(*)
FROM tool_calls WHERE ts > now() - interval '1 hour' AND outcome = 'auth_error'
GROUP BY 1, 2 ORDER BY 3 DESC;

An audience that is not https://mcp.example.com/mcp is the vanity-domain loop from Chapter 22. Separate 401, 403, and failures before the request reaches the server (TLS, routing, protocol-version rejection).

Mitigate. Roll back the auth or discovery configuration if it changed recently; move affected clients to a known-good endpoint; pause onboarding of new tenants; preserve logs that show which audience or issuer check failed.

Do Not. “Fix” an auth incident by temporarily weakening audience, issuer, or tenant checks unless leadership has consciously accepted the risk. That kind of shortcut creates the next incident.

Escalate to the identity team when the issuer or its metadata changed.

Follow-up. Add the per-hostname resource check to every deploy.

Runbook: Model Looping or Misusing a Tool

Confirm. Run Chapter 23’s loop detector. Check whether the loops concentrate in one client_name and version, and read the last result the model received for the looping args_hash: is it an error that says “try again”, an empty result, or a result missing the field the model wanted?

Mitigate. Cap calls per principal per tool for the session (for example 5 identical calls in 5 minutes, then a tool error saying the same arguments will return the same answer). If the loop is burning a write path, disable that tool with the kill switch.

Do Not. Block the principal; the user did nothing wrong. Do not “fix” it by adding retries or timeouts on the server.

Escalate to the client vendor if the loop follows one client release across all your tools.

Follow-up. The real fix is in the surface: the error message, the empty-result wording, or the description. Ship it as a new catalog_version and watch the detector.

Runbook: Surface Change Altered Behaviour / Stale Client Catalog

There are two symptoms. With no errors, outcomes went wrong after a deploy: selection share moved or calls per task climbed. Or clients call tools the running version does not know.

Confirm.

SELECT catalog_version, tool, count(*),
       round(100.0 * count(*) / sum(count(*)) OVER (PARTITION BY catalog_version), 1) AS share
FROM tool_calls WHERE ts > now() - interval '24 hours' AND tool LIKE '%ticket%'
GROUP BY 1, 2 ORDER BY 1, 4 DESC;

For a stale catalog, look for error_code = 'unknown_tool' grouped by client_name.

Mitigate. Roll back the surface, not the code: restore the previous descriptions and schemas as a new catalog_version. For unknown tools, deploy the removal stubs from Chapter 28 so each returns an error naming its replacement, and emit notifications/tools/list_changed for clients listening on subscriptions/listen.

Do Not. Silently remove a tool callers have used; roll back the whole service when only a description changed.

Escalate when a client ignores catalog updates and keeps calling removed tools.

Follow-up. Gate description changes on a selection-share comparison, the same way code changes are gated on tests.

Runbook: Budget or Token Spike

Confirm. Find which tenant, tool family, or client drives the volume; check the loop detector; compare calls per completed task with the last known-good window.

Mitigate. Impose temporary per-tenant or per-tool caps; disable expensive non-critical tools; force cached read paths; notify owners before the platform starts refusing work.

Do Not. Apply a global cap that throttles well-behaved tenants to contain one.

Escalate when one tenant’s spend crosses its contracted limit.

Follow-up. Decide whether the real fix is interface simplification rather than quota tightening alone (Chapter 13).

Runbook: Security Incident or Suspected Data Exposure

First priorities.

1. Stop the unsafe behavior: disable the affected tool, route, tenant, or environment.
2. Revoke compromised credentials or sessions.
3. Preserve logs, traces, task state, and deployment context.
4. Determine whether the issue is isolated or potentially cross-tenant.

Do Not. Delete or rotate logs before they are preserved; perform certainty theater. Clear uncertainty is better than false reassurance.

Communication and review follow Chapter 21’s incident playbook.

Runbook: Long-Running Task Backlog

Confirm. Are tasks queued, executing slowly, or failing on completion? Did a dependency outage turn normal work into retries? Are status paths (tasks/get) healthy?

Mitigate. Reduce new task intake; extend status messaging so clients see delay instead of retrying blindly; prioritize critical tenant queues.

Do Not. Drain or replay work without a deduplication plan. The worst response to task backlog is to create duplicated work while trying to “clear” it.

Escalate when completed work cannot be matched to its original request.

Follow-up. Record the queue depth at which clients began retrying, and alert below it.

After the Incident

After mitigation, capture the timeline, the first reliable detection signal, the runbook step that helped most, the step that was missing or unclear, and the durable change to code, surface, policy, or documentation. When the model was involved, the change is usually to the surface.

Runbooks improve by being used and corrected. The dangerous ones are the documents that look polished but do not survive contact with an actual incident.

What Comes Next

Everything so far has been about running one server well. Part VIII is what changes when the audience is an enterprise: tenants, gateways, cost attribution and change control.