Chapter 23: Observability & Resilience

An MCP server is not operable just because it returns correct responses on a good day.

It becomes operable when the team can answer, under pressure, what happened, for whom, why it happened, and what safe fallback exists.

These systems are harder to reason about than ordinary APIs because failures cross boundaries. The model may choose poorly. The client may render approvals awkwardly. The server may validate correctly and then hit a weak downstream dependency. Auth may succeed at the gateway and fail in the application. A long-running task may complete after the user has already retried twice.

Without structured visibility, those all collapse into “the server feels weird.”

The operator rule for this chapter: the server never sees the model’s prompt or reasoning, only its calls, so log every call in a shape that lets you reconstruct what the model was trying to do. The rest of the chapter is that shape and what it lets you detect.

What You Need to Measure

Start with the smallest set of signals that explain real operating behavior.

If you only track total request count and average latency, you will miss the problems that matter.

One Event Per Tool Call

Most real incidents are chain reactions, and if every layer names the same event differently, the investigation slows immediately. Fix the vocabulary once, as one structured event the tickets server writes for every tools/call:

{
  "ts": "2026-09-28T14:03:11.482Z",
  "event": "tool_call",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "request_id": 41,
  "tool": "search_tickets",
  "args_hash": "sha256:9f2c61e0",
  "principal": "user_8841",
  "client_id": "https://client.example.net/oauth/metadata.json",
  "tenant": "acme",
  "outcome": "validation_error",
  "error_code": "invalid_status",
  "dependency": "tickets-db",
  "latency_ms": 38,
  "retry_count": 1,
  "protocol_version": "2026-07-28",
  "client_name": "ExampleClient",
  "client_version": "3.4.1",
  "catalog_version": "4.2.0",
  "token_aud": "https://mcp.example.com/mcp"
}

Each field earns its place:

• trace_id and span_id join this call to the client’s trace and to every downstream call (next section).
• principal, client_id, and tenant come from the validated AuthContext, never from tool arguments.
• outcome is a closed set: success, empty, validation_error, auth_error, policy_denied, dependency_error, timeout. error_code narrows it. Free-text outcomes cannot be grouped.
• args_hash is a hash of the normalized arguments. It lets you detect repeated identical calls without storing ticket text or customer data.
• retry_count counts calls with the same args_hash and principal in the last five minutes, computed at ingress. The client does not report it.
• protocol_version, client_name, and client_version come from the request’s _meta (io.modelcontextprotocol/protocolVersion and io.modelcontextprotocol/clientInfo). They are self-reported and unverified, but they are the fastest way to tell “our deploy broke it” from “a client release changed behavior.”
• catalog_version identifies the tool descriptions and schemas the server was serving. Without it, you cannot tell a behavior change caused by a description edit from one caused by traffic.
• token_aud is the audience of the presented token, logged on 401s too. It is the field that exposes the vanity-domain loop from Chapter 22.

What stays out: raw tokens, secrets, full request or response payloads, and anything you would not paste into an incident channel. When you need more detail during an investigation, raise sampling for one tenant or tool, log that you did so, and set an expiry.

Tracing Crosses the Client Hop in _meta

Distributed tracing matters for MCP because “slow” usually hides in a seam: approval UX, the model, the MCP server, or the downstream API. The seam that trips teams up is the one between client and server.

The 2026-07-28 revision reserves three _meta keys for OpenTelemetry context: traceparent, tracestate, and baggage, with W3C Trace Context values and no io.modelcontextprotocol/ prefix. The rule at ingress:

1. If params._meta.traceparent is present and well formed, continue that trace.
2. Otherwise, if an HTTP traceparent header is present (a gateway may have added one), continue that.
3. Otherwise, mint a new trace at ingress.
4. Put the resulting trace_id on the log event, and send traceparent on every downstream call the tool makes.

import re
from opentelemetry import trace
from opentelemetry.propagate import extract

TRACEPARENT = re.compile(r"^[0-9a-f]{2}-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}$")
tracer = trace.get_tracer("tickets")

def carrier_for(meta: dict, headers: dict) -> dict:
    for source in (meta, headers):
        tp = source.get("traceparent")
        if isinstance(tp, str) and TRACEPARENT.match(tp):
            return {k: source[k] for k in ("traceparent", "tracestate", "baggage") if k in source}
    return {}  # empty carrier: start_as_current_span mints a new root

def traced_call(tool: str, meta: dict, headers: dict, run):
    ctx = extract(carrier_for(meta, headers))
    with tracer.start_as_current_span(f"tools/call {tool}", context=ctx) as span:
        return run(span)

Treat inbound trace context as untrusted input: validate the format, and never use baggage for anything that affects authorization.

Seeing the Model From the Server

Failures that cross the model boundary are the ones a standard SRE dashboard never reaches. The server cannot see what the model intended. It can see the pattern of calls, and four patterns are worth detecting because each points to a specific fix.

Retry loops. The same principal calls the same tool with the same args_hash several times inside a few minutes. The model is not getting what it needs from the result, and it retries by reasoning, not by a backoff policy, so it will keep going until the client or the user stops it. The usual causes are an error that says “try again” when nothing will change, an empty result the model reads as a transient failure, or a result that omits the one field the model was looking for. This is the signal the “Model Looping” runbook in Chapter 24 starts from.

A validation error followed by success. search_tickets returns validation_error with error_code: invalid_status, and within the same trace the next call to the same tool succeeds. The model guessed wrong, read your error, and recovered. That is error design working, but a high rate of first-call failures means the schema invites the wrong guess, for example a free-text status that should have been an enum. Each common first-wrong-guess belongs in Chapter 12’s contract tests.

A shift in selection share after a description change. For each catalog_version, compute each tool’s share of calls within its family. When you ship 4.2 with a reworded search_tickets description and its share drops from 60% to 25% while list_tickets rises, the model’s behavior changed with no errors at all. This is a behavioral regression you can see in production without running a model eval, and it is why catalog_version is on the event.

Calls per completed task. Chapter 13’s scenario benchmarks measured tool calls per successful task in the lab. In production, pick a terminal tool that marks completion (update_ticket with a status change, say) and count the calls in the same trace before it. When that ratio climbs from 3 to 7 after a deploy, the model is working harder for the same outcome, and you are paying for it in tokens and latency.

Here is a detector for the first pattern, over the log events loaded into a SQL table:

-- Sessions where the model repeats an identical call: 4+ times in 5 minutes.
SELECT principal,
       tenant,
       tool,
       args_hash,
       client_name,
       count(*)                                  AS calls,
       count(*) FILTER (WHERE outcome <> 'success') AS failed,
       min(ts)                                   AS first_seen,
       max(ts)                                   AS last_seen
FROM tool_calls
WHERE ts > now() - interval '15 minutes'
GROUP BY principal, tenant, tool, args_hash, client_name
HAVING count(*) >= 4
   AND max(ts) - min(ts) <= interval '5 minutes'
ORDER BY calls DESC;

The thresholds are a starting point: four identical calls in five minutes is rarely deliberate for a read tool. Tune per tool: a status-polling tool will legitimately repeat, so exclude it or give it a higher bar. Group the output by client_name before paging anyone. A loop concentrated in one client version is a client problem; a loop spread across clients is your surface.

Dashboards That Operators Can Use

The primary operational view should answer:

1. Is the service healthy right now?
2. Which tool families are failing or slowing down?
3. Which tenants, users, or workflows are driving unusual behavior?
4. Are auth or downstream systems the real source of pain?
5. What changed recently, including catalog_version and client versions?

If those answers require opening six tabs and guessing, the dashboard is decorative. For example layouts and starter alerts, see Appendix I.

Resilience Has to Produce Errors the Model Can Act On

MCP resilience is mostly standard backend resilience applied to a model-facing interface. The difference is that your caller retries by reasoning, not by a backoff policy, so every resilience pattern must end in an error the model can act on. A circuit breaker that opens and returns “internal error” produces a retry loop. The same breaker returning a tool error that says the ticket database is unavailable, that retrying within a minute will not help, and that get_ticket_cached still works produces one sensible next step.

The patterns themselves are familiar:

• deadlines and timeouts that fail clearly,
• retries with backoff only for genuinely transient failure classes,
• circuit breakers or dependency guards for unstable downstream systems,
• caching on stable read paths,
• admission control and quota protections during load spikes,
• and kill switches for risky or expensive tool families.

Apply them per tool or dependency class, not globally. A global breaker on a shared dependency, tripped by a failing write path, takes the read-only tools down with it, and the model loses exactly the tools it would have used to work around the problem. The same goes for retries: a global retry policy multiplies load on the dependency that is already failing.

Every error should tell the model whether retry is sensible, whether the failure is auth, validation, policy, or dependency, and what to do instead. Chapter 6 covers the shape. Poor errors increase load: the model retries blindly, users approve the same action again, and operators get paged for noise.

Graceful Degradation Is Part of the Product

Not every dependency failure should become a total outage:

• when an enrichment service is down, return the primary result and label the missing enhancement,
• when a write path becomes unsafe, disable it while preserving diagnostics or read-only access,
• when auth is degraded, fail with a clear recoverable error rather than an indefinite timeout,
• when one region is impaired, restrict that environment instead of silently widening access elsewhere.

Graceful degradation only works if the system can explain what still works and what does not. Drills that prove these paths belong in Chapter 24.

Failure Modes

1. No correlation ID across the client hop. The server ignores _meta.traceparent, so every call starts a new trace and one user task becomes a dozen unrelated fragments.
2. A global retry or breaker policy. One failing write path trips protection for every tool, or retries multiply load on the dependency that is already struggling.
3. Dashboards that aggregate all tools. A 40% failure rate on one tool disappears inside a 2% service-wide error rate.
4. Errors that tell the model to retry an auth failure. An expired or wrong-audience token gets “please try again”, and the model loops on a call that can never succeed.
5. No catalog version on events. A description change shifts selection share, and nobody can connect the behavior change to the deploy that caused it.
6. Logging payloads to detect loops. Full arguments land in logs to compare calls, and the log store becomes the most sensitive data in the system.

Implementation Checklist

• Every tools/call emits one event with the fields above, and outcome is a closed set.
• principal and tenant come from AuthContext, never from arguments.
• Inbound traceparent is validated and continued; a trace is minted when absent; downstream calls carry it.
• args_hash, not raw arguments, supports repeat detection.
• The loop detector runs on a schedule and groups by client_name before it pages.
• Selection share per tool is charted by catalog_version.
• Calls per completed task are tracked for at least one critical workflow.
• Breakers, retries, and kill switches are scoped per tool family, and each returns a tool error naming what still works.

The server never sees the model, only its calls. Logged in this shape, those calls are enough to see what the model was trying to do.

What Comes Next

Signals tell you something is wrong. The next chapter is what the on-call engineer does in the first ten minutes after they fire.