Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Chapter 12: Testing Strategy

A surprising number of bad MCP server launches look fine in a demo. The team can connect with the Inspector, call one tool, see a cheerful JSON result, and conclude that the integration works. Then the first real rollout exposes the parts that demos hide: a schema that reads clearly to humans but not to models, a transport mismatch between local and remote environments, auth behavior that fails only after token expiry, or error messages so vague that the client keeps retrying the wrong thing.

That is why testing an MCP server cannot stop at “the tool ran once.” You are not only testing application code. You are testing an interface that a model must discover, interpret, and use correctly through a client that may have its own capability constraints, approval flows, timeout behavior, and transport quirks. In practice, the MCP surface is product behavior. A broken description, an ambiguous enum, or a slow tool that times out in one client but not another is a real defect, not polish.

Here is the rule this chapter is built on: if a bug would change what the model chooses to do next, it needs coverage at the MCP surface, not just in your unit tests.

Test the Server in Four Layers

A practical testing strategy for an MCP server has four layers: unit tests for domain logic, contract tests for the MCP surface, integration tests with real dependencies and transports, and a small set of client-in-the-loop or model-in-the-loop evaluations. The layers exist because they cost different amounts. Unit and contract tests are cheap and fast, so they run on every change. Integration tests are heavier and run per merge. Model-in-the-loop tests are the expensive ones, in time and money, and also the noisiest, so they run on a schedule or as a release gate for a few important journeys. Put each risk in the cheapest layer that can catch it; Appendix L maps a kind of change to its layer. If you skip a layer entirely, you are choosing to discover that class of bug in production.

Every example in this part uses the same tickets server and its search_tickets tool, so the four layers build on one codebase.

Layer 1: Unit Tests for Domain Logic

Start below the protocol. Most of your business logic should be testable without booting an MCP server process at all.

If your search_tickets tool applies permission checks, normalizes filters, talks to a ticket backend, and transforms records into a model-facing shape, you should be able to test most of that as ordinary application code. The tool wrapper should be thin. The hard work should live in functions or services that can be exercised with plain unit tests: validation and normalization, authorization, the transformation from backend payloads to MCP content, and the translation of failures into stable error categories.

Here is the underlying idea: if every bug requires a full MCP client to reproduce, your design is too entangled.

A Python MCPServer tool might look like this when kept testable:

from typing import Literal

from mcp.server.auth.middleware.auth_context import get_access_token
from mcp.server.mcpserver import MCPServer
from mcp.server.mcpserver.exceptions import ToolError

STATUSES = ("open", "closed", "pending")


class TicketService:
    def __init__(self, api):
        self.api = api

    async def search(self, owner: str, status: str) -> list[dict]:
        if status not in STATUSES:
            raise ValueError(f"status must be one of: {', '.join(STATUSES)}")
        records = await self.api.fetch_tickets(owner=owner, status=status)
        return [{"id": r["id"], "title": r["title"], "status": r["status"]} for r in records]


def token_subject() -> str:
    token = get_access_token()
    if token is None or token.subject is None:
        raise ToolError("Not signed in. Reconnect the tickets server and sign in again.")
    return token.subject


def build_server(api, whoami=token_subject) -> MCPServer:
    mcp = MCPServer("tickets")
    service = TicketService(api)

    @mcp.tool()
    async def search_tickets(status: Literal["open", "closed", "pending"] = "open") -> dict:
        """Search support tickets assigned to the signed-in user, filtered by status."""
        try:
            return {"tickets": await service.search(owner=whoami(), status=status)}
        except ValueError as exc:
            raise ToolError(str(exc)) from exc
        except TimeoutError as exc:
            raise ToolError("Ticket backend timed out. Retry the same search once.") from exc

    return mcp

The owner of the tickets is not a tool argument. It comes from the validated access token on the request, because identity comes from the session, never from the model (Chapter 18). build_server takes the backend and the identity lookup as parameters, which is what lets every later layer swap them out.

The unit tests should hammer TicketService.search() long before they ever touch the MCP transport, including the error path:

class FakeAPI:
    async def fetch_tickets(self, owner, status):
        return [{"id": "T-1", "title": "Deploy failed", "status": status, "body": "..."}]

async def test_search_trims_records():
    tickets = await TicketService(FakeAPI()).search(owner="u_alice", status="open")
    assert tickets == [{"id": "T-1", "title": "Deploy failed", "status": "open"}]

async def test_search_rejects_unknown_status():
    with pytest.raises(ValueError, match="status must be one of"):
        await TicketService(FakeAPI()).search(owner="u_alice", status="urgent")

The first test pins what the model will see: three fields, not the backend’s full record. The second pins the message the model gets when it guesses a status that does not exist. Both run in milliseconds and fail with a precise diff.

Layer 2: Contract Tests for the MCP Surface

Once the core logic is solid, test the interface the client and model actually see.

This layer is easy to skip, even though it is where a lot of MCP breakage lives. The official SDKs make it easy to register tools, resources, and prompts, but easy registration is not the same thing as a stable interface. A rename from repo to repository, a schema description that silently disappears during refactoring, or a tool that widens its behavior without changing its description can change how a model behaves even when the backend remains correct.

Contract tests should verify the server’s published surface, including:

  • tool names,
  • descriptions,
  • input schemas and required fields,
  • resource URIs and MIME types,
  • prompt argument contracts,
  • annotations and metadata where you rely on them,
  • and structured output shapes where supported.

The Python SDK’s Client accepts an MCPServer instance and connects to it in-process, so a contract test sees exactly what a real client would receive from tools/list, with no subprocess and no network:

from mcp import Client

async def test_search_tickets_contract():
    server = build_server(api=FakeAPI(), whoami=lambda: "u_alice")
    async with Client(server) as client:
        tools = {t.name: t for t in (await client.list_tools()).tools}

    search = tools["search_tickets"]
    assert search.description.startswith("Search support tickets")
    status = search.input_schema["properties"]["status"]
    assert status["enum"] == ["open", "closed", "pending"]
    assert "owner" not in search.input_schema["properties"]  # identity is never an argument

The last assertion is the kind that earns this layer its place. A well-meaning refactor that adds owner back as an argument would pass every unit test and every demo, and it would hand the model control over whose tickets it reads. Treat the MCP surface as an API contract, because that is what it is.

Snapshot tests can help here, but use them carefully. A full JSON snapshot of every tool definition often becomes noisy and gets approved blindly. A better pattern is targeted assertions on the parts that matter for model behavior: names, descriptions, required fields, enums, limits, and structured result fields.

Layer 3: Integration Tests With Real Dependencies and Real Transport Behavior

After the contract layer, you need tests that prove the pieces work together. This means running the MCP server the way it will actually run and exercising real seams: authentication, network behavior, serialization, dependency failures, and transport-specific behavior.

This layer matters because many MCP server bugs are not logic bugs. They are environment bugs.

Test the transport you deploy. If you develop on stdio and ship Streamable HTTP, the transport bugs (proxy headers, idle timeouts, token refresh) will show up for the first time in production.

Integration tests should therefore answer questions like:

  • Does the MCP server start with the environment variables your deployment actually provides?
  • Does auth work after the first token expires, not just on first connect?
  • Does the server respect client timeouts and cancellation?
  • Does a large resource or tool result still serialize correctly?
  • Do progress notifications behave the way the client expects?
  • Does a reverse proxy preserve the headers your auth layer needs?

Layer 4: Client-in-the-Loop and Model-in-the-Loop Tests

The top layer is where you test the behavior that only emerges once a real client and, sometimes, a real model are involved.

These are related but different concerns.

Client-in-the-loop tests answer questions like:

  • Can the target client discover the MCP server?
  • Does it expose the right tools to the model?
  • Does approval UX interrupt the workflow?
  • Does auth or OAuth complete successfully?
  • Do long-running requests time out or render progress correctly?

Model-in-the-loop tests ask a different question: given the published surface, does the model use it well?

This is the layer that catches failures such as:

  • a tool description that is technically correct but too vague,
  • parameters that are valid JSON Schema but semantically confusing,
  • error messages that do not help the model recover,
  • duplicate tools that create indecision,
  • and result payloads that are too verbose or too sparse for reliable follow-up actions.

You do not need model-in-the-loop tests for every tool. That is usually wasteful. Use them for workflows where incorrect model behavior is expensive: a deployment tool, a finance action, a data mutation flow, or a multi-step support workflow with strict policy rules.

Write them as scenarios, not transport checks: give the model a realistic request, check which tool it chooses and the arguments it sends, inject a controlled failure, and check whether it recovers. Here is one for search_tickets, where the backend times out on the first call:

MODEL = "claude-sonnet-4-5-20250929"  # pinned snapshot; bump it in a reviewed commit
TRIALS, GATE = 10, 9
pytestmark = [pytest.mark.anyio,
              pytest.mark.skipif(not os.getenv("ANTHROPIC_API_KEY"), reason="no model key")]


class FlakyAPI:  # injected failure: the first backend call times out
    calls = 0

    async def fetch_tickets(self, owner, status):
        self.calls += 1
        if self.calls == 1:
            raise TimeoutError
        return [{"id": "T-7", "title": "VPN down", "status": status}]


async def run_trial(llm) -> tuple[bool, list]:
    async with Client(build_server(FlakyAPI(), whoami=lambda: "u_alice")) as mcp:
        tools = [{"name": t.name, "description": t.description, "input_schema": t.input_schema}
                 for t in (await mcp.list_tools()).tools]
        messages = [{"role": "user", "content": "What's still open in my ticket queue?"}]
        calls = []
        for _ in range(4):
            reply = await llm.messages.create(model=MODEL, max_tokens=512,
                                              tools=tools, messages=messages)
            messages.append({"role": "assistant", "content": reply.content})
            uses = [b for b in reply.content if b.type == "tool_use"]
            if not uses:
                break
            results = []
            for use in uses:
                calls.append((use.name, use.input.get("status", "open")))
                r = await mcp.call_tool(use.name, use.input)
                results.append({"type": "tool_result", "tool_use_id": use.id,
                                "is_error": r.is_error, "content": r.content[0].text})
            messages.append({"role": "user", "content": results})
    answer = " ".join(b.text for b in reply.content if b.type == "text")
    return calls == [("search_tickets", "open")] * 2 and "T-7" in answer, messages


async def test_search_recovers_from_backend_timeout():
    llm, passed, out = anthropic.AsyncAnthropic(), 0, Path("eval-transcripts")
    for i in range(TRIALS):
        ok, transcript = await run_trial(llm)
        passed += ok
        if not ok:
            out.mkdir(exist_ok=True)
            (out / f"trial-{i}.json").write_text(json.dumps(transcript, indent=2, default=vars))
    assert passed >= GATE, f"{passed}/{TRIALS} passed; transcripts in {out}/"

A trial passes only if the model picks search_tickets with status="open", reads the timeout error, retries exactly once, and reports the ticket. Calling a different status, giving up, or retrying in a loop all fail it.

Four choices make this a test rather than a demo. First, model output is probabilistic, so one run proves nothing in either direction. The eval runs ten trials and gates on a pass rate of at least nine, which tolerates one unlucky sample without hiding a real regression. Second, the model is pinned to a dated snapshot. When a test that passed yesterday fails today, the cause should be your diff, not a silent model update, and moving to a new model becomes a reviewed change with its own eval run. Third, every failing trial writes its full transcript to disk. A pass rate says something broke; the transcript says whether the model misread the description, ignored the error, or looped. Fourth, the test skips when no key is set, so it never blocks a contributor’s local run and never runs by accident in a cheap CI tier. Ten trials of a four-turn conversation is real spend, which is why this layer covers a handful of journeys, not every tool.

Where the Inspector Fits

The MCP Inspector deserves a permanent place in your workflow. It spans layers 2 and 3: it lets you verify connectivity, inspect resources, test prompts, invoke tools with custom arguments, and watch logs without needing a full host application in the loop.

That makes it perfect for three jobs.

First, it is a fast development harness. You can iterate on tool descriptions, schemas, and result shapes in minutes. Second, it is an integration debugger. It helps answer whether the problem is in your server, your transport, or the client you were previously using. Third, it is a release gate. Before shipping, you can confirm that the published interface still looks the way your documentation claims.

But be precise about what the Inspector proves. It proves that a capable MCP client can connect and interact with your MCP server. It does not prove that Claude Desktop, Cursor, ChatGPT, an internal enterprise host, or your own custom client will behave the same way. Capability negotiation, approval UX, auth support, cancellation semantics, and timeout defaults can differ. That gap is what Layer 4 exists to close.

It also has security implications. The Inspector’s proxy can spawn local processes and should not be exposed to untrusted networks. It defaults to localhost binding and session-token authentication, which is the right posture. Your test tooling is part of your attack surface too.

Failure Modes Worth Testing on Purpose

Teams usually test the happy path and assume the rest will behave proportionally. In MCP work, that assumption is expensive. The ugly cases are often where the model loses the plot.

Test these on purpose.

Ambiguous or Missing Descriptions

A tool can be perfectly implemented and still fail if the description does not make its purpose, scope, or constraints obvious. Models infer intent from names and descriptions. If those are muddy, behavior will be muddy.

Schema Shapes That Invite Wrong Calls

A schema with too many optional fields, overloaded meanings, or weak enums can cause the model to guess. When a model guesses, it often guesses confidently.

Timeout and Cancellation Mismatch

Client-side timeouts and server-side timeouts are separate concerns. If the client cancels after thirty seconds but your server keeps running for five minutes, you now have abandoned work, duplicate retries, or both.

Auth Expiry and Reauthentication Loops

Many auth demos only prove first login. Real failures happen after token expiry, missing scopes, revoked sessions, or partial reauthentication.

Partial Failure in Multi-Step Tools

If a tool aggregates data from three systems and one of them fails, what does the model see? A useful partial result with clear warnings, or a generic internal error? The latter usually leads to bad follow-up behavior.

Implementation Checklist

Before you call your testing strategy done, make sure your MCP server has all of the following:

  • Unit tests for business logic outside the transport layer, including every error message the model can receive
  • Contract tests for tool names, descriptions, schemas, resources, and prompts, run through an in-process client
  • Integration tests that run the actual transport you plan to deploy
  • A repeatable MCP Inspector verification flow
  • At least one client-in-the-loop test for each supported host or client family
  • A small, curated set of model-in-the-loop evaluations for high-risk workflows, each with a pinned model, N trials and a pass-rate gate
  • CI gates that distinguish fast checks from expensive checks
  • Release notes or versioning discipline when the MCP surface changes

If even half of that feels like overkill, go back to the rule: if a bug would change what the model chooses to do next, it needs surface-level coverage. Naming regressions, missing descriptions, malformed payloads, unhelpful errors, and timeouts that cause retries all qualify. In MCP, these are not edge cosmetics. They shape agent behavior.

What Comes Next

A server that behaves correctly can still behave expensively. The next chapter turns from correctness to economics: how much context the surface consumes, how much latency it adds, and how to keep a useful MCP server from quietly becoming a costly one.