MCP Server Production Design: A Tool That Takes Prose Cannot Pass a Regression Suite

Why does an MCP tool that accepts a prose argument fail its first code review?

MCP server production design reduces to one constraint. A tool that accepts a natural-language instruction has no test surface, because its input is a sample from a model's distribution, and a tool that takes typed arguments and returns a typed result is the only kind you can put behind a regression suite.

The first MCP server I shipped got this wrong in an ordinary way. It exposed a run_report tool whose schema was one field: { instruction: string }. The handler read that string and guessed — regexes for a date range, a lookup table for thirty-odd metric names, a fallback that meant "revenue by month" when nothing matched. It worked in the demo, which is the trap: a demo is a single sample, and everything works on a single sample.

It ran for two months against live reporting traffic and produced three defects that reached a person. A quarter boundary dropped one week. An account priced in EUR was totalled in USD because the handler defaulted to USD. A table silently stopped at 500 rows because of a LIMIT. None of them reproduced, because editing the agent's prompt changed what arrived at the tool. In Git the handler was untouched; in behaviour it was a different program.

An MCP tool that takes a natural-language instruction is untestable, because its input is a sample from a model's distribution and you cannot write an assertion against a distribution.

The fix is not a better parser. Move the interpretation to where it belongs — the model — and make the boundary a schema. Everything below is what that costs and what it buys.

What does a typed tool contract actually contain?

Three objects, and the model reads them in tools/list before it decides anything.

import { z } from "zod";

export const LineItem = z.object({
  sku: z.string().regex(/^[A-Z]{3}-\d{4}$/),
  quantity: z.number().int().positive().max(10_000),
  unitPriceMinor: z.number().int().nonnegative(),
});

export const QuoteRequest = z.object({
  accountId: z.string().uuid(),
  currency: z.enum(["USD", "EUR", "CNY"]),
  lines: z.array(LineItem).min(1).max(200),
});

export const QuoteResult = z.object({
  quoteId: z.string().uuid(),
  totalMinor: z.number().int().nonnegative(),
  flags: z.array(z.enum(["credit_hold", "below_floor", "manual_review"])),
});

export type QuoteRequestInput = z.infer<typeof QuoteRequest>;
export type QuoteResultOutput = z.infer<typeof QuoteResult>;

Each decision in that file maps to a defect I have actually watched happen. Money is an integer in minor units because JSON has no decimal type: once an amount is a float, the sum your server computes and the sum the ledger computes disagree by a cent. Currency is an enum rather than a string, because usd and USD are the same thing to a person and two different branches to code. The line array is bounded at 200 items, because an unbounded array invites four thousand lines and nine seconds of parsing on the request thread.

The pattern on sku is worth more than any description I could write. A model that sends ab-1 gets a validation error naming the exact path, and almost always corrects it on the next attempt. A model that reads "SKU must be three uppercase letters, a hyphen and four digits" in prose gets it right most of the time, which is the worst result: right often enough to pass a demo, wrong often enough to reach a human.

Where does the model's job end and the handler's begin?

The handler is a plain function. It receives the validated object and returns the result shape. It does not import MCP, and it does not know it is being called by an agent.

import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { QuoteRequest, QuoteResult, type QuoteRequestInput, type QuoteResultOutput } from "./schema";

export type ToolResult<T> =
  | { isError: true; content: [{ type: "text"; text: string }] }
  | { isError?: false; content: [{ type: "text"; text: string }]; structuredContent: T };

export function createQuoteHandler(repo: QuoteRepo) {
  return async ({ accountId, currency, lines }: QuoteRequestInput): Promise<ToolResult<QuoteResultOutput>> => {
    const account = await repo.findAccount(accountId);
    if (account === null) {
      return { isError: true, content: [{ type: "text", text: "unknown_account" }] };
    }
    const draft = await repo.createQuote({ accountId, currency, lines });
    const structuredContent = QuoteResult.parse(draft);
    return {
      content: [{ type: "text", text: `quote ${draft.quoteId} created` }],
      structuredContent,
    };
  };
}

export function registerQuoteTool(server: McpServer, repo: QuoteRepo): void {
  server.registerTool("create_quote", {
    description: "Create a quote from validated line items. Amounts are integer minor units.",
    inputSchema: QuoteRequest.shape,
    outputSchema: QuoteResult.shape,
  }, createQuoteHandler(repo));
}

registerQuoteTool is the only thing in that file that imports the SDK, and that separation is the whole reason the test file in the next section needs no transport, no server process and no model.

Two failure classes live here and conflating them is a defect in itself. A schema violation is the caller's problem: the model sent a string where it should have sent a quantity, and the correct response is the issue path so the next attempt can be right. A domain rejection — unknown_account, credit limit exceeded, a price below the floor — is the business's problem: retrying it spends tokens to receive the same answer. My servers return the first as a validation error and the second as isError: true with a short code, and the agent loop treats them differently: one retries once, the other stops and reports.

How do you regression-test a system whose caller is a model?

Three layers, and each one catches a class of defect the others cannot see.

import { describe, expect, it } from "vitest";
import { QuoteRequest, type QuoteRequestInput } from "../src/tools/create-quote/schema";
import { createQuoteHandler } from "../src/tools/create-quote";
import { InMemoryQuoteRepo } from "../src/repos/quote-repo";

const ACCOUNT = "8f14e45f-ceea-467a-9a2f-1b2c3d4e5f60";
const repo = new InMemoryQuoteRepo({ [ACCOUNT]: { creditLimitMinor: 5_000_000, priceFloorBps: 1_200 } });
const handler = createQuoteHandler(repo);

// The server validates before it invokes the handler; mirror that order here.
const call = async (raw: unknown) => {
  const parsed = QuoteRequest.safeParse(raw);
  if (!parsed.success) return { rejected: parsed.error.issues.map((i) => i.path.join(".")) };
  return handler(parsed.data);
};

describe("create_quote", () => {
  it("rejects a malformed sku without touching the repo", async () => {
    const res = await call({ accountId: ACCOUNT, currency: "EUR", lines: [{ sku: "ab-1", quantity: 1, unitPriceMinor: 100 }] });
    expect(res).toEqual({ rejected: ["lines.0.sku"] });
    expect(repo.writes).toBe(0);
  });
  it("totals minor units and flags a below-floor sale", async () => {
    const input: QuoteRequestInput = { accountId: ACCOUNT, currency: "EUR", lines: [{ sku: "ABC-1234", quantity: 3, unitPriceMinor: 2_500 }] };
    const res = await handler(input);
    expect(res.isError).toBeFalsy();
    expect(res.structuredContent?.totalMinor).toBe(7_500);
    expect(res.structuredContent?.flags).toContain("below_floor");
    expect(repo.writes).toBe(1);
  });
});

The in-memory repo counts writes, which is what lets the first test assert that validation rejected the call before any side effect — the assertion that matters, because a schema that runs after the insert is decoration.

The first two layers are ordinary engineering: schema tests exercise the contract with no model and no network, and handler tests pin behaviour against fixtures. The third layer is the one people skip. Record a fixture per tool call from real traffic — the conversation up to the decision, the tool the model chose, the arguments it filled — then score tool selection weekly: the share of cases where the right tool was chosen and every required argument matched exactly. Forty triples catches a prompt edit that quietly breaks account resolution. Score the tool name and the arguments, not the prose, because those are typed values you can compare.

The last two rows are why a typed contract is necessary and not sufficient. Types stop the failures that come from the boundary being language; they do not stop a wrong tool choice, and they do not make arithmetic correct. The suite tells you whether the tools behave, not what the last failing call in production cost or which account it touched, which is the run record that answers which account saw which failure and what it cost. Tools and run records are two separate pieces of engineering that get approved by the same person.

What does the result look like on the wire?

A typed tool returns two things: a short text block for clients that only read text, and the structured object the schema promised.

{
  "jsonrpc": "2.0",
  "id": 42,
  "result": {
    "content": [{ "type": "text", "text": "quote q_9f2c created" }],
    "structuredContent": {
      "quoteId": "0f9d2c1a-6d3b-4f5e-9a71-2c8b4d6e1f30",
      "totalMinor": 7500,
      "flags": ["below_floor"]
    },
    "isError": false
  }
}

The failure variant differs in one field: isError is true and the text carries a code such as unknown_account. A code, never a stack trace, for two reasons. The model reads whatever you put there, and a stack trace either leaks internals to whoever is on the other end of the conversation or invites the model to apologise in prose instead of reporting the failure. A code is also comparable: you can count how often below_floor appears in a month, and a paragraph of English cannot be counted.

What does the typed boundary cost?

Schemas are not free. Tool definitions are sent on every request, so a description that runs to a paragraph is a paragraph you pay for at every turn. Arithmetic from stated inputs, using illustrative unit prices you should replace with your provider's current rates: fourteen tools averaging 90 tokens each is 1,260 tokens of tool definitions per request; at 40,000 requests a month that is 50.4 million fresh input tokens, about $126 at $2.50 per million. Averaging 190 tokens each — the same tools with guidance attached to every description — is 106.4 million tokens and about $266.

The lesson is not fewer tools; it is putting constraints in the schema, where they are short and machine-enforced, rather than in prose the model must interpret. An enum of three currencies costs about four tokens on the wire; a sentence explaining which currencies are supported costs twenty and is obeyed less reliably.

Engineering time is the other cost, and these are my own timings rather than a study. Adding a schema, extracting the handler and writing five tests takes me two to four hours when the logic already lives in a function, and two to four days when pricing is interleaved with transport handling, because the extraction is the work and the schema is the easy part.

When is a typed tool the wrong shape?

There are real cases, and I would rather name them than pretend the rule is universal.

If the output is genuinely open-ended — summarising a support thread, drafting a reply — do not invent a large object schema for the payload. Type the boundary and leave the prose inside it: { threadId, maxWords, format: "bullets" | "prose" } in, { summary: string, sourceCount: number } out. You still get a regression suite, and you have not forced the model to fill twelve fields with hollow text. I have reviewed servers where every key in a nine-field output object was present and four were empty strings, which is a schema paying tokens to encode nothing.

If the tool is a code interpreter or a SQL runner, flexibility is the product, and you cannot type the query. The typed boundary moves to the sandbox contract instead: which tables are readable, a row limit, a statement timeout, and a result shape. That is still a contract with assertions behind it, and it is the only thing standing between an agent and your production replica.

If it is a one-week prototype for one user, the calculation changes. I do not write schemas for a demo I intend to delete. The calculation flips the moment a second caller exists — another agent, a colleague's script, a customer — because the boundary then has a consumer who cannot read your code.

An enum with forty values kept in sync with a database table turns adding a product line into a deploy; prefer a pattern plus a server-side lookup, and return a domain error the model can act on. And a strict schema does not make an answer true. A quote can pass every type check and still be wrong, so the arithmetic needs its own test against a real invoice, not a shape assertion.

What do you ship before the next tool?

Take the one tool on your server that currently accepts a free-text argument, write its schema, extract its handler into a plain function, and add five tests: three for the schema, two for the domain. Then measure the retry rate for that tool over a week before and after. In the servers I have migrated, the retry rate falls: a validation error naming a path is a better second attempt than the same vague string sent with more words. Migrate the tools that touch money first, since those are the ones where a wrong answer carries a number and a witness. The regression suite is not a quality artefact you add after the design; it is the reason someone other than you can change the system without asking you first.