Automated SEO Regression Testing: The Failures Are Silent, So the Invariants Have to Be Asserted

Why does a broken hreflang look exactly like a working one?

Because every layer between the source file and the crawler accepts it. When I moved this site's hreflang alternates into the sitemap, the value emitted for the Chinese route was /zh/about — relative. tsc --noEmit passed. next build passed. The page rendered, the language toggle worked, the XML was well-formed. A relative alternate is not an error to any of those steps. It is a value the consumer discards, and discarding it changes nothing about the page a human sees, so nothing anywhere raises a hand. The site went on telling crawlers that two translations existed while giving them no usable address for either.

That defect is representative rather than unlucky. Content-site failures are silent in one specific way: the document is still well-formed, still renders, still returns 200. A sameAs pointing at a 404, a second <h1> introduced by a layout change, a JSON-LD block truncated because a headline contained a closing script tag, an internal link that resolves in English and 404s in Chinese — the build, the type checker and the renderer report success on all of them. Such a page is not broken. It is incorrect, and nothing in the pipeline expresses the difference.

Almost every failure mode on a content site is silent, so an invariant that is not asserted is an invariant that is already broken somewhere nobody has looked; the harness exists to turn those regressions into a non-zero exit code before the deploy rather than a slow decline in visibility afterward.

The inventory is worth writing down before writing any code, because each row says where its assertion has to live. This is the table I keep next to the audit script.

What should the harness assert, and where does it get the route list?

Three decisions carry the design. The first is that the harness audits the deployed response rather than the source: it fetches HTML over HTTP and never imports from application code, so it keeps working through refactors, and it can run against a local production build or against the live origin with a flag. The second is that the route list is discovered, not maintained. A hand-written list of URLs to check goes stale the first time someone publishes an article and forgets to add it, and the new article is then silently unaudited — which is the exact class of failure the harness exists to prevent. The third is that expectations are data attached to path patterns, so a rule covers every future article instead of the fifty that exist today.

interface RouteExpectation {
  path: string;
  article: boolean;
  mustRender: ReadonlyArray<'canonical' | 'alternates' | 'jsonld' | 'h1'>;
}

/**
 * The sitemap is the route list. Discovery means a new post is covered the moment
 * it is buildable, rather than the moment someone remembers to add a test.
 */
async function discoverRoutes(origin: string): Promise<RouteExpectation[]> {
  const response = await fetch(`${origin}/sitemap.xml`);
  if (!response.ok) throw new Error(`sitemap responded ${response.status}`);
  const xml = await response.text();

  const paths = [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].flatMap((match) =>
    // Locale is stripped so one rule covers /blog/x and /zh/blog/x.
    match[1] ? [new URL(match[1]).pathname.replace(/^\/zh(?=\/|$)/, '') || '/'] : [],
  );

  return [...new Set(paths)].map((path) => {
    const article = /^\/blog\/[^/]+$/.test(path);
    return {
      path,
      article,
      mustRender: (article
        ? ['canonical', 'jsonld', 'h1']
        : ['canonical', 'alternates', 'jsonld', 'h1']) as const,
    };
  });
}

Discovery has one honest gap, and it is worth encoding rather than hiding. The booking route is deliberately excluded from the sitemap and carries a noindex directive, so the sitemap cannot tell the harness it exists; it is appended by hand with an inverted expectation, since a route that must not be indexed still has to serve a canonical, a graph and exactly one <h1>. A route that is intentionally absent from the sitemap is precisely the route the sitemap cannot describe.

The expectations themselves are a fixture, which keeps the interesting decisions in a file a reviewer can read without following control flow.

{
  "https://willchan.me/blog/the-verification-harness-nobody-builds": {
    "article": true,
    "jsonldTypes": ["BlogPosting", "TechArticle", "BreadcrumbList", "Person", "WebSite"],
    "authorId": "https://willchan.me/about#person",
    "requiredFields": ["wordCount", "timeRequired", "articleSection", "datePublished", "mainEntityOfPage"],
    "h1Count": 1,
    "alternates": ["en", "zh-Hans", "x-default"],
    "robots": "index, follow"
  }
}

That fixture is short and its coverage is wider than it looks. Five required fields across 64 English articles is 320 assertions per run from one array, and the assertions are of the kind that fail loudly: a field is present in the serialised graph or it is not. What they cannot express is agreement — that the wordCount a page claims matches the words on the page, or that the person node means the same entity on every URL. Those need a comparison rather than a lookup, and they are where most of the harness's value sits. The reason the identity assertion matters more than the markup assertion is that identity is established by stable @id anchors reconciled across URLs, not by a valid block on a single page: one graph per page with stable @id anchors is what keeps your entities resolvable, and a per-page validation pass proves nothing about it.

How do you stop a check from reporting a pass it did not earn?

A check that cannot read its input is worse than no check, because it reports success. I learned that on the contrast probe rather than on the SEO assertions. The first version understood hex and rgb(), and returned null for anything else. Tailwind v4 emits color(srgb 0.95 0.95 0.96 / 0.08) for color-mix(), so every inline code span and solid button fell into that null branch and was never measured at all. The audit printed a pass, on the elements that were in fact unreadable in dark mode. The check was not wrong about the pages it measured; it was wrong about which pages it had measured, and it said so in the same font as a real pass.

The repair is a rule I now apply to every extraction in the harness: return a count of what you could not interpret, and fail on it. Three consequences follow. A regex that matches zero times fails, because zero matches and correct output are indistinguishable to the code downstream of it — that is how a site loses its JSON-LD entirely and the harness reports a clean run. A parse error fails with the offending snippet attached, since the fix for a truncated graph is nearly always visible in the first 200 characters. And any value pulled from a browser or a stylesheet that the harness cannot parse is counted as a failure rather than skipped, because "skipped" is the word that turns a check into a ritual.

How does a word counter lose a word boundary?

By stripping markup with an empty string. Replace <em> with '' and word</em><em>Count becomes one token, wordCount; do it a few hundred times across a document and the count drifts low by a small amount that nobody will ever trace back. The counter has to substitute a space for every tag, entity and script it removes, which means one function whose only job is turning HTML into words.

/**
 * Words in rendered HTML. Markup is replaced by a space, never by an empty
 * string, or adjacent inline elements merge into a single token and the total
 * drifts low for reasons no one will trace.
 */
export function countRenderedWords(html: string): number {
  const text = html
    .replace(/<(script|style)\b[^>]*>[\s\S]*?<\/\1>/gi, ' ')
    .replace(/<[^>]+>/g, ' ')
    .replace(/&(?:amp|lt|gt|quot|#39|nbsp);/gi, ' ');

  const cjk = (text.match(/[\u3400-\u4dbf\u4e00-\u9fff]/g) ?? []).length;
  const latin = text
    .replace(/[\u3400-\u4dbf\u4e00-\u9fff]/g, ' ')
    .split(/\s+/)
    .filter((token) => /[A-Za-z0-9]/.test(token)).length;

  // Roughly 1.7 CJK characters carry the information of one English word, so the
  // two scripts are summed on a common scale instead of compared directly.
  return latin + Math.round(cjk / 1.7);
}

The second half of that function is the part I would have got wrong without the Chinese pages. A whitespace tokenizer reports a 3,500-character Chinese essay as about 40 words, because Chinese is not whitespace-delimited; count ideographs individually, convert the Latin runs, and sum. The asymmetry that bit me later was subtler than the one I expected: the first version stripped fenced code blocks before counting ideographs but not before counting Latin words, so a code-heavy Chinese article reported more words than its English original, purely because Chinese characters inside listings were counted. Both scripts have to be measured on the same text. The harness then compares the recomputed number against the wordCount in the page's BlogPosting node and fails when they disagree by more than a few percent. I should say plainly that the version running today asserts the presence of that field and not its value; the comparison is the assertion I am adding, and the extraction above is what it needs to be possible.

When is a verification harness the wrong approach?

When the site is small enough that a human can hold it. Five pages in one language do not need a crawler; open two responses side by side, diff them, and spend the afternoon on the prose instead. The harness earns its place somewhere around the point where the same invariant has to hold on more URLs than anyone will read, which for this site is the moment the 64 English and 3 Chinese article files are all routed through one layout.

When the checks assert implementation rather than output. A harness that imports the component and confirms it called the graph builder is testing the source, and it will fail on a refactor that changes nothing about the response while passing on a routing bug that changes everything. Fetch the HTML. Assert on the text of the response.

When you cannot run the artifact you ship. Three of the defects in the table above exist only in built output: the relative alternate was produced by the sitemap route, the locale-prefixed 404 comes from a rewrite rule, and the truncation comes from serialisation. A harness pointed at a dev server tests a different program, and its passes mean correspondingly less.

When a failure has no repair path. A check that is permanently red gets ignored within a week, and it takes the rest of the suite's credibility with it. If an invariant cannot be fixed today — an identity list with no corroborating profiles behind it, say — it belongs in an observation log that prints on every run and does not block the deploy, not in the failure set.

When the harness produces false positives of its own. One of my visual checks flagged a <script> element as a collapsed layout element. Nothing was wrong with the page; the check was wrong about the page. False positives erode a gate at the same rate as false negatives, and the repair is to narrow the check rather than to leave it in place and learn to ignore it.

And when the cost is not paid back. The HTML audit needs a production server listening, and the visual check needs a real browser rendering 22 page and viewport combinations. If neither is available in the pipeline, the honest subset is a static audit of the prerendered output on disk: it still catches relative URLs, missing nodes and unparseable graphs, and it catches nothing at all that depends on the request.

What should you build first?

Write the identity assertion before anything else: fetch every URL in the sitemap, extract the person node from each graph, collapse identical definitions, and fail unless the set of distinct values has exactly one member. Two counts carry the whole check — URLs that emitted no person node at all, and distinct identity cores — and each maps to a different repair, a route that stopped calling the wrapper or a page that disagrees about the entity. Then add absolute canonicals, complete alternates, the single-<h1> rule, and internal link status in that order, and wire the whole thing into the pre-deploy step so it can actually stop a release. An invariant that is only checked when someone remembers to check it is not a gate, it is a habit, and habits do not survive the quarter in which two people are editing the same layout. The first run will report pages with no graph at all far more often than divergent graphs, which is the better outcome: a route that never called the wrapper has an obvious fix, while two versions of an entity that have been live for a year have both already been read.