Running SEO Checks in a CI Pipeline: What to Automate
The deploy went green. Every test passed. Two days later, someone in marketing posted a Search Console screenshot in Slack: pages were dropping out of the index. A template change had shipped a noindex tag that was only ever meant for staging.
Nothing in the pipeline caught it, because nothing in the pipeline was looking. Unit tests check that your code behaves. They don't check the HTML that Googlebot actually receives.
This post is a practical plan for running SEO checks in a CI pipeline. It covers which checks to automate, where each one should run, and which ones to leave to humans. Every script below was tested before publishing, and you can drop them into a GitHub Actions setup today. If you've wanted to start automated SEO testing but weren't sure where the line was, this is it.
Why SEO Breaks Quietly in Deploys
SEO regressions are sneaky because nothing errors out. The page loads, the design looks right, and users can click around normally. Meanwhile:
- A staging noindex meta tag or X-Robots-Tag header goes to production.
- robots.txt ships with Disallow: / copied from a preview environment.
- Canonical tags point at the staging host.
- A layout refactor wipes page-specific titles and meta descriptions.
- Routes get renamed with no redirects, so old URLs return 404.
- A JSON-LD block breaks because someone added a trailing comma.
- The sitemap still lists URLs that no longer exist.
Each of these takes seconds to detect with the right check and weeks to notice from traffic graphs. That gap is exactly what CI is good at closing.
The Rule for What to Automate
Not every SEO check belongs in a pipeline. Use one simple rule:
- Automate checks that are deterministic, fast and binary. A tag is present or it isn't. A URL returns 200 or it doesn't. These make good gates.
- Monitor checks that are noisy or depend on real users. Lab performance scores vary between runs, and Core Web Vitals field data comes from real visitors over weeks. Track these, but don't block merges on them.
- Keep judgment human. Content quality, search intent, keyword targeting and title copywriting need a person.
The Four Layers at a Glance
1. Pull request gate: lint the built HTML for missing or broken tags. Runs in seconds.
2. Lighthouse sample: audit a handful of key templates for SEO and performance budgets.
3. Post-deploy check: test the live server for robots rules, headers (including canonical), redirects and sitemap health.
4. Scheduled monitoring: catch slow drift such as broken external links and field performance.
To decide which individual rules to codify, start from a reference list rather than a blank page. SEO Site Checker's 70 SEO checks explained groups on-page, performance, security and advanced checks with plain-language explanations, which makes it easy to pick the deterministic ones for your linter.
Layer 1: Gate Pull Requests on the Built HTML
The most important habit is to test the build output, not your source templates. Your framework, your CMS data and your layout components only come together in the final HTML, and that HTML is what search engines see. Point the checks at dist/, out/ or whatever folder your static export produces.
Checks That Belong Here
- Exactly one non-empty <title>, unique across the site, with a warning above 60 characters
- A meta description, with a warning above 160 characters
- Exactly one canonical tag, absolute and pointing to the production host
- No noindex in meta robots, except on pages you deliberately allow
- One <h1> per page (warning only, since some layouts have good reasons)
- A lang attribute on <html>
- An alt attribute on every image (an empty alt="" is fine for decorative images)
- Basic Open Graph tags
- JSON-LD that parses and includes @context and @type, including inside @graph
A Small Checker Script
This script uses cheerio to parse every HTML file in the build folder. It separates errors, which fail the build, from warnings, which are only printed. Install it with npm install --save-dev cheerio.
// scripts/seo-check.mjs
// Usage: node scripts/seo-check.mjs <build-dir> <production-origin>
import { readdir, readFile } from 'node:fs/promises';
import path from 'node:path';
import * as cheerio from 'cheerio';
const [dir = 'dist', origin = 'https://example.com'] = process.argv.slice(2);
const NOINDEX_ALLOWED = new Set(['404.html', 'search/index.html']);
const errors = [];
const warnings = [];
const titles = new Map();
async function findHtml(folder) {
const files = [];
for (const entry of await readdir(folder, { withFileTypes: true })) {
const full = path.join(folder, entry.name);
if (entry.isDirectory()) files.push(...(await findHtml(full)));
else if (entry.name.endsWith('.html')) files.push(full);
}
return files;
}
function checkPage(page, html) {
const $ = cheerio.load(html);
const error = (msg) => errors.push(`${page}: ${msg}`);
const warn = (msg) => warnings.push(`${page}: ${msg}`);
// Indexing
const robots = ($('meta[name="robots"]').attr('content') ?? '').toLowerCase();
if (robots.includes('noindex') && !NOINDEX_ALLOWED.has(page)) error('meta robots contains noindex');
// Title
const titleTags = $('head > title');
const title = titleTags.first().text().trim();
if (titleTags.length !== 1 || !title) {
error(`expected one non-empty <title>, found ${titleTags.length}`);
} else {
if (title.length > 60) warn(`title is ${title.length} characters`);
titles.set(title, [...(titles.get(title) ?? []), page]);
}
// Meta description
const description = $('meta[name="description"]').attr('content')?.trim();
if (!description) error('missing meta description');
else if (description.length > 160) warn(`meta description is ${description.length} characters`);
// Canonical
const canonicals = $('link[rel~="canonical"]');
if (canonicals.length === 0) {
error('missing canonical tag');
} else if (canonicals.length > 1) {
error(`found ${canonicals.length} canonical tags, expected exactly one`);
} else {
const canonical = canonicals.attr('href');
try {
const url = new URL(canonical);
if (url.origin !== origin) error(`canonical points to ${url.origin}`);
} catch {
error(`canonical is not an absolute URL: ${canonical}`);
}
}
// Headings, language, images, social
const h1Count = $('h1').length;
if (h1Count !== 1) warn(`found ${h1Count} <h1> elements`);
if (!$('html').attr('lang')) warn('missing lang attribute on <html>');
$('img:not([alt])').each((_, img) => warn(`image without alt: ${$(img).attr('src')}`));
if (!$('meta[property="og:title"]').length) warn('missing og:title');
// Structured data
$('script[type="application/ld+json"]').each((i, el) => {
let data;
try {
data = JSON.parse($(el).text());
} catch {
return error(`JSON-LD block ${i + 1} is not valid JSON`);
}
for (const block of Array.isArray(data) ? data : [data]) {
if (!block['@context']) error(`JSON-LD block ${i + 1} has no @context`);
const items = block['@graph'] ?? [block];
if (items.some((item) => !item['@type'])) error(`JSON-LD block ${i + 1} has an item without @type`);
}
});
}
for (const file of await findHtml(dir)) {
checkPage(path.relative(dir, file).split(path.sep).join('/'), await readFile(file, 'utf8'));
}
for (const [title, pages] of titles) {
if (pages.length > 1) errors.push(`duplicate title "${title}" on: ${pages.join(', ')}`);
}
warnings.forEach((w) => console.warn(`warning ${w}`));
errors.forEach((e) => console.error(`error ${e}`));
console.log(`\n${errors.length} errors, ${warnings.length} warnings`);
process.exit(errors.length ? 1 : 0);
The canonical check counts every <link rel="canonical"> instead of reading the first one. A page with two (say a hand-written tag plus a framework or plugin default) sends conflicting signals, and Google may ignore both, so the check fails on anything other than exactly one. Using rel~= also matches tags like rel="canonical nofollow".
Run against a sample build with a few planted mistakes, the output looks like this:
warning blog/post/index.html: found 2 <h1> elements
warning blog/post/index.html: missing lang attribute on <html>
warning blog/post/index.html: image without alt: b.png
warning blog/post/index.html: missing og:title
error about/index.html: canonical is not an absolute URL: /about/
error blog/post/index.html: meta robots contains noindex
error blog/post/index.html: missing meta description
error blog/post/index.html: canonical points to https://staging.example.com
error blog/post/index.html: JSON-LD block 1 is not valid JSON
error services/index.html: found 2 canonical tags, expected exactly one
error duplicate title "Home | Example" on: blog/post/index.html, index.html
7 errors, 4 warnings
Every message names the page and the rule, so whoever opened the PR knows exactly what to fix.
Wire It Into GitHub Actions
# .github/workflows/seo-pr.yml
name: SEO checks
on:
pull_request:
jobs:
seo:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- run: npm run build
- name: Lint built HTML
run: node scripts/seo-check.mjs dist https://example.com
- name: Check internal links
uses: lycheeverse/lychee-action@v2
with:
args: --offline --no-progress --root-dir "${{ github.workspace }}/dist" "dist/**/*.html"
The lychee link checker runs in offline mode here, so it only checks links between your own files. That keeps it fast and immune to third-party sites being slow. External links get checked later, on a schedule.
Keep this whole job under about a minute. If an SEO gate becomes the slowest step in CI, people start looking for ways around it.
Layer 2: Lighthouse CI on Key Templates, Not Every Page
Lighthouse is useful, but it's slow, and its performance scores change from run to run. Running it on every page of every PR is a recipe for flaky builds. Instead, audit three to five representative templates: the homepage, an article, a product or service page and a listing page.
Lighthouse CI can serve your static build itself and assert on the results:
{
"ci": {
"collect": {
"staticDistDir": "./dist",
"url": [
"http://localhost/index.html",
"http://localhost/blog/sample-post/index.html",
"http://localhost/pricing/index.html"
],
"numberOfRuns": 3
},
"assert": {
"assertions": {
"categories:seo": ["error", { "minScore": 0.9 }],
"categories:performance": ["warn", { "minScore": 0.8 }],
"largest-contentful-paint": ["warn", { "maxNumericValue": 2500 }],
"cumulative-layout-shift": ["warn", { "maxNumericValue": 0.1 }]
}
},
"upload": {
"target": "filesystem",
"outputDir": "./lhci-reports"
}
}
}
Save that as lighthouserc.json and add one step to the workflow: npx @lhci/cli autorun.
Notice the split. The SEO category is an error, because its audits are mostly deterministic (crawlable links, valid hreflang, a document title and so on). Performance is a warning, because CI runners vary in speed. Three runs per URL lets Lighthouse CI use the median result, which smooths out the noise.
Layer 3: Check Production Right After Deploy
Some of the most damaging problems only exist on the live server. Response headers, robots.txt, redirects and host settings often come from CDN rules, server config or a hosting dashboard rather than your repository. A PR check can't see them, so test production immediately after every deploy.
robots.txt and X-Robots-Tag
Fail the run if robots.txt blocks everything for User-agent: *, or if key pages send noindex through an X-Robots-Tag header or a meta tag. The header is easy to miss because it never appears in the HTML. Google's robots meta tag and X-Robots-Tag docs explain how both are interpreted.
Be precise about Disallow: /. Plenty of sites block specific bots, such as AI crawlers, on purpose, so the check should only fail when the rule applies to every crawler.
Canonical in the Link Header
A canonical can also be sent as an HTTP header (Link: <https://example.com/page>; rel="canonical"). If it disagrees with the one in the HTML, you've sent conflicting signals, and the mismatch never shows up in the built files. The script below compares both against the live response, and also counts canonicals in the live HTML, which catches duplicates that a CDN or edge script injects after the build.
A Redirect Map Test
Keep a redirects.csv file in the repository with two columns, from and to. After each deploy, assert that every old URL returns a single 301 or 308 straight to the expected target, and that the target returns 200. This catches missing redirects, temporary 302s, wrong destinations and chains.
Sitemap Sanity
Request every URL in the sitemap and expect a 200. Then compare the URL count against a stored baseline, and fail if it drops sharply. A sitemap that suddenly loses a large share of its URLs usually means a routing change or a broken data source, not a deliberate cleanup.
A count alone can hide a problem, though. Delete ten established pages, add ten unrelated ones, and the total doesn't move. So the baseline also stores the URL set. Every URL that disappeared since the last clean run must have an entry in redirects.csv, otherwise it's either a missing route or a missing redirect, and the run fails. The removed URLs are also printed, so whoever reviews the deploy gets a concrete list instead of a reassuring total.
The Post-Deploy Script
// scripts/post-deploy-check.mjs
// Usage: node scripts/post-deploy-check.mjs https://example.com
import { readFile, writeFile } from 'node:fs/promises';
const origin = (process.argv[2] ?? 'https://example.com').replace(/\/$/, '');
const KEY_PATHS = ['/', '/blog/', '/pricing/'];
const MAX_SITEMAP_DROP = 0.1; // fail if the sitemap shrinks by more than 10%
const failures = [];
const fail = (msg) => failures.push(msg);
// 1. robots.txt must not block everything for all crawlers
function blocksAllCrawlers(txt) {
let agents = [];
let inRules = false;
for (const raw of txt.split(/\r?\n/)) {
const line = raw.replace(/#.*/, '').trim();
const sep = line.indexOf(':');
if (sep === -1) continue;
const key = line.slice(0, sep).trim().toLowerCase();
const value = line.slice(sep + 1).trim();
if (key === 'user-agent') {
if (inRules) { agents = []; inRules = false; }
agents.push(value);
} else if (key === 'allow' || key === 'disallow') {
inRules = true;
if (key === 'disallow' && value === '/' && agents.includes('*')) return true;
}
}
return false;
}
const robotsRes = await fetch(`${origin}/robots.txt`);
if (!robotsRes.ok) fail(`robots.txt returned ${robotsRes.status}`);
else if (blocksAllCrawlers(await robotsRes.text())) fail('robots.txt has "Disallow: /" for User-agent: *');
// 2. Key pages: 200, no noindex, one canonical that matches the Link header
for (const p of KEY_PATHS) {
const res = await fetch(origin + p);
if (res.status !== 200) { fail(`${p} returned ${res.status}`); continue; }
if (/noindex/i.test(res.headers.get('x-robots-tag') ?? '')) fail(`${p} sends X-Robots-Tag: noindex`);
const html = await res.text();
if (/<meta[^>]+name=["']robots["'][^>]+noindex/i.test(html)) fail(`${p} has a noindex meta tag`);
const htmlCanonicals = [...html.matchAll(/<link\b[^>]*\brel=["'][^"']*\bcanonical\b[^"']*["'][^>]*>/gi)]
.map((m) => m[0].match(/\bhref=["']([^"']+)["']/i)?.[1])
.filter(Boolean);
if (htmlCanonicals.length > 1) fail(`${p} has ${htmlCanonicals.length} canonical tags in the live HTML`);
const headerCanonical = (res.headers.get('link') ?? '')
.match(/<([^>]+)>\s*;[^,]*\brel=["']?canonical["']?/i)?.[1];
if (headerCanonical && htmlCanonicals[0]) {
const a = new URL(headerCanonical, origin).href;
const b = new URL(htmlCanonicals[0], origin).href;
if (a !== b) fail(`${p} Link header canonical (${a}) differs from HTML canonical (${b})`);
}
}
// 3. Redirect map: one hop, permanent, correct target
const csv = await readFile('redirects.csv', 'utf8').catch(() => '');
const redirectedPaths = new Set();
for (const line of csv.split(/\r?\n/).slice(1).filter(Boolean)) {
const [from, to] = line.split(',').map((s) => s.trim());
redirectedPaths.add(new URL(from, origin).pathname);
const res = await fetch(origin + from, { redirect: 'manual' });
const location = res.headers.get('location');
if (![301, 308].includes(res.status)) { fail(`${from} returned ${res.status}, expected 301 or 308`); continue; }
const target = new URL(location, origin);
if (target.href !== new URL(to, origin).href) { fail(`${from} redirects to ${target.pathname}, expected ${to}`); continue; }
const final = await fetch(target, { redirect: 'manual' });
if (final.status !== 200) fail(`${from} -> ${to} ends in ${final.status} (chain or broken target)`);
}
// 4. Sitemap: every URL returns 200, the count has not dropped sharply,
// and every URL removed since the last clean run has a redirect
const sitemapRes = await fetch(`${origin}/sitemap.xml`);
if (!sitemapRes.ok) {
fail(`sitemap.xml returned ${sitemapRes.status}`);
} else {
const urls = [...(await sitemapRes.text()).matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)].map((m) => m[1]);
const baseline = JSON.parse(await readFile('seo-baseline.json', 'utf8').catch(() => '{}'));
if (baseline.sitemapCount && urls.length < baseline.sitemapCount * (1 - MAX_SITEMAP_DROP)) {
fail(`sitemap has ${urls.length} URLs, down from ${baseline.sitemapCount}`);
}
const current = new Set(urls.map((u) => new URL(u).pathname));
const removed = (baseline.urls ?? []).filter((u) => !current.has(u));
const unredirected = removed.filter((u) => !redirectedPaths.has(u));
if (removed.length) console.log(`Removed since last clean run: ${removed.join(', ')}`);
unredirected.forEach((u) => fail(`${u} was removed from the sitemap with no entry in redirects.csv`));
for (let i = 0; i < urls.length; i += 10) {
await Promise.all(urls.slice(i, i + 10).map(async (url) => {
const res = await fetch(url, { redirect: 'manual' });
if (res.status !== 200) fail(`sitemap URL ${url} returned ${res.status}`);
}));
}
if (!failures.length) await writeFile('seo-baseline.json', JSON.stringify({ sitemapCount: urls.length, urls: [...current].sort() }, null, 2));
}
failures.forEach((f) => console.error(`FAIL ${f}`));
console.log(failures.length ? `\n${failures.length} problems found` : 'All post-deploy SEO checks passed');
process.exit(failures.length ? 1 : 0);
Tested against a deliberately broken server, it reports:
Removed since last clean run: /old-pricing, /removed/
FAIL robots.txt has "Disallow: /" for User-agent: *
FAIL / has 2 canonical tags in the live HTML
FAIL /blog/ Link header canonical (http://localhost:8765/news/) differs from HTML canonical (http://localhost:8765/blog/)
FAIL /pricing/ sends X-Robots-Tag: noindex
FAIL /old-pricing redirects to /plans/, expected /pricing/
FAIL /old-blog returned 302, expected 301 or 308
FAIL /removed/ was removed from the sitemap with no entry in redirects.csv
FAIL sitemap URL http://localhost:8765/gone/ returned 404
8 problems found
A few notes on the script:
- The baseline file only updates after a clean run, so one bad deploy can't quietly lower the bar. On GitHub Actions, persist seo-baseline.json with actions/cache or commit it, because each runner starts fresh.
- /old-pricing left the sitemap too, but it has a redirect entry, so only /removed/ fails. The sitemap count was unchanged, so a count-only check would have passed this deploy.
- It reads a single sitemap.xml. If you use a sitemap index, fetch each child sitemap first.
- Sitemap URLs are checked ten at a time to avoid hammering your own server.
Trigger it after a successful production deployment:
# .github/workflows/seo-post-deploy.yml
name: Post-deploy SEO check
on:
deployment_status:
jobs:
smoke:
if: github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'production'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: node scripts/post-deploy-check.mjs https://example.com
Check what your host calls the production environment. Some platforms report it as Production with a capital P. If you deploy from your own workflow instead, run the script as a final job that depends on the deploy job.
Layer 4: Scheduled Monitoring for What CI Can't Judge
Some problems develop slowly or depend on things outside your code. Handle them with a scheduled workflow that alerts people instead of failing builds:
on:
schedule:
- cron: "17 6 * * 1" # Mondays at 06:17 UTC
workflow_dispatch:
Good candidates for a weekly run:
- External link checks with lychee in online mode, since other sites change without telling you
- Core Web Vitals field data from the Chrome UX Report (CrUX) API, which reflects real users rather than a lab run
- Indexing coverage through the Search Console API, to spot a jump in excluded pages
- The post-deploy script again, to catch config changes made outside a deploy
Post the results to Slack or open a GitHub issue. Nobody should be blocked from merging because a partner's website went down.
Automation also doesn't replace a periodic human look at the whole site. Every month or so, run a full SEO audit on your key pages and compare it with the last one.
What Not to Automate
It's tempting to turn everything into a gate. Resist that for:
- Content quality and helpfulness. No linter can tell whether a page actually answers the searcher's question.
- Search intent and keyword choices. These are strategy decisions, not syntax errors.
- Title copywriting. You can check length automatically, but not whether the title is good.
- Internal linking strategy. You can check for broken links, but deciding what to link is editorial work.
- Lighthouse scores on every page as hard gates. They flake, people stop trusting the pipeline, and then they ignore real failures too.
Automation keeps the floor up. Humans still have to raise the ceiling. For the judgment side, a manual technical SEO audit checklist covers what a script can't.
Rolling It Out Without Annoying Your Team
- Start with warnings only. Run every check in warning mode for two weeks, fix the existing backlog, then promote the stable checks to errors.
- Keep an allowlist. Some pages, like internal search results or a 404 page, should be noindex. Put exceptions in one obvious place, like NOINDEX_ALLOWED in the script.
- Make failures readable. Each message should name the page, the rule and the fix. Consider posting a summary as a sticky PR comment.
- Give every check an owner. When a rule fires falsely, someone needs to fix the rule rather than add || true to the workflow.
Frequently Asked Questions
Can you automate SEO testing?
Yes, for the technical layer. Anything with a clear pass or fail can be automated: tags, status codes, redirects, robots rules, structured data syntax and sitemap health. Content quality, search intent and strategy still need human review, so automation complements an SEO specialist rather than replacing one.
Is Lighthouse enough for SEO checks in CI?
No. Lighthouse's SEO category covers useful basics, but it audits one page at a time, doesn't compare titles across pages, can't see your redirects or robots.txt in production, and doesn't validate your structured data against your own rules. Use it as one layer alongside an HTML linter and post-deploy checks.
Should SEO checks fail the build or just warn?
Fail the build for deterministic, high-impact problems such as noindex on indexable pages, missing titles, missing, duplicate or wrong canonicals and invalid JSON-LD. Warn for things that are noisy or subjective, like performance scores, title length or multiple H1 tags. Start everything as a warning, then promote checks once they've proven reliable.
How do I test for noindex and robots.txt mistakes before they reach production?
Lint the built HTML for noindex meta tags in every pull request, then run a post-deploy check against production for robots.txt rules and X-Robots-Tag headers. Headers and robots files often come from server or CDN config rather than your code, so the production check is essential.
How do I check for broken links in GitHub Actions?
Use a link checker such as lychee through its GitHub Action. Run it in offline mode on your build output during pull requests to check internal links quickly, then run it in online mode on a schedule to check external links without blocking merges when a third-party site is down.
Do SEO checks slow down a CI pipeline?
They don't have to. An HTML linter over a typical static build finishes in seconds. Lighthouse is the slow part, so limit it to a few key templates. Post-deploy and scheduled checks run outside the pull request flow entirely, so they never block developers.
Wrapping Up
A sensible SEO pipeline has four layers. It gates pull requests on the built HTML, samples key templates with Lighthouse, checks production right after deploy, and monitors slow drift on a schedule. Automate what's binary, monitor what's noisy, and keep judgment human.
When you're ready to add your next rule, browse the full list of on-page and technical SEO checks and pick one that your pipeline doesn't cover yet.
---
Update: thanks to Posting Dude for pointing out the duplicate-canonical and Link-header gaps, and to Marcus Kim for suggesting the URL-set baseline diffed against the redirect map. All three are now in the scripts above.
6 comments
Advanced Performance Monitoring with Lighthouse, PageSpeed Insights, and Search Console
---
title: Advanced Performance Monitoring with Lighthouse, PageSpeed Insights, and Search Console
published: true
published_at: 2026-10-20 13:00 +0000
tags: performance, lighthouse, pagespeedinsights, searchconsole
canonical_url: khriztianmoreno.dev
---
Tools do not fix performance. Systems do.
Most developers treat performance as a checklist: run a Lighthouse audit, fix a few warnings, and move on. But performance is not a static state; it is a changing ecosystem. Your users are on different devices, networks, and browsers. Your code changes daily.
If you are only looking at performance when you remember to run a report, you are already behind.
This guide explains how to move from one-off audits to continuous performance monitoring using the three pillars of Google's tooling ecosystem: Lighthouse, PageSpeed Insights (PSI), and Google Search Console (GSC).
1. Intro: Performance is a system, not a report
We have all been there. You run a Lighthouse audit locally and get a 95. You push to production, run it again, and get a 72. You check PageSpeed Insights, and it shows completely different numbers. Then you look at Search Console, and it says everything is fine—but the data is from three weeks ago.
It feels contradictory, but it isn't. Each tool is answering a different question.
To build a robust performance culture, you must stop chasing a single "score" and start listening to the signals your system is sending.
2. The three pillars of Google performance tooling
Before diving into workflows, let's establish a mental model for what each tool actually does.
| Tool | Data Type | Question it Answers |
| :--------------------- | :------------------- | :--------------------------- |
| Lighthouse | Synthetic (Lab) | _What could go wrong?_ |
| PageSpeed Insights | Hybrid (Lab + Field) | _What is going wrong?_ |
| Search Console | Field (CrUX) | _Does Google see a problem?_ |
Understanding this distinction is crucial. You use Lighthouse to catch regressions before they ship. You use PageSpeed Insights to validate real-world impact. You use Search Console to track long-term health and SEO ranking signals.
3. Lighthouse: advanced usage beyond scores
Lighthouse is a lab tool. It runs in a controlled environment (your machine or a CI server) with a predefined device and network profile. Because it is synthetic, it is deterministic and reproducible. This makes it the ideal tool for Continuous Integration (CI).
Advanced Lighthouse workflows
Don't just run Lighthouse manually in Chrome DevTools. That relies on your local computer's CPU power and extensions, which skews the results.
Run Lighthouse in CI to prevent performance regressions before they merge.
# Example: Running Lighthouse via CLI
npx lighthouse https://example.com \
--preset=desktop \
--only-categories=performance
By integrating Lighthouse CI, you can enforce performance budgets. For example, fail the build if the bundle size exceeds 200KB or if the Largest Contentful Paint (LCP) is forecasted to be over 2.5s.
Ignore the score, read the trace
The 0-100 score is a summary for managers. Engineers need the Trace.
The trace reveals exactly what happened during the page load:
- Main-thread blocking time: Which specific JavaScript function locked up the UI?
- Render-blocking resources: Did a CSS file delay the first paint?
- Unused JavaScript: Are you shipping library code that is never executed?
Use the Lighthouse Trace Viewer to drill down into the milliseconds.
4. PageSpeed Insights: bridging lab and field
PageSpeed Insights (PSI) is often misunderstood because it presents two sets of data next to each other:
1. Lab Data (Lighthouse): A simulation run on Google's servers.
2. Field Data (CrUX): Real data from real users visiting your site (Chrome User Experience Report).
This is why teams get confused. _"Lighthouse says my LCP is 1.2s, but Field Data says it's 3.5s. Which is true?"_
The Field Data is the truth. It represents what your actual users are experiencing. If your Lab data is fast but Field data is slow, your local simulation assumes a faster network or device than your users actually have.
Reading PSI correctly
- Field data → Reality. This is what Google uses for ranking. If this is red, you have a problem, no matter what Lighthouse says.
- Lab data → Diagnosis. Use this to reproduce issues found in the field.
- Opportunities → Hypotheses. These are algorithmic suggestions. Not all of them will improve Core Web Vitals.
Important Rule: Never optimize for a Lab metric that does not correlate with a Field problem. If PSI says "Remove unused CSS" but your FCP is already 0.8s in the field, that optimization is a waste of time.
5. Google Search Console: performance at scale
Google Search Console (GSC) provides the "Core Web Vitals" report. This is aggregated CrUX data grouped by URL patterns.
Limitations
- Delayed: Data is a 28-day rolling average. You won't see the impact of a deploy today until weeks later.
- Aggregated: It groups pages together (e.g., "200 similar URLs"). It can be hard to debug a specific outlier page.
Why it matters
Despite the lag, GSC is critical because this is exactly how Google groups and ranks your site. It tells you if you have systemic issues affecting entire sections of your application (e.g., "All product pages have poor CLS").
Use GSC to:
1. Detect systemic issues: If CLS spikes across all /blog/* pages, you likely introduced a layout shift in your blog post template.
2. Validate long-term improvements: After a fix, use the "Validate Fix" button to tell Google to start verifying the new data.
6. Synthetic vs. Real User Monitoring (RUM)
This distinction is the conceptual core of modern performance engineering.
Synthetic Monitoring (Lighthouse, WebPageTest)
- Pros: Fast feedback, controlled environment, cheap to run, CI-friendly.
- Cons: Not representative of real users, misses device diversity, "clean room" environment.
- Best for: Preventing regressions during development.
Real User Monitoring (CrUX, web-vitals.js)
- Pros: Actual user experience, captures device/network diversity, correlates with business metrics and SEO.
- Cons: Slower feedback loop, requires instrumentation, data can be noisy.
- Best for: Understanding reality and verifying fixes.
Why you need both: Synthetic finds problems early (while coding). RUM confirms they actually matter to users.
7. Building a modern monitoring stack
A mature engineering team uses a pipeline that looks like this:
1. Local Development: Engineers use Lighthouse in DevTools to spot obvious issues.
2. CI/CD Pipeline: Lighthouse CI runs on every Pull Request. It blocks merges if metrics degrade beyond a threshold.
3. Production (Synthetic): A service runs a Lighthouse check on key pages every hour to catch infrastructure issues or third-party script regressions.
4. Production (RUM): The site reports Core Web Vitals to an analytics endpoint (using web-vitals.js) to track real-time trends.
5. SEO Health: The team reviews Search Console weekly to ensure no new URL groups are flagged as "Poor".
8. Common mistakes teams make
- Chasing Lighthouse 100: A score of 100 on a Developer's MacBook Pro means nothing if your users are on low-end Android devices.
- Ignoring INP in field data: Interaction to Next Paint (INP) is hard to reproduce in Lab tools because they don't click around. You _must_ rely on Field data for INP.
- Treating Search Console as real-time: Don't panic if GSC doesn't update the day after a fix. It takes time.
- Optimizing Lab-only regressions: If Lighthouse complains about a metric that looks green in CrUX, deprioritize it.
9. What’s next: where monitoring is going
Performance monitoring is moving towards attribution. It's not enough to know _that_ the page is slow; we need to know _what code_ made it slow.
- Route-level CWV: Tools are getting better at attributing metrics to specific SPA routes (Soft Navigations).
- More granular CrUX data: Google is exposing more detailed data in the CrUX History API.
- Interaction breakdowns: LoAF (Long Animation Frames) API is revolutionizing how we debug main-thread blocking, giving us stack traces for long tasks in the wild.
10. Conclusion
No single tool gives you the full picture. Lighthouse helps you build fast software. PageSpeed Insights helps you debug user issues. Search Console helps you maintain your search ranking.
Performance is not a task you finish. It is a signal you continuously listen to. Start listening today.
Key References
- Web Vitals Documentation
- Google Search Central: Core Web Vitals
- Lighthouse Documentation
- PageSpeed Insights
I hope this has been helpful and/or taught you something new!
!Profile
@khriztianmoreno
Until next time.
Mentions
How to configure a Web Performance Budget with Lighthouse CI Hey folks, it's me again! :) Today, I would like to share with you how you can use Lighthouse CI to audit various software qualities of your project in Continous Integration but also how you can set up a Web Performance Budget strategy to not ship unintentional regressions. I used this techique in various kinds of projects and I really liked the outcome of it. Enjoy! Problem In Software Development, we always aimed to develop and ship fast. Nowadays with the growing popularity of AI agents that help us ship code even faster, being able to set certain quality guardrails (i.e. in Continous Integration/Continous Deployment) is essential to maintain software quality. This is especially visible in the area of Web Performance where we can very easily ship performance regressions by adding a third party script, an unoptimized image or multiple custom fonts. To maintain good performance and exceptional user experience, we need to find a way to warn us early about unexpected regressions. This is where Lighthouse CI can help us. Solution Lighthouse CI is a tool that can be used both in local development as well (and primarily) in the CI GitHub Action. The idea is simple: 1. Define a configuration file 2. Add a GitHub action/step 3. See results and react to them before they reach production The basic example could just be like following: name: CI on: [push] jobs: lighthouseci: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 22 - run: npm install && npm install -g @lhci/cli@0.15.x - run: npm run build - run: lhci autorun But this will run the test in a basic setup - in our case we want to add some more configuration that will allow us to define our custom budget, and update it on the go to more restrict setup after time. ci: collect: numberOfRuns: 5 startServerCommand: rails server -e production url: - http://localhost:3000/ - http://localhost:3000/pricing - http://localhost:3000/support assert: preset: lighthouse:recommended assertions: first-contentful-paint: - error - maxNumericValue: 2000 aggregationMethod: optimistic largest-contentful-paint: - error - maxNumericValue: 2500 aggregationMethod: optimistic cumulative-layout-shift: - error - maxNumericValue: 100 aggregationMethod: optimistic upload: target: lhci serverBaseUrl: https://lhci.example.com This configuration can either be in the yaml,js,json or any other supported format. What is important is that with it, you can be warned early about unexpected regressions and update the budget later to match your needs and goals better. And if you would like to compare one LHCI result to the previous one to show instant improvements/regressions, here is a great example for that adevinta/actions-lighthouseci-compare Learn more Here is the recording of my talk at JS Poland conference where I was talking about this topic (and also a bit about Playwright and Accessibility): {% embed youtube.com %}
Running SEO Checks in a CI Pipeline: What to Automate The deploy went green. Every test passed. Two days later, someone in marketing posted a Search Console screenshot in Slack: pages were dropping out of the index. A template change had shipped a noindex tag that was only ever meant for staging. Nothing in the pipeline caught it, because nothing in the pipeline was looking. Unit tests check that your code behaves. They don't check the HTML that Googlebot actually receives. This post is a practical plan for running SEO checks in a CI pipeline. It covers which checks to automate, where each one should run, and which ones to leave to humans. Every script below was tested before publishing, and you can drop them into a GitHub Actions setup today. If you've wanted to start automated SEO testing but weren't sure where the line was, this is it. Why SEO Breaks Quietly in Deploys SEO regressions are sneaky because nothing errors out. The page loads, the design looks right, and users can click around normally. Meanwhile: - A staging noindex meta tag or X-Robots-Tag header goes to production. - robots.txt ships with Disallow: / copied from a preview environment. - Canonical tags point at the staging host. - A layout refactor wipes page-specific titles and meta descriptions. - Routes get renamed with no redirects, so old URLs return 404. - A JSON-LD block breaks because someone added a trailing comma. - The sitemap still lists URLs that no longer exist. Each of these takes seconds to detect with the right check and weeks to notice from traffic graphs. That gap is exactly what CI is good at closing. The Rule for What to Automate Not every SEO check belongs in a pipeline. Use one simple rule: - Automate checks that are deterministic, fast and binary. A tag is present or it isn't. A URL returns 200 or it doesn't. These make good gates. - Monitor checks that are noisy or depend on real users. Lab performance scores vary between runs, and Core Web Vitals field data comes from real visitors over weeks. Track these, but don't block merges on them. - Keep judgment human. Content quality, search intent, keyword targeting and title copywriting need a person. The Four Layers at a Glance 1. Pull request gate: lint the built HTML for missing or broken tags. Runs in seconds. 2. Lighthouse sample: audit a handful of key templates for SEO and performance budgets. 3. Post-deploy check: test the live server for robots rules, headers (including canonical), redirects and sitemap health. 4. Scheduled monitoring: catch slow drift such as broken external links and field performance. To decide which individual rules to codify, start from a reference list rather than a blank page. SEO Site Checker's 70 SEO checks explained groups on-page, performance, security and advanced checks with plain-language explanations, which makes it easy to pick the deterministic ones for your linter. Layer 1: Gate Pull Requests on the Built HTML The most important habit is to test the build output, not your source templates. Your framework, your CMS data and your layout components only come together in the final HTML, and that HTML is what search engines see. Point the checks at dist/, out/ or whatever folder your static export produces. Checks That Belong Here - Exactly one non-empty <title>, unique across the site, with a warning above 60 characters - A meta description, with a warning above 160 characters - Exactly one canonical tag, absolute and pointing to the production host - No noindex in meta robots, except on pages you deliberately allow - One <h1> per page (warning only, since some layouts have good reasons) - A lang attribute on <html> - An alt attribute on every image (an empty alt="" is fine for decorative images) - Basic Open Graph tags - JSON-LD that parses and includes @context and @type, including inside @graph A Small Checker Script This script uses cheerio to parse every HTML file in the build folder. It separates errors, which fail the build, from warnings, which are only printed. Install it with npm install --save-dev cheerio. // scripts/seo-check.mjs // Usage: node scripts/seo-check.mjs <build-dir> <production-origin> import { readdir, readFile } from 'node:fs/promises'; import path from 'node:path'; import * as cheerio from 'cheerio'; const [dir = 'dist', origin = 'https://example.com'] = process.argv.slice(2); const NOINDEX_ALLOWED = new Set(['404.html', 'search/index.html']); const errors = []; const warnings = []; const titles = new Map(); async function findHtml(folder) { const files = []; for (const entry of await readdir(folder, { withFileTypes: true })) { const full = path.join(folder, entry.name); if (entry.isDirectory()) files.push(...(await findHtml(full))); else if (entry.name.endsWith('.html')) files.push(full); } return files; } function checkPage(page, html) { const $ = cheerio.load(html); const error = (msg) => errors.push(`${page}: ${msg}`); const warn = (msg) => warnings.push(`${page}: ${msg}`); // Indexing const robots = ($('meta[name="robots"]').attr('content') ?? '').toLowerCase(); if (robots.includes('noindex') && !NOINDEX_ALLOWED.has(page)) error('meta robots contains noindex'); // Title const titleTags = $('head > title'); const title = titleTags.first().text().trim(); if (titleTags.length !== 1 || !title) { error(`expected one non-empty <title>, found ${titleTags.length}`); } else { if (title.length > 60) warn(`title is ${title.length} characters`); titles.set(title, [...(titles.get(title) ?? []), page]); } // Meta description const description = $('meta[name="description"]').attr('content')?.trim(); if (!description) error('missing meta description'); else if (description.length > 160) warn(`meta description is ${description.length} characters`); // Canonical const canonicals = $('link[rel~="canonical"]'); if (canonicals.length === 0) { error('missing canonical tag'); } else if (canonicals.length > 1) { error(`found ${canonicals.length} canonical tags, expected exactly one`); } else { const canonical = canonicals.attr('href'); try { const url = new URL(canonical); if (url.origin !== origin) error(`canonical points to ${url.origin}`); } catch { error(`canonical is not an absolute URL: ${canonical}`); } } // Headings, language, images, social const h1Count = $('h1').length; if (h1Count !== 1) warn(`found ${h1Count} <h1> elements`); if (!$('html').attr('lang')) warn('missing lang attribute on <html>'); $('img:not([alt])').each((_, img) => warn(`image without alt: ${$(img).attr('src')}`)); if (!$('meta[property="og:title"]').length) warn('missing og:title'); // Structured data $('script[type="application/ld+json"]').each((i, el) => { let data; try { data = JSON.parse($(el).text()); } catch { return error(`JSON-LD block ${i + 1} is not valid JSON`); } for (const block of Array.isArray(data) ? data : [data]) { if (!block['@context']) error(`JSON-LD block ${i + 1} has no @context`); const items = block['@graph'] ?? [block]; if (items.some((item) => !item['@type'])) error(`JSON-LD block ${i + 1} has an item without @type`); } }); } for (const file of await findHtml(dir)) { checkPage(path.relative(dir, file).split(path.sep).join('/'), await readFile(file, 'utf8')); } for (const [title, pages] of titles) { if (pages.length > 1) errors.push(`duplicate title "${title}" on: ${pages.join(', ')}`); } warnings.forEach((w) => console.warn(`warning ${w}`)); errors.forEach((e) => console.error(`error ${e}`)); console.log(`\n${errors.length} errors, ${warnings.length} warnings`); process.exit(errors.length ? 1 : 0); The canonical check counts every <link rel="canonical"> instead of reading the first one. A page with two (say a hand-written tag plus a framework or plugin default) sends conflicting signals, and Google may ignore both, so the check fails on anything other than exactly one. Using rel~= also matches tags like rel="canonical nofollow". Run against a sample build with a few planted mistakes, the output looks like this: warning blog/post/index.html: found 2 <h1> elements warning blog/post/index.html: missing lang attribute on <html> warning blog/post/index.html: image without alt: b.png warning blog/post/index.html: missing og:title error about/index.html: canonical is not an absolute URL: /about/ error blog/post/index.html: meta robots contains noindex error blog/post/index.html: missing meta description error blog/post/index.html: canonical points to https://staging.example.com error blog/post/index.html: JSON-LD block 1 is not valid JSON error services/index.html: found 2 canonical tags, expected exactly one error duplicate title "Home | Example" on: blog/post/index.html, index.html 7 errors, 4 warnings Every message names the page and the rule, so whoever opened the PR knows exactly what to fix. Wire It Into GitHub Actions # .github/workflows/seo-pr.yml name: SEO checks on: pull_request: jobs: seo: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 22 cache: npm - run: npm ci - run: npm run build - name: Lint built HTML run: node scripts/seo-check.mjs dist https://example.com - name: Check internal links uses: lycheeverse/lychee-action@v2 with: args: --offline --no-progress --root-dir "${{ github.workspace }}/dist" "dist/**/*.html" The lychee link checker runs in offline mode here, so it only checks links between your own files. That keeps it fast and immune to third-party sites being slow. External links get checked later, on a schedule. Keep this whole job under about a minute. If an SEO gate becomes the slowest step in CI, people start looking for ways around it. Layer 2: Lighthouse CI on Key Templates, Not Every Page Lighthouse is useful, but it's slow, and its performance scores change from run to run. Running it on every page of every PR is a recipe for flaky builds. Instead, audit three to five representative templates: the homepage, an article, a product or service page and a listing page. Lighthouse CI can serve your static build itself and assert on the results: { "ci": { "collect": { "staticDistDir": "./dist", "url": [ "http://localhost/index.html", "http://localhost/blog/sample-post/index.html", "http://localhost/pricing/index.html" ], "numberOfRuns": 3 }, "assert": { "assertions": { "categories:seo": ["error", { "minScore": 0.9 }], "categories:performance": ["warn", { "minScore": 0.8 }], "largest-contentful-paint": ["warn", { "maxNumericValue": 2500 }], "cumulative-layout-shift": ["warn", { "maxNumericValue": 0.1 }] } }, "upload": { "target": "filesystem", "outputDir": "./lhci-reports" } } } Save that as lighthouserc.json and add one step to the workflow: npx @lhci/cli autorun. Notice the split. The SEO category is an error, because its audits are mostly deterministic (crawlable links, valid hreflang, a document title and so on). Performance is a warning, because CI runners vary in speed. Three runs per URL lets Lighthouse CI use the median result, which smooths out the noise. Layer 3: Check Production Right After Deploy Some of the most damaging problems only exist on the live server. Response headers, robots.txt, redirects and host settings often come from CDN rules, server config or a hosting dashboard rather than your repository. A PR check can't see them, so test production immediately after every deploy. robots.txt and X-Robots-Tag Fail the run if robots.txt blocks everything for User-agent: *, or if key pages send noindex through an X-Robots-Tag header or a meta tag. The header is easy to miss because it never appears in the HTML. Google's robots meta tag and X-Robots-Tag docs explain how both are interpreted. Be precise about Disallow: /. Plenty of sites block specific bots, such as AI crawlers, on purpose, so the check should only fail when the rule applies to every crawler. Canonical in the Link Header A canonical can also be sent as an HTTP header (Link: <https://example.com/page>; rel="canonical"). If it disagrees with the one in the HTML, you've sent conflicting signals, and the mismatch never shows up in the built files. The script below compares both against the live response, and also counts canonicals in the live HTML, which catches duplicates that a CDN or edge script injects after the build. A Redirect Map Test Keep a redirects.csv file in the repository with two columns, from and to. After each deploy, assert that every old URL returns a single 301 or 308 straight to the expected target, and that the target returns 200. This catches missing redirects, temporary 302s, wrong destinations and chains. Sitemap Sanity Request every URL in the sitemap and expect a 200. Then compare the URL count against a stored baseline, and fail if it drops sharply. A sitemap that suddenly loses a large share of its URLs usually means a routing change or a broken data source, not a deliberate cleanup. A count alone can hide a problem, though. Delete ten established pages, add ten unrelated ones, and the total doesn't move. So the baseline also stores the URL set. Every URL that disappeared since the last clean run must have an entry in redirects.csv, otherwise it's either a missing route or a missing redirect, and the run fails. The removed URLs are also printed, so whoever reviews the deploy gets a concrete list instead of a reassuring total. The Post-Deploy Script // scripts/post-deploy-check.mjs // Usage: node scripts/post-deploy-check.mjs https://example.com import { readFile, writeFile } from 'node:fs/promises'; const origin = (process.argv[2] ?? 'https://example.com').replace(/\/$/, ''); const KEY_PATHS = ['/', '/blog/', '/pricing/']; const MAX_SITEMAP_DROP = 0.1; // fail if the sitemap shrinks by more than 10% const failures = []; const fail = (msg) => failures.push(msg); // 1. robots.txt must not block everything for all crawlers function blocksAllCrawlers(txt) { let agents = []; let inRules = false; for (const raw of txt.split(/\r?\n/)) { const line = raw.replace(/#.*/, '').trim(); const sep = line.indexOf(':'); if (sep === -1) continue; const key = line.slice(0, sep).trim().toLowerCase(); const value = line.slice(sep + 1).trim(); if (key === 'user-agent') { if (inRules) { agents = []; inRules = false; } agents.push(value); } else if (key === 'allow' || key === 'disallow') { inRules = true; if (key === 'disallow' && value === '/' && agents.includes('*')) return true; } } return false; } const robotsRes = await fetch(`${origin}/robots.txt`); if (!robotsRes.ok) fail(`robots.txt returned ${robotsRes.status}`); else if (blocksAllCrawlers(await robotsRes.text())) fail('robots.txt has "Disallow: /" for User-agent: *'); // 2. Key pages: 200, no noindex, one canonical that matches the Link header for (const p of KEY_PATHS) { const res = await fetch(origin + p); if (res.status !== 200) { fail(`${p} returned ${res.status}`); continue; } if (/noindex/i.test(res.headers.get('x-robots-tag') ?? '')) fail(`${p} sends X-Robots-Tag: noindex`); const html = await res.text(); if (/<meta[^>]+name=["']robots["'][^>]+noindex/i.test(html)) fail(`${p} has a noindex meta tag`); const htmlCanonicals = [...html.matchAll(/<link\b[^>]*\brel=["'][^"']*\bcanonical\b[^"']*["'][^>]*>/gi)] .map((m) => m[0].match(/\bhref=["']([^"']+)["']/i)?.[1]) .filter(Boolean); if (htmlCanonicals.length > 1) fail(`${p} has ${htmlCanonicals.length} canonical tags in the live HTML`); const headerCanonical = (res.headers.get('link') ?? '') .match(/<([^>]+)>\s*;[^,]*\brel=["']?canonical["']?/i)?.[1]; if (headerCanonical && htmlCanonicals[0]) { const a = new URL(headerCanonical, origin).href; const b = new URL(htmlCanonicals[0], origin).href; if (a !== b) fail(`${p} Link header canonical (${a}) differs from HTML canonical (${b})`); } } // 3. Redirect map: one hop, permanent, correct target const csv = await readFile('redirects.csv', 'utf8').catch(() => ''); const redirectedPaths = new Set(); for (const line of csv.split(/\r?\n/).slice(1).filter(Boolean)) { const [from, to] = line.split(',').map((s) => s.trim()); redirectedPaths.add(new URL(from, origin).pathname); const res = await fetch(origin + from, { redirect: 'manual' }); const location = res.headers.get('location'); if (![301, 308].includes(res.status)) { fail(`${from} returned ${res.status}, expected 301 or 308`); continue; } const target = new URL(location, origin); if (target.href !== new URL(to, origin).href) { fail(`${from} redirects to ${target.pathname}, expected ${to}`); continue; } const final = await fetch(target, { redirect: 'manual' }); if (final.status !== 200) fail(`${from} -> ${to} ends in ${final.status} (chain or broken target)`); } // 4. Sitemap: every URL returns 200, the count has not dropped sharply, // and every URL removed since the last clean run has a redirect const sitemapRes = await fetch(`${origin}/sitemap.xml`); if (!sitemapRes.ok) { fail(`sitemap.xml returned ${sitemapRes.status}`); } else { const urls = [...(await sitemapRes.text()).matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)].map((m) => m[1]); const baseline = JSON.parse(await readFile('seo-baseline.json', 'utf8').catch(() => '{}')); if (baseline.sitemapCount && urls.length < baseline.sitemapCount * (1 - MAX_SITEMAP_DROP)) { fail(`sitemap has ${urls.length} URLs, down from ${baseline.sitemapCount}`); } const current = new Set(urls.map((u) => new URL(u).pathname)); const removed = (baseline.urls ?? []).filter((u) => !current.has(u)); const unredirected = removed.filter((u) => !redirectedPaths.has(u)); if (removed.length) console.log(`Removed since last clean run: ${removed.join(', ')}`); unredirected.forEach((u) => fail(`${u} was removed from the sitemap with no entry in redirects.csv`)); for (let i = 0; i < urls.length; i += 10) { await Promise.all(urls.slice(i, i + 10).map(async (url) => { const res = await fetch(url, { redirect: 'manual' }); if (res.status !== 200) fail(`sitemap URL ${url} returned ${res.status}`); })); } if (!failures.length) await writeFile('seo-baseline.json', JSON.stringify({ sitemapCount: urls.length, urls: [...current].sort() }, null, 2)); } failures.forEach((f) => console.error(`FAIL ${f}`)); console.log(failures.length ? `\n${failures.length} problems found` : 'All post-deploy SEO checks passed'); process.exit(failures.length ? 1 : 0); Tested against a deliberately broken server, it reports: Removed since last clean run: /old-pricing, /removed/ FAIL robots.txt has "Disallow: /" for User-agent: * FAIL / has 2 canonical tags in the live HTML FAIL /blog/ Link header canonical (http://localhost:8765/news/) differs from HTML canonical (http://localhost:8765/blog/) FAIL /pricing/ sends X-Robots-Tag: noindex FAIL /old-pricing redirects to /plans/, expected /pricing/ FAIL /old-blog returned 302, expected 301 or 308 FAIL /removed/ was removed from the sitemap with no entry in redirects.csv FAIL sitemap URL http://localhost:8765/gone/ returned 404 8 problems found A few notes on the script: - The baseline file only updates after a clean run, so one bad deploy can't quietly lower the bar. On GitHub Actions, persist seo-baseline.json with actions/cache or commit it, because each runner starts fresh. - /old-pricing left the sitemap too, but it has a redirect entry, so only /removed/ fails. The sitemap count was unchanged, so a count-only check would have passed this deploy. - It reads a single sitemap.xml. If you use a sitemap index, fetch each child sitemap first. - Sitemap URLs are checked ten at a time to avoid hammering your own server. Trigger it after a successful production deployment: # .github/workflows/seo-post-deploy.yml name: Post-deploy SEO check on: deployment_status: jobs: smoke: if: github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'production' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 22 - run: node scripts/post-deploy-check.mjs https://example.com Check what your host calls the production environment. Some platforms report it as Production with a capital P. If you deploy from your own workflow instead, run the script as a final job that depends on the deploy job. Layer 4: Scheduled Monitoring for What CI Can't Judge Some problems develop slowly or depend on things outside your code. Handle them with a scheduled workflow that alerts people instead of failing builds: on: schedule: - cron: "17 6 * * 1" # Mondays at 06:17 UTC workflow_dispatch: Good candidates for a weekly run: - External link checks with lychee in online mode, since other sites change without telling you - Core Web Vitals field data from the Chrome UX Report (CrUX) API, which reflects real users rather than a lab run - Indexing coverage through the Search Console API, to spot a jump in excluded pages - The post-deploy script again, to catch config changes made outside a deploy Post the results to Slack or open a GitHub issue. Nobody should be blocked from merging because a partner's website went down. Automation also doesn't replace a periodic human look at the whole site. Every month or so, run a full SEO audit on your key pages and compare it with the last one. What Not to Automate It's tempting to turn everything into a gate. Resist that for: - Content quality and helpfulness. No linter can tell whether a page actually answers the searcher's question. - Search intent and keyword choices. These are strategy decisions, not syntax errors. - Title copywriting. You can check length automatically, but not whether the title is good. - Internal linking strategy. You can check for broken links, but deciding what to link is editorial work. - Lighthouse scores on every page as hard gates. They flake, people stop trusting the pipeline, and then they ignore real failures too. Automation keeps the floor up. Humans still have to raise the ceiling. For the judgment side, a manual technical SEO audit checklist covers what a script can't. Rolling It Out Without Annoying Your Team - Start with warnings only. Run every check in warning mode for two weeks, fix the existing backlog, then promote the stable checks to errors. - Keep an allowlist. Some pages, like internal search results or a 404 page, should be noindex. Put exceptions in one obvious place, like NOINDEX_ALLOWED in the script. - Make failures readable. Each message should name the page, the rule and the fix. Consider posting a summary as a sticky PR comment. - Give every check an owner. When a rule fires falsely, someone needs to fix the rule rather than add || true to the workflow. Frequently Asked Questions Can you automate SEO testing? Yes, for the technical layer. Anything with a clear pass or fail can be automated: tags, status codes, redirects, robots rules, structured data syntax and sitemap health. Content quality, search intent and strategy still need human review, so automation complements an SEO specialist rather than replacing one. Is Lighthouse enough for SEO checks in CI? No. Lighthouse's SEO category covers useful basics, but it audits one page at a time, doesn't compare titles across pages, can't see your redirects or robots.txt in production, and doesn't validate your structured data against your own rules. Use it as one layer alongside an HTML linter and post-deploy checks. Should SEO checks fail the build or just warn? Fail the build for deterministic, high-impact problems such as noindex on indexable pages, missing titles, missing, duplicate or wrong canonicals and invalid JSON-LD. Warn for things that are noisy or subjective, like performance scores, title length or multiple H1 tags. Start everything as a warning, then promote checks once they've proven reliable. How do I test for noindex and robots.txt mistakes before they reach production? Lint the built HTML for noindex meta tags in every pull request, then run a post-deploy check against production for robots.txt rules and X-Robots-Tag headers. Headers and robots files often come from server or CDN config rather than your code, so the production check is essential. How do I check for broken links in GitHub Actions? Use a link checker such as lychee through its GitHub Action. Run it in offline mode on your build output during pull requests to check internal links quickly, then run it in online mode on a schedule to check external links without blocking merges when a third-party site is down. Do SEO checks slow down a CI pipeline? They don't have to. An HTML linter over a typical static build finishes in seconds. Lighthouse is the slow part, so limit it to a few key templates. Post-deploy and scheduled checks run outside the pull request flow entirely, so they never block developers. Wrapping Up A sensible SEO pipeline has four layers. It gates pull requests on the built HTML, samples key templates with Lighthouse, checks production right after deploy, and monitors slow drift on a schedule. Automate what's binary, monitor what's noisy, and keep judgment human. When you're ready to add your next rule, browse the full list of on-page and technical SEO checks and pick one that your pipeline doesn't cover yet. --- Update: thanks to Posting Dude for pointing out the duplicate-canonical and Link-header gaps, and to Marcus Kim for suggesting the URL-set baseline diffed against the redirect map. All three are now in the scripts above.
Advanced Performance Monitoring with Lighthouse, PageSpeed Insights, and Search Console --- title: Advanced Performance Monitoring with Lighthouse, PageSpeed Insights, and Search Console published: true published_at: 2026-10-20 13:00 +0000 tags: performance, lighthouse, pagespeedinsights, searchconsole canonical_url: khriztianmoreno.dev --- Tools do not fix performance. Systems do. Most developers treat performance as a checklist: run a Lighthouse audit, fix a few warnings, and move on. But performance is not a static state; it is a changing ecosystem. Your users are on different devices, networks, and browsers. Your code changes daily. If you are only looking at performance when you remember to run a report, you are already behind. This guide explains how to move from one-off audits to continuous performance monitoring using the three pillars of Google's tooling ecosystem: Lighthouse, PageSpeed Insights (PSI), and Google Search Console (GSC). 1. Intro: Performance is a system, not a report We have all been there. You run a Lighthouse audit locally and get a 95. You push to production, run it again, and get a 72. You check PageSpeed Insights, and it shows completely different numbers. Then you look at Search Console, and it says everything is fine—but the data is from three weeks ago. It feels contradictory, but it isn't. Each tool is answering a different question. To build a robust performance culture, you must stop chasing a single "score" and start listening to the signals your system is sending. 2. The three pillars of Google performance tooling Before diving into workflows, let's establish a mental model for what each tool actually does. | Tool | Data Type | Question it Answers | | :--------------------- | :------------------- | :--------------------------- | | Lighthouse | Synthetic (Lab) | _What could go wrong?_ | | PageSpeed Insights | Hybrid (Lab + Field) | _What is going wrong?_ | | Search Console | Field (CrUX) | _Does Google see a problem?_ | Understanding this distinction is crucial. You use Lighthouse to catch regressions before they ship. You use PageSpeed Insights to validate real-world impact. You use Search Console to track long-term health and SEO ranking signals. 3. Lighthouse: advanced usage beyond scores Lighthouse is a lab tool. It runs in a controlled environment (your machine or a CI server) with a predefined device and network profile. Because it is synthetic, it is deterministic and reproducible. This makes it the ideal tool for Continuous Integration (CI). Advanced Lighthouse workflows Don't just run Lighthouse manually in Chrome DevTools. That relies on your local computer's CPU power and extensions, which skews the results. Run Lighthouse in CI to prevent performance regressions before they merge. # Example: Running Lighthouse via CLI npx lighthouse https://example.com \ --preset=desktop \ --only-categories=performance By integrating Lighthouse CI, you can enforce performance budgets. For example, fail the build if the bundle size exceeds 200KB or if the Largest Contentful Paint (LCP) is forecasted to be over 2.5s. Ignore the score, read the trace The 0-100 score is a summary for managers. Engineers need the Trace. The trace reveals exactly what happened during the page load: - Main-thread blocking time: Which specific JavaScript function locked up the UI? - Render-blocking resources: Did a CSS file delay the first paint? - Unused JavaScript: Are you shipping library code that is never executed? Use the Lighthouse Trace Viewer to drill down into the milliseconds.  4. PageSpeed Insights: bridging lab and field PageSpeed Insights (PSI) is often misunderstood because it presents two sets of data next to each other: 1. Lab Data (Lighthouse): A simulation run on Google's servers. 2. Field Data (CrUX): Real data from real users visiting your site (Chrome User Experience Report). This is why teams get confused. _"Lighthouse says my LCP is 1.2s, but Field Data says it's 3.5s. Which is true?"_ The Field Data is the truth. It represents what your actual users are experiencing. If your Lab data is fast but Field data is slow, your local simulation assumes a faster network or device than your users actually have. Reading PSI correctly - Field data → Reality. This is what Google uses for ranking. If this is red, you have a problem, no matter what Lighthouse says. - Lab data → Diagnosis. Use this to reproduce issues found in the field. - Opportunities → Hypotheses. These are algorithmic suggestions. Not all of them will improve Core Web Vitals. Important Rule: Never optimize for a Lab metric that does not correlate with a Field problem. If PSI says "Remove unused CSS" but your FCP is already 0.8s in the field, that optimization is a waste of time. 5. Google Search Console: performance at scale Google Search Console (GSC) provides the "Core Web Vitals" report. This is aggregated CrUX data grouped by URL patterns. Limitations - Delayed: Data is a 28-day rolling average. You won't see the impact of a deploy today until weeks later. - Aggregated: It groups pages together (e.g., "200 similar URLs"). It can be hard to debug a specific outlier page. Why it matters Despite the lag, GSC is critical because this is exactly how Google groups and ranks your site. It tells you if you have systemic issues affecting entire sections of your application (e.g., "All product pages have poor CLS"). Use GSC to: 1. Detect systemic issues: If CLS spikes across all /blog/* pages, you likely introduced a layout shift in your blog post template. 2. Validate long-term improvements: After a fix, use the "Validate Fix" button to tell Google to start verifying the new data. 6. Synthetic vs. Real User Monitoring (RUM) This distinction is the conceptual core of modern performance engineering. Synthetic Monitoring (Lighthouse, WebPageTest) - Pros: Fast feedback, controlled environment, cheap to run, CI-friendly. - Cons: Not representative of real users, misses device diversity, "clean room" environment. - Best for: Preventing regressions during development. Real User Monitoring (CrUX, web-vitals.js) - Pros: Actual user experience, captures device/network diversity, correlates with business metrics and SEO. - Cons: Slower feedback loop, requires instrumentation, data can be noisy. - Best for: Understanding reality and verifying fixes. Why you need both: Synthetic finds problems early (while coding). RUM confirms they actually matter to users. 7. Building a modern monitoring stack A mature engineering team uses a pipeline that looks like this: 1. Local Development: Engineers use Lighthouse in DevTools to spot obvious issues. 2. CI/CD Pipeline: Lighthouse CI runs on every Pull Request. It blocks merges if metrics degrade beyond a threshold. 3. Production (Synthetic): A service runs a Lighthouse check on key pages every hour to catch infrastructure issues or third-party script regressions. 4. Production (RUM): The site reports Core Web Vitals to an analytics endpoint (using web-vitals.js) to track real-time trends. 5. SEO Health: The team reviews Search Console weekly to ensure no new URL groups are flagged as "Poor". 8. Common mistakes teams make - Chasing Lighthouse 100: A score of 100 on a Developer's MacBook Pro means nothing if your users are on low-end Android devices. - Ignoring INP in field data: Interaction to Next Paint (INP) is hard to reproduce in Lab tools because they don't click around. You _must_ rely on Field data for INP. - Treating Search Console as real-time: Don't panic if GSC doesn't update the day after a fix. It takes time. - Optimizing Lab-only regressions: If Lighthouse complains about a metric that looks green in CrUX, deprioritize it. 9. What’s next: where monitoring is going Performance monitoring is moving towards attribution. It's not enough to know _that_ the page is slow; we need to know _what code_ made it slow. - Route-level CWV: Tools are getting better at attributing metrics to specific SPA routes (Soft Navigations). - More granular CrUX data: Google is exposing more detailed data in the CrUX History API. - Interaction breakdowns: LoAF (Long Animation Frames) API is revolutionizing how we debug main-thread blocking, giving us stack traces for long tasks in the wild. 10. Conclusion No single tool gives you the full picture. Lighthouse helps you build fast software. PageSpeed Insights helps you debug user issues. Search Console helps you maintain your search ranking. Performance is not a task you finish. It is a signal you continuously listen to. Start listening today. Key References - Web Vitals Documentation - Google Search Central: Core Web Vitals - Lighthouse Documentation - PageSpeed Insights I hope this has been helpful and/or taught you something new! !Profile @khriztianmoreno Until next time.