Visual Regression Testing

Functional tests check that a button exists and does the right thing. They do not notice that a CSS change pushed the button off-screen on narrow viewports, that a font failed to load, or that the checkout total now overlaps the coupon field. Visual regression testing catches these by comparing screenshots of the UI against approved baseline images and failing when they differ beyond a threshold.

It is powerful and, done carelessly, the flakiest kind of test you can own. This lesson is about getting the value without the noise.

How screenshot comparison works

1. The first run captures a screenshot and stores it as the baseline (also called a golden or reference image).
2. Later runs capture a new screenshot of the same page or component in the same state.
3. A comparison algorithm checks the images pixel by pixel, with a tolerance for tiny colour differences.
4. If the difference exceeds the threshold, the test fails and produces a diff image highlighting the changes.
5. A person reviews the diff. If the change is intended, they update the baseline; if not, it is a bug.

The tool cannot tell whether a change is good. It only tells you that something changed. Visual testing is therefore a change detector plus human review, and your process must make that review easy.

Screenshot assertions in Playwright

Playwright Test has visual comparison built in:

import { test, expect } from '@playwright/test';

test('product page looks right', async ({ page }) => {
  await page.goto('/products/blue-mug');
  await expect(page.getByRole('heading', { name: 'Blue Mug' })).toBeVisible();
  await expect(page).toHaveScreenshot('product-page.png');
});

test('price card component', async ({ page }) => {
  await page.goto('/products/blue-mug');
  await expect(page.getByTestId('price-card')).toHaveScreenshot('price-card.png');
});

toHaveScreenshot is a web-first assertion: it takes screenshots repeatedly until two consecutive captures match (the page has stopped changing) and then compares with the baseline, within the assertion timeout. On the first run, with no baseline, the test fails and writes the actual screenshot as the new baseline; commit it after checking it looks right.

Element screenshots, like the price card above, are usually more stable and more useful than full pages. A full-page diff fails on any change anywhere, and the failure does not tell you what component is responsible.

Making screenshots stable

await expect(page).toHaveScreenshot('dashboard.png', {
  fullPage: true,
  animations: 'disabled',
  mask: [page.getByTestId('last-updated'), page.locator('.user-avatar')],
  maxDiffPixelRatio: 0.01,
});

• animations: 'disabled' stops CSS animations and transitions (finite ones are fast-forwarded to their end, infinite ones reset to their initial state), so captures are not taken mid-animation.
• mask covers the given locators with a solid box. Use it for content that legitimately changes: timestamps, avatars, ads, random recommendations, live data.
• maxDiffPixelRatio allows a fraction of pixels (0 to 1) to differ. maxDiffPixels allows an absolute count instead. threshold (0 to 1) controls how different a single pixel’s colour must be to count as different.
• The text caret is hidden by default (caret: 'hide'), so a focused input does not cause blinking-cursor diffs.

Set sensible defaults once in the config rather than repeating them:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  expect: {
    toHaveScreenshot: {
      maxDiffPixelRatio: 0.01,
      animations: 'disabled',
    },
  },
});

Controlling the data and state

The biggest source of visual flakiness is not rendering; it is content. Freeze anything that varies:

• Use fixed test data (seeded accounts, fixed product catalogue) or mock API responses with page.route.
• Freeze time with Playwright’s clock API (page.clock) when the UI shows dates or relative times.
• Wait for fonts and images: assert on a visible element first, and avoid screenshots while skeleton loaders are showing.
• Set a fixed viewport and, where relevant, colorScheme and locale in the project config.
• Dismiss cookie banners and chat widgets consistently, for example by setting the consent cookie in the context.

Baselines across operating systems

Fonts, anti-aliasing and rendering differ between Windows, macOS and Linux, and between browsers. A screenshot taken on a Mac will not match one taken on a Linux CI runner, even for identical HTML.

Playwright handles this by naming baselines per browser project and platform. By default a snapshot called product-page.png in a project named chromium on Linux is stored as something like:

tests/product.spec.ts-snapshots/product-page-chromium-linux.png

and on macOS it would look for product-page-chromium-darwin.png. Each platform has its own baselines, so a developer on macOS does not fail against Linux images; they simply have no baseline yet. You can change the layout with the snapshotPathTemplate config option.

That raises a practical question: which platform is the source of truth? For most teams the answer is “the CI platform”, usually Linux. Then you need a way for developers on macOS or Windows to produce and update Linux baselines. That is where Docker helps.

Docker for consistent rendering

Running the tests inside the official Playwright image gives everyone, locally and in CI, the same OS, fonts and browser builds:

docker run --rm --ipc=host -v "$(pwd):/work" -w /work \
  mcr.microsoft.com/playwright:v1.65.0-noble \
  /bin/bash -c "npm ci && npx playwright test --update-snapshots"

Keep the image tag identical to your @playwright/test version and identical between local updates and CI. Upgrading Playwright (and therefore browser versions) or the base image can shift rendering slightly, so plan baseline refreshes as part of those upgrades and review the diffs carefully.

Notice that npm ci runs inside the container, so node_modules are built for Linux. If your project folder is mounted from macOS or Windows, avoid reusing host-installed node_modules inside the container.

The review workflow

Visual tests are only as good as the review of their failures:

• Upload the HTML report as a CI artifact. Playwright’s report shows expected, actual and diff images side by side, with a slider.
• Never update baselines blindly. --update-snapshots overwrites baselines with whatever the page currently looks like, including bugs. Review every changed image in the PR.
• Keep baselines in the repository next to the tests, so a UI change and its baseline update are reviewed together. If the images are large and many, consider Git LFS.
• Assign ownership. Designers or front-end owners should be able to approve visual changes in their area.

Choosing scope and tools

Visual tests are relatively expensive to maintain, so aim them where they pay off:

• Design-system components in a component explorer such as Storybook (one screenshot per component state), which gives stable, isolated images.
• A handful of key pages per critical journey, at the viewports your users actually use.
• Layout-sensitive areas: responsive navigation, tables, checkout summaries, emails and PDFs.

Avoid screenshotting every page in every state; you will spend your time approving diffs.

Beyond Playwright’s built-in comparison, there are dedicated visual testing services and tools. Commercial services such as Percy and Applitools offer hosted baseline management, review dashboards and cross-browser rendering, and some use comparison approaches that aim to ignore insignificant rendering noise. Open-source options exist too. Evaluate them against your actual pain: if the problem is review workflow and cross-browser coverage at scale, a service may help; if it is unstable test data, no tool will fix it.

Practice exercise

A team added 80 full-page screenshot tests. Developers on macOS see constant failures, CI on Linux fails about once a day on the dashboard page (which shows “Last updated 3 minutes ago” and a carousel), and someone set maxDiffPixelRatio: 0.2 to make the noise stop. Propose a recovery plan.

Show a model answer

1. Single source of truth: treat Linux CI as the baseline platform. Run visual tests (locally and in CI) in the mcr.microsoft.com/playwright:v<version>-noble image matching the project’s Playwright version. Stop running visual tests natively on macOS, or accept that macOS has no baselines.
2. Baseline updates via CI: a manual workflow runs --update-snapshots in the container and opens a PR; images are reviewed in the PR.
3. Fix the dashboard: mask the “Last updated” text, freeze time with the clock API, use animations: 'disabled', and stop or mask the carousel (or fix its first slide via test data).
4. Reduce the tolerance: remove 0.2 (20% of pixels may change, which hides real regressions); start near zero, prefer maxDiffPixels for small known noise, and justify any increase.
5. Narrow the scope: replace many full-page shots with component-level screenshots of key areas, and keep a few full pages for critical journeys.
6. Stabilise data: seed fixed data or mock API responses for the pages under test.

Key takeaways

• Visual regression testing detects unintended UI changes; humans decide whether a change is a bug.
• Use await expect(page).toHaveScreenshot() or element screenshots, with animations: 'disabled', mask for dynamic regions and strict thresholds.
• Baselines are per browser and platform; pick one source of truth, usually Linux CI.
• Run visual tests in the official Playwright Docker image with a version matching your package, locally and in CI.
• Never update baselines blindly; review diffs in pull requests.
• Focus on components and key pages; tools such as Percy and Applitools exist for hosted review at scale.