Regression Testing

Software is interconnected. A change to the tax calculation can break invoices, a library upgrade can change date formatting everywhere, and a configuration tweak can disable a feature nobody touched. Regression testing is the re-testing of previously working functionality after a change, to detect defects that the change introduced or exposed. Those defects are called regressions.

Regression testing is where automation earns most of its value, and where badly managed suites waste the most time. This lesson covers how to decide what to run and how to keep the suite healthy.

Regression vs confirmation testing

ISTQB treats these as two distinct kinds of change-related testing:

	Confirmation testing (retesting)	Regression testing
Purpose	Check that a specific defect has been fixed	Check that a change has not broken anything else
Scope	The original failing test, plus close variations	Areas that could be affected by the change
When	After a defect fix	After any change: fix, feature, refactor, config, dependency, environment
Typical size	Small	Small to very large

After a bug fix you usually need both: confirm the fix, then regress around it. See The Bug Lifecycle for how this fits into defect workflow.

What triggers regression testing

Any change can cause regressions, not just code changes in the feature you are testing:

• New features and enhancements.
• Defect fixes (a fix in one place often has side effects elsewhere).
• Refactoring, where behaviour is meant to stay the same.
• Dependency upgrades: frameworks, libraries, runtime versions, browsers.
• Configuration and infrastructure changes: feature flags, environment variables, database migrations, cloud settings.
• Changes in integrated systems you do not own, such as a payment provider’s API.

Choosing what to run

There are three broad strategies, and most teams combine them.

Retest all

Run the entire regression suite. It is the safest option and the simplest to reason about, but it gets slower as the product grows. It is realistic when the suite is automated, parallelised and fast, or before a major release.

Risk-based selection

Prioritise tests by risk: the likelihood that an area is broken multiplied by the impact if it is. Factors include:

• Business criticality: login, checkout, payments, data export, anything regulated.
• Change proximity: areas close to the change in code or in behaviour.
• Defect history: modules that have had many bugs tend to keep having them (defects cluster).
• Complexity and integrations: complex logic and interfaces between systems break more often.
• Usage: the paths most users take every day.

Impact analysis

Work out what the change can affect and select tests for those areas. Inputs include the code diff, the architecture (which services call the changed one), the requirements traceability matrix, and a conversation with the developer. Some teams automate parts of this, for example by running only tests that cover changed files, but even the automated forms are a heuristic and need a safety net such as a nightly full run.

Strategy	Strength	Weakness
Retest all	Maximum coverage, no judgement needed	Slow and expensive at scale
Risk-based	Focuses effort where failure hurts most	Depends on good risk knowledge; can miss surprising side effects
Impact analysis	Targets the actual change	Hidden dependencies (config, shared data, third parties) are easy to miss

Organising a regression suite in tiers

A single, monolithic “regression suite” that takes hours tends to be run rarely and trusted little. Split it into tiers that run at different points:

Tier	Contents	Runs	Target duration
Pre-merge	Unit tests, fast API/component tests, a few critical UI journeys	Every pull request	Minutes
Post-merge / deploy	Smoke tests plus broader API and UI coverage of critical paths	Every deployment to a test environment	Tens of minutes at most
Nightly / full	The full automated regression suite across browsers and data variations	Nightly or before release	Can be longer; must still finish before the working day
Release	Full automated suite plus targeted manual and exploratory testing of risky areas	Before a release	Planned in advance

Tag tests so tiers can be selected without copying files. In Playwright, tags can be added in the test title or the tag option and filtered with --grep:

import { test, expect } from '@playwright/test';

test('guest can complete checkout with a card', { tag: ['@critical', '@checkout'] }, async ({ page }) => {
  await page.goto('/');
  // ...
});

npx playwright test --grep @critical          # pre-merge tier
npx playwright test --grep-invert @slow       # everything except slow tests

In pytest the equivalent is markers (@pytest.mark.critical, run with pytest -m critical); in JUnit 5 it is @Tag("critical"), selected through your build tool’s include-tags configuration; in TestNG it is groups.

The test pyramid applies to regression too

Most regression coverage should sit at the lowest level that can detect the problem. A tax calculation regression is best caught by unit tests on the calculation, a contract change by API tests, and only the integration of the whole journey by a UI test. UI tests are slower and more brittle, so a regression suite made mostly of UI tests becomes slow and flaky as it grows.

Keeping the suite healthy

Regression suites decay unless someone maintains them. Watch for these problems:

• Flaky tests. A test that sometimes fails without a product change teaches people to ignore failures. Quarantine flaky tests quickly (move them out of the blocking tier, with a ticket), fix the cause, then return them. Retries can hide flakiness; use them as a diagnostic aid, not a cure.
• Obsolete tests. Features change. Tests for removed or redesigned behaviour should be updated or deleted, not left skipped forever.
• Duplicate coverage. The same rule checked at unit, API and UI level wastes time. Keep the cheapest effective check, plus one end-to-end journey.
• The pesticide paradox. Running the same tests repeatedly stops finding new defects. Refresh test data, add tests for recent escaped bugs, and complement automation with exploratory testing.
• Slow feedback. Parallelise, shard, and move checks down the pyramid. A suite that finishes after everyone has gone home does not influence decisions.

Feeding the suite

Good sources of new regression tests:

• Every escaped defect. When a bug reaches production, add a test that would have caught it, at the lowest sensible level.
• Every fixed defect in a high-risk area.
• New features, once they are stable: their acceptance tests become tomorrow’s regression tests.
• Exploratory session findings that reveal fragile behaviour.

Manual regression

Not everything is worth automating. Visual polish, complex one-off setups, and rarely changing areas may be cheaper to check manually with a checklist before a release. Keep manual regression lists short, risk-based and reviewed each release, and track which items are candidates for automation. The Test Execution Report can summarise a manual regression run.

Practice exercise

A developer changes the function that rounds currency amounts, used across an e-commerce platform. The full automated regression suite takes three hours. You have 40 minutes before the release decision.

1. Using impact analysis and risk, list the areas you would test and in what order.
2. Which tests would you run automatically and what would you check manually?
3. What would you add to the suite afterwards?

Show a model answer

1. Areas, in order

1. Checkout totals: subtotal, discounts, tax, shipping, grand total, in at least two currencies (one with no minor units, such as JPY, if supported).
2. Payment amounts sent to the provider: the charged amount must equal the displayed total.
3. Refunds and partial refunds.
4. Invoices and order confirmation emails.
5. Reports and exports that sum amounts.
6. Display-only formatting such as product listing prices.

The order follows impact (money moving and legal documents first) and proximity to the change.

2. Automated vs manual

Run automated unit tests for the rounding function and all tax and total calculations, the API tests for cart, checkout, payment and refunds tagged @critical, and one end-to-end checkout journey per supported currency. Manually check a generated invoice PDF and a confirmation email, and spot-check a finance report. Report clearly what was not run in the time available, and recommend the full suite runs immediately after release if the release goes ahead.

3. Afterwards

Add unit tests with boundary values for rounding (for example values ending in exactly .005, negative amounts for refunds, very large totals) and an API test asserting that the amount sent to the payment provider equals the order total.

Key takeaways

• Regression testing re-checks previously working behaviour after any change; confirmation testing checks a specific fix.
• Combine retest all, risk-based selection and impact analysis depending on time and risk.
• Organise the suite into tiers (pre-merge, post-deploy, nightly, release) and select them with tags.
• Keep most regression checks low in the test pyramid and keep the suite green, fast and trusted.
• Every escaped defect should produce a new regression test.