Which tests should run on a pull request, on merge, and nightly?

The goal is the fastest useful feedback at each point. A developer waiting on a pull request needs an answer in minutes; a release decision can wait hours for a deeper answer. So I tier suites by speed, reliability and cost, not by “what we have”.

On every pull request

Target: well under 15 minutes, close to zero flakiness, because a PR check that fails randomly teaches people to re-run and ignore it.

• Static checks: lint, formatting, type-check, dependency and secret scanning.
• Unit and component tests.
• API or service-level tests against the service running in the pipeline, with dependencies stubbed or in containers.
• Contract tests if the service publishes or consumes APIs.
• A small UI smoke set covering critical journeys (login, core transaction).

Optionally, test impact analysis: run the full component suite but only the UI tests tagged for the changed area.

After merge to main

The code is now deployed to a shared environment (dev or staging), so this is where real integration shows up.

• Broader API and UI regression against the deployed build.
• Integration with real downstream services where available.
• Post-deployment smoke checks that block promotion to the next environment.

Nightly or scheduled

Slow, expensive or noisy suites belong here, with someone owning the results every morning:

• Full end-to-end regression, cross-browser and mobile viewport runs.
• Performance and load tests (often weekly, or before a release).
• DAST security scans and accessibility audits of the full site.
• Long-running data or migration checks.

Before release

A release candidate gets the gates that matter for that release: full regression green or with triaged known issues, performance compared against the previous baseline, and any manual exploratory sessions for new features.

Why this split

Each tier has a different cost of failure. A bug caught on the PR costs the author minutes. A bug caught nightly is found within a day but needs triage to find the culprit commit. If the nightly run keeps finding issues that PR checks could have caught, push those tests left: write lower-level tests for that area or promote a few into the PR smoke set.

Common mistakes

• Running the entire UI suite on every PR, making PRs slow and flaky.
• Nightly suites that nobody looks at.
• No smoke check after deployment, so a broken environment fails every later suite.

What interviewers look for

• A clear tiering by feedback speed and cost.
• Reliability requirements for PR-blocking checks.
• Ownership of scheduled results and the habit of pushing tests left.