How do you manage flaky tests in a CI pipeline?

A flaky test is one that can pass and fail on the same code. The cost is not the single failure; it is the erosion of trust. Once people learn that red often means nothing, they re-run, then ignore, then a real regression gets merged. So I treat flakiness as a reliability problem with a process, not as individual annoyances.

1. Detect with data, not anecdotes

• Enable a limited retry (one retry is usually enough) and record tests that failed then passed. Playwright marks these as “flaky” in its report; for JUnit-based tools, Maven Surefire’s rerunFailingTestsCount reports flakes separately.
• Store results per test across runs (JUnit XML into a database, a test analytics tool, or the CI’s test history) and compute a flake rate: failures on commits where the same test also passed.
• Periodically run the suite several times on an unchanged commit to surface flakes proactively.

2. Quarantine, with ownership

When a test is confirmed flaky:

• Move it out of the blocking set (tag it @quarantine and exclude it from the gating job), but keep running it in a non-blocking job so you still collect data.
• Create a ticket with an owner and a deadline. Quarantine without ownership becomes a graveyard.
• Cap the quarantine size; if it grows, the team stops feature work on tests until it shrinks.

# Gating job runs everything except quarantined tests
- run: npx playwright test --grep-invert @quarantine
# Separate non-blocking job keeps collecting data
- run: npx playwright test --grep @quarantine
  continue-on-error: true

3. Fix root causes

Common categories, roughly in the order I check them:

• Synchronisation: fixed sleeps, missing waits for network or animation, asserting before the UI settles. Use condition-based waits and web-first assertions.
• Test data and isolation: shared accounts, tests that depend on another test’s data, order dependence revealed by parallelism. Create data per test via API.
• Environment: shared environments under load, third-party sandboxes, slow cold starts. Stub what you don’t own; health-check before running.
• Test design: locators tied to layout, time zones and dates, randomness without a seed.
• The product: sometimes the flake is a real race condition in the application. Before blaming the test, check the app logs and traces for the failing run.

4. Retries: a tool, not a fix

Retries make pipelines green but hide information. Use them only if retried passes are reported and counted, never as an unlimited loop, and never as the long-term resolution of a ticket.

5. Make it visible

Publish the flake rate and the quarantine list in a dashboard reviewed at a regular team ritual. The trend matters more than any individual test.

Common mistakes

• Adding retries globally and declaring victory.
• Deleting flaky tests without understanding whether they found real bugs.
• Quarantining tests forever with no owner.

What interviewers look for

• A systematic process: detect, quarantine, fix, prevent.
• Concrete root-cause categories and fixes.
• Recognition that flaky tests can reveal real product concurrency bugs.
• An opinion on retries that balances pipeline throughput against signal.