Flaky Tests: A Field Guide to Finding and Fixing the Real Cause

A flaky test is one that can pass and fail against the same code. That definition matters, because it rules out the most common excuse: “it’s just flaky”. If the code did not change and the result did, something in the test, the application or the environment is non-deterministic. Your job is to find which one.

Flakiness is expensive in a way that rarely shows up on a dashboard. Every unexplained red build teaches the team to click “re-run” instead of reading the failure. Once that habit sets in, real regressions slip through because they look like noise. Fixing flaky tests is less about the individual test and more about protecting the signal of the whole suite.

This guide covers the causes we see most often, a workflow for diagnosing them, and a quarantine policy that stops flaky tests from eroding trust while you fix them. Examples use Playwright, but the categories apply to any framework.

A taxonomy of causes

Most flaky tests fall into one of seven buckets. Naming the bucket is half the diagnosis.

1. Asynchronous waits and race conditions

The test checks something before the application has finished doing it. Classic symptoms: failures cluster on slower CI machines, and adding a sleep “fixes” it.

// Flaky: reads the text once, immediately after the click
await page.getByRole('button', { name: 'Save' }).click();
const status = await page.getByRole('status').textContent();
expect(status).toBe('Saved');

// Stable: web-first assertion retries until it passes or times out
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByRole('status')).toHaveText('Saved');

The first version uses a generic expect on a value captured once. The second uses a Playwright web-first assertion that polls the locator. Any time you see textContent(), isVisible() or count() fed into a plain expect, look closely.

A related race is waiting for a network response after triggering it:

// Race: the response may arrive before waitForResponse starts listening
await page.getByRole('button', { name: 'Search' }).click();
await page.waitForResponse('**/api/search*');

// Correct: start listening first, then trigger
const searchResponse = page.waitForResponse('**/api/search*');
await page.getByRole('button', { name: 'Search' }).click();
await searchResponse;

2. Test order and shared state

The test passes alone but fails when run after a specific other test, or vice versa. Causes include a shared logged-in user whose settings another test changed, a module-level variable that is mutated, a cart that is not emptied, or a feature flag toggled and never reset.

The fix is isolation: each test creates what it needs and does not rely on what ran before. Playwright gives every test a fresh browser context by default, which handles cookies and storage. It cannot reset your backend for you.

3. Data collisions in parallel runs

Two workers create a user with the same email, or both try to edit the same seeded record. The tests pass in serial and fail intermittently with more workers. Generate unique data per test and, where you need a stable identity per worker, use the worker index:

import { test } from '@playwright/test';
import { randomUUID } from 'node:crypto';

test('user can update profile', async ({ page }, testInfo) => {
  const email = `qa+${testInfo.parallelIndex}-${randomUUID()}@example.test`;
  // create the user through the API, then exercise the UI
});

testInfo.parallelIndex is stable between 0 and workers - 1, which makes it useful for pre-provisioned accounts (one per worker slot). testInfo.workerIndex is unique per worker process and increases when a worker restarts after a failure, so it is better suited to fully unique names.

4. Environment

The test depends on something outside its control: a shared staging environment that another team deploys to, a third-party sandbox that rate-limits, a slow database on Monday mornings, a different browser version on one CI runner, or resource starvation when too many workers share a small machine. These failures often appear in clusters across unrelated tests.

5. Time and timezones

Tests that assume “today”, a particular timezone or a particular day of the month will eventually fail. The usual suspects: assertions on formatted dates, logic around midnight or month-end, daylight saving transitions, and expiry checks that use the real clock.

Pin the browser’s timezone and locale in config, and control the clock where the behaviour depends on it:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    timezoneId: 'Europe/London',
    locale: 'en-GB',
  },
});

test('shows the trial expiry banner on the last day', async ({ page }) => {
  await page.clock.setFixedTime(new Date('2026-03-29T09:00:00Z'));
  await page.goto('/account');
  await expect(page.getByRole('alert')).toContainText('Your trial ends today');
});

Remember the server side too. If the backend computes “today” in UTC and the test expects the browser’s local date, a test run late in the evening can fail in some timezones.

6. Animations and transitions

An element is visible but still moving, or a modal is fading in and intercepts the click. Playwright’s actionability checks wait for elements to be stable, which handles most cases, but custom animations, skeleton loaders and toast stacks can still cause trouble, especially in screenshot tests. Options include reducedMotion: 'reduce' in use (if your app honours prefers-reduced-motion), disabling animations in a test build, and the default animations: 'disabled' behaviour of toHaveScreenshot().

7. Network and external services

Real calls to payment providers, email services, maps or analytics introduce latency and failure modes you do not control. For UI tests that are not about the integration itself, stub them:

await page.route('**/api/recommendations', (route) =>
  route.fulfill({ json: { items: [] } }),
);

Keep a small number of separate, clearly labelled tests that hit the real integration, so you still find out when the contract changes.

A diagnosis workflow

When a test is reported flaky, resist editing it straight away. Reproduce first, then explain, then fix.

Step 1: Reproduce it on demand

A failure you cannot reproduce is a failure you cannot verify you fixed. Run the test many times:

# Run one test 50 times with the current worker count
npx playwright test tests/checkout.spec.ts -g "applies coupon" --repeat-each=50

# Same, serially, to separate timing problems from parallelism problems
npx playwright test tests/checkout.spec.ts -g "applies coupon" --repeat-each=50 --workers=1

Compare the two. Failures only with multiple workers point at shared state or data collisions. Failures in both point at waits, time or the environment. If it never fails locally, try reproducing CI conditions: run headless, run in the same container image, and throttle CPU or add more workers to simulate a loaded runner.

Also run the whole file, or the whole project, in the same order CI uses. Order-dependent failures disappear when you run a test in isolation.

Step 2: Capture evidence

Turn on traces for failures so you have something to read:

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
});

on-first-retry records a trace only when a test is retried, which keeps the overhead low. While reproducing locally, use --trace on instead so every run is recorded.

Open a trace with npx playwright show-trace path/to/trace.zip, or from the HTML report. The trace viewer shows each action with DOM snapshots before and after, the network log, console messages and the source line. For flaky tests, the most useful questions are:

• What did the DOM look like at the moment of the failed assertion?
• Which network requests were still pending?
• Did a console error appear just before the failure?
• How long did each step take compared with a passing run?

Compare a passing trace with a failing one side by side. The difference is usually the answer.

Step 3: Name the cause, then fix the cause

Map what you saw to the taxonomy above, then fix at the right layer:

Evidence	Likely cause	Fix
Assertion ran before the UI updated	Async wait	Web-first assertion, or wait on the specific response
Fails only after another test	Shared state	Isolate data and setup per test
Fails only with multiple workers	Data collision	Unique data, per-worker accounts
Fails near midnight or on certain dates	Time	Pin timezone, control the clock
Click intercepted by another element	Animation or overlay	Wait for the overlay to disappear, reduce motion
Timeout on a third-party call	Network	Stub in UI tests, test the integration separately
Many unrelated tests fail together	Environment	Fix or isolate the environment, not the tests

The fix that is almost never right is a longer timeout or a waitForTimeout. A fixed sleep makes the test slower on every run and still fails when the machine is slower than your guess.

Step 4: Prove the fix

Run the same --repeat-each command that reproduced the failure. A fix that has not been verified against the reproduction is a guess. Then let it run in CI for a few days before declaring victory.

Retries are a signal, not a fix

Retries are useful. A test that fails then passes on retry is reported by Playwright as flaky rather than passed, and that classification is data you can track. The problem is when retries are treated as the solution:

• A retried test can hide a real intermittent bug in the product.
• Retries multiply suite time when many tests are unstable.
• “Passed on retry” becomes invisible unless someone looks for it.

A reasonable policy is to allow a small number of retries in CI, but report flaky results prominently and treat each one as a ticket. If your reporting surfaces only pass and fail, the retry has erased the evidence you need.

A quarantine policy that works

When a test is known to be flaky and the fix is not immediate, you have three bad options: leave it in and keep tolerating red builds, delete it and lose coverage, or retry it until it passes. Quarantine is the fourth option.

The rules

1. Quarantine is explicit. Mark the test with a tag or annotation, never by commenting it out, so it is searchable.
2. Every quarantined test has an owner and a ticket. The annotation links to it.
3. Quarantined tests still run, in a separate non-blocking job, so you keep collecting evidence and know when the fix works.
4. Quarantine has a time limit. If the test is not fixed by the deadline, the owning team decides to fix it, rewrite it at a lower test level or delete it. Leaving it indefinitely is not an option.
5. The quarantine list is visible. Review it in the same meeting where you review bugs.

In Playwright, tags and annotations make this easy:

test(
  'refund updates the ledger',
  {
    tag: '@quarantine',
    annotation: { type: 'issue', description: 'https://tracker.example.com/QA-482' },
  },
  async ({ page }) => {
    // ...
  },
);

# Blocking job: everything except quarantined tests
npx playwright test --grep-invert @quarantine

# Non-blocking job: only quarantined tests
npx playwright test --grep @quarantine

Avoid test.skip() for quarantine. A skipped test produces no evidence, so you cannot tell when it has been fixed.

Preventing the next one

Diagnosis gets easier with practice, but prevention is cheaper. Habits that reduce flakiness at the source:

• Use web-first assertions and locator actions instead of reading values and asserting on them once.
• Create test data through APIs in setup, uniquely per test, rather than relying on shared seeded records.
• Test at the lowest level that can catch the bug. Business rules covered by unit or API tests do not need slow, timing-sensitive UI tests.
• Pin timezone, locale and viewport in config so every machine runs the same conditions.
• Run new tests repeatedly before merging. A --repeat-each=20 run in the pull request catches many flaky tests before they reach the main branch.
• Lint for floating promises. A missing await in an async test is a reliable source of intermittent failures.

Key takeaways

• A flaky test has a cause. Find it before changing anything.
• Most causes fall into seven categories: async waits, shared state, parallel data collisions, environment, time, animations and network.
• Reproduce with --repeat-each, compare serial and parallel runs, and read traces from passing and failing runs side by side.
• Fix the cause at the right layer. Longer timeouts and sleeps are not fixes.
• Retries are useful for classification, but the flaky result must stay visible.
• Quarantine explicitly, with an owner, a ticket, a deadline and continued execution.