Flaky test

A flaky test gives a different result on the same commit. Run it ten times against unchanged code and you get some passes and some failures. The defining feature is that nothing in the product changed: the variation comes from the test, its data or its environment.

Why it matters

A suite with a few flaky tests teaches the team to re-run failures until they go green. Once that habit sets in, real regressions get re-run away too. Flakiness also slows pipelines, because retries multiply run time, and it erodes trust in automation faster than missing coverage does.

Common causes

• Timing. The test acts before the page or API is ready: a fixed sleep, a check that reads a value once instead of retrying, or an animation still running.
• Shared state. Two tests write the same user, cart or database row. They pass alone and fail in parallel or in a different order.
• Test-order dependence. A test relies on data that an earlier test created.
• Environment. Slow or shared test environments, third-party services, rate limits, time zones and clock changes.
• Non-deterministic data. Random inputs without a logged seed, or assertions on “now”.

How to deal with one

1. Reproduce it. Run the single test in a loop (Playwright: npx playwright test my.spec.ts --repeat-each=20) and capture a trace or video of a failing run.
2. Find the cause instead of adding a retry. Replace sleeps with condition-based waits, such as web-first assertions in Playwright or WebDriverWait in Selenium. Give each test its own data.
3. Quarantine it if you can’t fix it today. Move it out of the blocking gate and into a tracked, non-blocking job with an owner, so the main signal stays trustworthy.
4. Track the flake rate over time. Retries that pass hide the problem unless the report shows them.

Retries (retries: 2 in playwright.config.ts) are a reasonable safety net in CI, but a test that only passes on retry is still flaky. Playwright reports those as “flaky” so you can see them.