Table of contents
You find flaky tests by looking at history, not at a single red build: a test is flaky when it both passes and fails on the same code. Record every attempt of every test, including retries, then list the tests whose results mix passes and failures on the same commit. Google measured that about 84% of the pass-to-fail transitions in its CI involved a flaky test (Google Testing Blog), so this list is usually where most of your "broken build" noise comes from.
What counts as a flaky test?
A flaky test is a test that can pass and fail without any change to the code or to the test. Google's definition is "a test that exhibits both a passing and a failing result with the same code" (Google Testing Blog). The same post reports that about 1.5% of all its test runs returned a flaky result, and that almost 16% of its tests had some level of flakiness.
Two conditions make the definition useful in practice:
Same code. A test that started failing after a commit and keeps failing is a regression, not a flaky test. The commit SHA (or build) is what lets you tell them apart.
Mixed outcomes. One failure proves nothing. You need at least one pass and at least one failure on that same code, either across runs or across retries of the same run.
The problem is common outside Google too. A February 2026 study of GitHub Actions found that 3.2% of builds were rerun, that 67.73% of those reruns were flaky (they failed, then passed with no change), and that flaky tests caused 64.99% of the flaky failures, ahead of network problems at 15.8% (Ge and Zhang, arXiv 2602.02307).
How do I find flaky tests in my suite?
Find flaky tests with five steps: keep every attempt, tie it to a commit, query for mixed outcomes, rank, and confirm.
Keep every attempt. Turn on a small number of retries in CI (one or two) and make sure your reporting keeps each attempt as its own result. If only the final outcome is stored, a test that failed and then passed on retry looks green, and the evidence is gone.
Tie each result to the code it ran against. Store the branch, commit SHA and CI build URL with every run. Without them you cannot tell "failed on the same code" from "failed after a change".
Query for mixed outcomes. List the tests that have at least one failure and at least one pass in a recent window, such as the last 30 days, on your main branch. Two signals count: a failure followed by a pass on retry in the same run, and a pass and a failure in different runs of the same commit.
Rank by cost. Sort the list by how often each test failed in the window. A test that flaked 12 times in a month costs more reruns and more trust than one that flaked once, so it goes to the top.
Confirm before you label. Rerun each suspect many times in isolation (next section). A test that fails only in the full suite but never alone points to shared state or test order, which is a different fix from a timing problem.
Run the query on a schedule, for example weekly, rather than once. Google notes that its rate of new flaky tests was "about the same as the fix rate" (Google Testing Blog), so a one-off cleanup does not stay clean.
How do I confirm that a test is really flaky?
Confirm a suspect by running it many times on unchanged code and counting failures. Most runners can do this directly:
# Playwright: run each matching test 50 times
npx playwright test tests/checkout.spec.ts --repeat-each=50
# Playwright: fail the CI job when any test is flagged as flaky
npx playwright test --retries=2 --fail-on-flaky-testsThe Playwright CLI documents --repeat-each <N> as "Run each test N times" and --fail-on-flaky-tests as "Fail if any test is flagged as flaky" (Playwright CLI).
Read the outcome this way:
Result of 50 isolated runs | What it suggests | Next step |
|---|---|---|
50 passes, but it fails in the full suite | Order dependency or shared state (data, cache, globals) | Run it after the tests that precede it in CI; reset state between tests |
Some failures in isolation | Timing, concurrency or an external dependency | Read the errors of failed attempts side by side; look for timeouts and waits |
50 failures | Not flaky: it is broken on this code | Treat it as a regression |
50 passes, and it passes in the suite too | Not reproducible here; often CI-only resources | Compare CI and local environments (CPU, network, data) |
Google lists the usual root causes as concurrency, reliance on non-deterministic or undefined behavior, flaky third-party code and infrastructure problems (Google Testing Blog). Compare the error messages of failed attempts: a timeout on the same locator every time is a very different lead from a different assertion each time.
Should I just turn on automatic retries?
Turn on retries to collect evidence and keep pipelines moving, but never treat a retry that passed as a pass you can ignore. Each framework has its own switch:
Framework | How to retry | What it records |
|---|---|---|
Playwright |
| Labels each test "passed", "flaky" (failed first, passed on retry) or "failed" (Playwright) |
Jest |
| Only available with the default |
pytest |
| Plugin version 16.7, released September 2026 (PyPI) |
The risk is that retries hide real bugs. Google describes marking a test as flaky so that it reports a failure only after 3 failures in a row, and calls the result "hardly a perfect solution": it reduces false positives but encourages developers to ignore flakiness, and a real break in a 15-minute flaky-marked test takes 45 minutes to surface (Google Testing Blog).
A practical rule: allow one or two retries, keep every attempt, and review the "failed then passed" list as part of the weekly query from the previous section.
Should I quarantine, fix or delete a flaky test?
Fix the flaky tests that guard important behavior, quarantine the ones you cannot fix this week, and delete the ones nobody would miss. Quarantine means the test still runs but no longer blocks the pipeline; Google built a tool that quarantines tests whose flakiness is too high and files a bug for the owner, while warning that this "could easily mask a real race condition" (Google Testing Blog).
Situation | Decision |
|---|---|
Covers a critical path (checkout, login, billing) and the cause is known | Fix now |
Flakes often, cause unknown, blocks other teams | Quarantine, with an owner and a review date |
Duplicates coverage another stable test already gives | Delete |
Quarantined for weeks with no pass at all | Treat it as broken: fix or delete |
Whatever you choose, label it in one place. A tag or a flag on the test case lets you count quarantined tests, see how long they have been there, and notice when one of them stops passing completely.
Where Probara fits
Probara keeps the attempt history that the method above depends on. Its reporters for Playwright, Jest, Vitest and Cypress send every attempt as its own result, with its own status, duration and errors: a test retried twice sends three results. The run shows the last outcome, and the earlier attempts stay in the case's history (Playwright results and retries). Each run also records the branch, commit and CI build URL.
On Pro and Enterprise plans, Queries turn step 3 into a saved query. This one lists automated cases with both a failure and a pass in the last 30 days:
FROM cases WHERE automated = true
AND exists(results where status = failed AND executed > now(-30d))
AND exists(results where status = passed AND executed > now(-30d))
ORDER BY titleEach test case has an is flaky field, which you can count on a dashboard with the Flaky test cases widget (Dashboards) and query with isFlaky. The PQL reference includes a query for flagged cases with no pass in the last 30 days, the "quarantined for weeks" row of the decision table. For a broader view of what a test management tool records, see What is a test management tool?.
FAQ
What is the difference between a flaky test and a failing test?
A failing test fails every time on the same code; a flaky test passes on some runs and fails on others with no change. Rerunning on the same commit tells them apart.
How many reruns does it take to prove a test is flaky?
One pass and one failure on the same code is enough to call it flaky. To estimate how often it flakes, run it in isolation many times, for example with Playwright's --repeat-each=50, and count the failures.
Do retries make flaky tests worse?
Retries do not cause flakiness, but they can hide it and the real bugs behind it. Keep every attempt, and review the tests that failed and then passed on retry instead of counting them as green.
What causes most flaky tests?
Google lists concurrency, non-deterministic or undefined behavior, flaky third-party code and infrastructure problems. In GitHub Actions, a 2026 study attributed 64.99% of flaky failures to flaky tests and 15.8% to network issues.
Should flaky tests block a release?
A flaky test on a critical path should block until someone confirms the failure is not real. A quarantined test should not block, but it needs an owner and a date to come back.
