A CI test fails once every twenty runs but passes locally. How would you investigate it?

Diagnose and fix a flaky test

Treats flakiness as an observable defect in timing, state, order, infrastructure, or product behavior.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I start with the failing test's logs, stack trace, seed, timing, environment, and nearby failures. Then I repeat it under the same configuration and vary order or parallelism to narrow the cause. Common sources are shared data, real time, unawaited tasks, arbitrary sleeps, port collisions, external services, and assumptions about execution order.

Full answer below

Practice focus

Reproduction, failure artifacts, timing, shared state, order dependence, environment drift, and quarantine policy.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Flaky tests erase trust in the suite and can hide real product races. Re-running until green is not a diagnosis.
I start with the failing test's logs, stack trace, seed, timing, environment, and nearby failures. Then I repeat it under the same configuration and vary order or parallelism to narrow the cause. Common sources are shared data, real time, unawaited tasks, arbitrary sleeps, port collisions, external services, and assumptions about execution order.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I classify whether the nondeterminism is in the test, infrastructure, or product. Running the test many times with controlled seeds can reveal a pattern, but I preserve every failed artifact rather than looking only at the final pass. I replace sleeps with observable synchronization, isolate mutable resources, and make random cases reproducible. A retry can measure frequency temporarily; it must not turn the pipeline green and discard the first failure.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Re-running until the build is green.
Adding a longer sleep.
Deleting the test without replacing coverage.
Assuming CI is simply slow.

A CI test fails once every twenty runs but passes locally. How would you investigate it?

Practice focus

Answer path

Why this question matters

Review signals

Follow-up prompts

Continue the section

Tech stacks

Skill areas

Suggest an improvement

Found this answer useful?