Debugging scenarioMedium~10 min

Interview question

A CI test fails once every twenty runs but passes locally. How would you investigate it?

Diagnose and fix a flaky test

Treats flakiness as an observable defect in timing, state, order, infrastructure, or product behavior.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I start with the failing test's logs, stack trace, seed, timing, environment, and nearby failures. Then I repeat it under the same configuration and vary order or parallelism to narrow the cause. Common sources are shared data, real time, unawaited tasks, arbitrary sleeps, port collisions, external services, and assumptions about execution order.

Full answer below

Practice focus

Reproduction, failure artifacts, timing, shared state, order dependence, environment drift, and quarantine policy.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Flaky tests erase trust in the suite and can hide real product races. Re-running until green is not a diagnosis.
I start with the failing test's logs, stack trace, seed, timing, environment, and nearby failures. Then I repeat it under the same configuration and vary order or parallelism to narrow the cause. Common sources are shared data, real time, unawaited tasks, arbitrary sleeps, port collisions, external services, and assumptions about execution order.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I classify whether the nondeterminism is in the test, infrastructure, or product. Running the test many times with controlled seeds can reveal a pattern, but I preserve every failed artifact rather than looking only at the final pass. I replace sleeps with observable synchronization, isolate mutable resources, and make random cases reproducible. A retry can measure frequency temporarily; it must not turn the pipeline green and discard the first failure.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
If the test blocks delivery repeatedly, I may quarantine it with an owner, tracked issue, expiry, and visible lost coverage. I also check whether it exposed a real race the product can hit, because labeling it test flakiness too early is dangerous. At suite level I track flaky rate, duration, and failure clusters. The repair is complete when the original failure mechanism is understood and the test repeatedly passes under stress for the right reason.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Re-running until the build is green.
Adding a longer sleep.
Deleting the test without replacing coverage.
Assuming CI is simply slow.

Red flagsCandidateSigns the approach is fragileSignals that an interviewer may treat as risky.Open details

Details

The first failing artifact is overwritten by a retry.
Quarantined tests have no owner or deadline.
A possible product race is dismissed without evidence.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

When is quarantine acceptable?
How would you detect order dependence?
What artifacts would you collect?

Continue practice

Move to the linked follow-up, next path step, prerequisite, or deeper variant.

1 linked

Go deeper

Practice the next layer of the same subject.

Failed CI Pipeline

Debugging scenarioMedium~10 min

Interview question

A CI test fails once every twenty runs but passes locally. How would you investigate it?

Diagnose and fix a flaky test

Treats flakiness as an observable defect in timing, state, order, infrastructure, or product behavior.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

Full answer below

Practice focus

Reproduction, failure artifacts, timing, shared state, order dependence, environment drift, and quarantine policy.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Flaky tests erase trust in the suite and can hide real product races. Re-running until green is not a diagnosis.
I start with the failing test's logs, stack trace, seed, timing, environment, and nearby failures. Then I repeat it under the same configuration and vary order or parallelism to narrow the cause. Common sources are shared data, real time, unawaited tasks, arbitrary sleeps, port collisions, external services, and assumptions about execution order.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I classify whether the nondeterminism is in the test, infrastructure, or product. Running the test many times with controlled seeds can reveal a pattern, but I preserve every failed artifact rather than looking only at the final pass. I replace sleeps with observable synchronization, isolate mutable resources, and make random cases reproducible. A retry can measure frequency temporarily; it must not turn the pipeline green and discard the first failure.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
If the test blocks delivery repeatedly, I may quarantine it with an owner, tracked issue, expiry, and visible lost coverage. I also check whether it exposed a real race the product can hit, because labeling it test flakiness too early is dangerous. At suite level I track flaky rate, duration, and failure clusters. The repair is complete when the original failure mechanism is understood and the test repeatedly passes under stress for the right reason.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Re-running until the build is green.
Adding a longer sleep.
Deleting the test without replacing coverage.
Assuming CI is simply slow.

Red flagsCandidateSigns the approach is fragileSignals that an interviewer may treat as risky.Open details

Details

The first failing artifact is overwritten by a retry.
Quarantined tests have no owner or deadline.
A possible product race is dismissed without evidence.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

When is quarantine acceptable?
How would you detect order dependence?
What artifacts would you collect?

Continue practice

Move to the linked follow-up, next path step, prerequisite, or deeper variant.

1 linked

Go deeper

Practice the next layer of the same subject.

Failed CI Pipeline

A CI test fails once every twenty runs but passes locally. How would you investigate it?

Practice focus

Answer path

Why this question matters

Review signals

Follow-up prompts

Continue practice

Go deeper

A CI test fails once every twenty runs but passes locally. How would you investigate it?

Practice focus

Answer path

Why this question matters

Review signals

Follow-up prompts

Continue practice

Go deeper