How can I debug flaky tests and intermittent CI failures?

The fastest way to debug a flaky test is to prove that the same test both passes and fails on the same commit, then inspect what changed between those attempts.

Depot gives you that loop across Depot CI and GitHub Actions runners managed by Depot:

1. Upload JUnit XML from every test job, including failed jobs.
2. Find tests that passed and failed on the same commit in Possibly flaky tests.
3. Open the failed attempt and inspect its stack trace, stdout, stderr, logs, and CPU and memory metrics.
4. Rerun only the failed job against the same commit.
5. If the failure depends on machine state, rerun the job and pause immediately before the test step with SSH.
6. Fix the source of nondeterminism, then repeat the same test enough times to verify the fix.

Don't start by adding retries to the whole test suite. A green retry tells you that a test is unstable. It doesn't make the test reliable.

Report test results on every run

Logs tell you that a command failed. JUnit results tell you which test failed, how often it fails, how long it takes, and whether it passed on another attempt for the same commit.

Add depot/test-report-action after the test step and run it even when the tests fail:

- name: Run tests
  run: |
    mkdir -p test-results
    pnpm exec vitest run \
      --reporter=default \
      --reporter=junit \
      --outputFile.junit=test-results/junit.xml

- name: Report tests
  uses: depot/test-report-action@v1
  if: ${{ !cancelled() }}
  with:
    path: test-results/

Depot CI authenticates the action automatically. On Depot GitHub Actions runners, grant the job id-token: write permission so the action can authenticate through GitHub OIDC.

The Test Results Analytics page combines results from both products. Filter by repository, branch, suite, file, class, test name, and timeframe to narrow the failure pattern.

The strongest flaky-test signal is a test that has both a passing and failing result for the same Git commit. The code didn't change. Something else did.

Depot marks those tests in Possibly flaky tests and adds a Passed on rerun badge to individual results. The Most frequent test failures list catches tests that fail repeatedly even when they don't yet have a matching same-commit rerun.

You can pull the same results into a terminal:

depot tests <job-id> --status failed

For a GitHub Actions job running on Depot, add --gha:

depot tests <github-job-id> --gha --status failed

Separate test failures from CI failures

Intermittent failures usually fall into three buckets:

• Test nondeterminism: The assertion fails because of shared state, test order, time, randomness, concurrency, or an external dependency.
• Resource pressure: The process is killed, stalls, or times out because it runs out of CPU, memory, disk, or network capacity.
• Transient infrastructure: The test never really starts because a package registry, container registry, service, or network request briefly fails.

Treating every bucket as "flaky tests" makes the fix harder to find.

Open the failed job in Depot and compare the attempt with a passing attempt. Step timing and CPU and memory graphs can show a test process saturating a runner or getting killed. Searchable logs can show whether the same error appears across repositories, workflow runs, or runner types.

For Depot CI, start with the bounded failure diagnosis and then read the exact attempt logs:

depot ci diagnose --attempt <attempt-id>
depot ci logs <attempt-id>

depot ci diagnose extracts the failed step, relevant error lines, likely cause, and suggested next move. Verify the suggestion against the full logs. If the test framework emitted a seed, shard, worker ID, port, temporary path, or fixture name, keep it. That's often the difference between reproducing the failure and chasing a different one.

Reproduce the same failure

First rerun the failed job against the same commit. Depot CI can create a new attempt without rerunning jobs that already passed:

depot ci retry <run-id> --job <job-id>

Use depot ci retry <run-id> --failed when several failed jobs need another attempt. Previous attempts remain available, so you can compare the failure instead of replacing its evidence.

Once you know the failing test, run the smallest useful target: one test, one file, one suite, or one shard. Preserve the same random seed and concurrency. If the failure only appears in the full suite, suspect order dependence, shared state, leaked processes, reused ports, or contention between workers.

With Depot CI, you can also run a workflow against uncommitted local changes without pushing another commit:

depot ci run --workflow .depot/workflows/ci.yml --job test

Each run uploads a fresh patch from your working tree. Fix, rerun, repeat.

Inspect the runner with SSH

Logs are a recording. Sometimes the useful evidence only exists inside the runner.

Rerun one job and use --ssh-after-step to pause after the last successful step:

depot ci run \
  --workflow .depot/workflows/ci.yml \
  --job test \
  --ssh-after-step 3

Depot runs the first three steps, pauses the job, and connects your terminal to the sandbox. The checked-out code, dependencies, environment variables, files, and processes are still there. Run the exact failing test repeatedly. Inspect open ports, temporary files, process state, available memory, clock and timezone settings, and any service the test expects to be ready.

Use direct SSH for a job that's already running:

depot ci ssh <run-id> --job <job>

See Debug a failing job with SSH for the difference between direct SSH and a step-aware tmate session.

Fix the source of nondeterminism

Common fixes are boring. Here are a few:

• Give each test or shard its own database, namespace, temporary directory, port, and object-storage prefix.
• Freeze or inject time instead of depending on the wall clock, timezone, or daylight-saving transitions.
• Seed randomness and print the seed so a failure can be replayed.
• Wait for observable service readiness instead of sleeping for an arbitrary number of seconds.
• Reset global state and close files, sockets, servers, workers, and database connections after every test.
• Remove test-order dependencies by making setup and cleanup explicit.
• Stub third-party services when the test is meant to verify your code rather than their availability.
• Increase runner size only when Depot's resource graphs show actual CPU or memory pressure.

Also inspect the Top 5 slowest tests list in Test Results Analytics. Depot ranks it by P95 duration, which exposes the slow tail rather than only the average. A test that occasionally takes 30 seconds in a suite where it normally takes two seconds may be waiting on contention, a race, a slow external service, or a missing timeout. That's often the same condition behind an intermittent failure.

Use retries only for known transient commands

Depot CI supports a native retry: key on run: steps. Use it for commands where a retry is the intended recovery, such as a package download or registry request that can fail because of a brief network interruption:

- name: Install dependencies
  run: pnpm install --frozen-lockfile
  retry:
    retries: 2
    backoff: exponential
    delay-seconds: 5
    max-delay-seconds: 30

Don't put this on the general test command. retry: runs again after any non-zero exit code, in the same sandbox, and filesystem side effects from the failed attempt remain. A passing second attempt can hide a product bug, an order dependency, or dirty state created by the first attempt.

Use a targeted manual rerun while investigating a test. Use automatic step retries only after you know the failure is transient infrastructure and the command is safe to repeat.

Split tests after the suite is reliable

Depot test splitting uses historical JUnit timing data to distribute tests across parallel jobs. It balances shards by past duration, so one slow shard is less likely to hold up the entire suite.

Splitting makes the feedback loop faster. It doesn't repair a flaky test. More concurrency can expose shared-state and ordering bugs that a serial suite happened to hide. Fix those failures first, then keep the balanced shards. Depot uses the latest reported durations to improve the split over time.

Evaluate a CI platform with the failure, not just the retry

A platform is useful for flaky tests when it preserves enough evidence to explain why the attempts differed. Evaluate it with an intermittent failure your team already knows, then check whether you can:

1. Find tests that passed and failed on the same commit.
2. Keep every attempt instead of replacing the first failure with the successful retry.
3. Compare test output, step timing, CPU, and memory between attempts.
4. Rerun one failed job without paying for every successful job again.
5. Open an SSH session at the step where the intermittent state exists.
6. Search for the same error across repositories and runner types.
7. Export the underlying JUnit result and logs instead of relying on a proprietary flake score.

Also measure the platform's own failure rate. Run a known-stable job enough times to separate test nondeterminism from runner startup failures, network errors, and service interruptions. Keep retries and infrastructure failures in the result. A provider that makes the average run faster but adds failures can lengthen the time to a completed green workflow.

Depot keeps individual attempts, same-commit pass/fail signals, JUnit output, resource metrics, targeted retries, and SSH in one workflow history. That lets a retry confirm the flake without deleting the evidence needed to fix it.