Flaky Tests and Pipeline Stability

Flaky tests are one of the biggest threats to a healthy CI/CD pipeline because they break trust; if a test sometimes fails for no good reason, people start ignoring all failures. QA engineers must learn how to detect, manage and eliminate flakiness.

Understanding and Managing Flaky Tests #

A flaky test is one that can pass and fail on the same code due to timing, environment issues, data clashes or hidden dependencies. In CI/CD, such tests cause intermittent red builds, reruns and frustration.

# Example: marking a flaky job for investigation (conceptual)
jobs:
  e2e_tests:
    runs-on: ubuntu-latest
    continue-on-error: true  # temporarily, while investigating
    steps:
      - name: Run E2E tests
        run: npm run test:e2e
      - name: Upload flaky test report
        if: failure()
        run: ./scripts/collect-flaky-tests.sh

Typical fixes include improving waits, stabilising test data, isolating environments and removing hidden dependencies on time or external systems.

Common Mistakes #

Mistake 1 — Accepting flaky tests as “just how UI tests are” #

This destroys confidence.

❌ Wrong: Ignoring flaky failures or always clicking “rerun” without investigation.

✅ Correct: Log, triage and resolve flaky tests as high-priority work.

Mistake 2 — Overusing retries to “fix” flakiness #

This hides problems.

❌ Wrong: Setting high retry counts so tests eventually pass.

✅ Correct: Use limited retries mainly for diagnostics, then fix root causes.