CI Optimisation and Flaky Test Management

Even with good isolation, some flakiness is inevitable in large UI suites. Managing it proactively in CI is part of advanced test engineering: you want to detect genuine issues quickly without being overwhelmed by noisy failures.

Retries, Tagging and Reporting #

Playwright supports per-test retries and powerful reporting options. Used carefully, retries can filter out transient issues while you investigate, and tagging lets you prioritise which tests run on which pipelines.

// playwright.config.ts (snippet for retries and reporter)
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [['html', { outputFolder: 'playwright-report' }], ['list']],
});

// Example of tagging a test
test('critical checkout flow @smoke', async ({ page }) => {
  // ...
});

// Run only smoke tests
// npx playwright test --grep "@smoke"

Integrating HTML or trace-based reports into CI artifacts makes it easier for developers and testers to inspect failures quickly.

Common Mistakes #

Mistake 1 — Treating flakiness as “just how UI tests are” #

This normalises poor quality.

❌ Wrong: Accepting red-to-green reruns as part of daily work.

✅ Correct: Use retries, tagging and reports to surface flaky tests and then fix them.

Mistake 2 — Overusing retries without visibility #

This hides instability.

❌ Wrong: Configuring retries but never reviewing which tests needed them.

✅ Correct: Monitor retry statistics and prioritise stabilising the worst offenders.