Test Automation Quality — Preventing Flaky Tests and Maintaining Test Suites

Flaky tests — tests that sometimes pass and sometimes fail without code changes — are the biggest threat to CI reliability. When CI is unreliable, developers stop trusting it, start ignoring failures, and eventually disable it. The BlogApp’s most common flakiness sources are time-dependent assertions (using DateTime.Now in tests), shared database state between parallel tests, and Cypress tests with race conditions. Fixing flakiness requires addressing root causes, not just retrying.

Fixing Flaky Tests #

// ── Problem: DateTime.Now creates flaky time-dependent tests ───────────────
// ❌ FLAKY: depends on actual system time
[Fact]
public void CreatePost_SetsCreatedAtToNow()
{
    var post = _sut.Create(/* ... */);
    // This will fail if the clock ticks between the service call and assertion:
    post.CreatedAt.Should().Be(DateTime.UtcNow);  // exact match — almost always fails!
}

// ✅ FIX 1: Use BeCloseTo with tolerance
post.CreatedAt.Should().BeCloseTo(DateTime.UtcNow, TimeSpan.FromSeconds(5));

// ✅ FIX 2 (better): Inject IClock — full control in tests
public interface IClock
{
    DateTime UtcNow { get; }
}

public class SystemClock : IClock
{
    public DateTime UtcNow => DateTime.UtcNow;
}

public class FrozenClock : IClock
{
    private readonly DateTime _frozenTime;
    public FrozenClock(DateTime frozenTime) => _frozenTime = frozenTime;
    public DateTime UtcNow => _frozenTime;
}

// In tests — deterministic time:
var frozenTime = new DateTime(2024, 7, 15, 10, 0, 0, DateTimeKind.Utc);
var clock      = new FrozenClock(frozenTime);
var sut        = new PostsService(_repo.Object, _user.Object, clock);

var post = await sut.CreateAsync(request, default);
post.CreatedAt.Should().Be(frozenTime);  // exact match — always deterministic

// ── Problem: Parallel tests sharing database state ─────────────────────────
// ❌ FLAKY: two test classes both insert a post with slug "test-post"
// ✅ FIX: use unique slugs per test (already covered in Chapter 80)
// ✅ FIX: use [Collection("Sequential")] to disable parallelism for a test class

// ── Problem: Cypress race condition on SignalR connection ──────────────────
// ❌ FLAKY: assertion runs before SignalR connects
// cy.visit('/posts/test-post');
// cy.getByCy('viewer-count').should('contain', '1');  // may fail if WS not connected

// ✅ FIX: wait for WebSocket connection to establish
// cy.visit('/posts/test-post');
// cy.getByCy('connection-status').should('have.class', 'connected');
// cy.getByCy('viewer-count').should('contain', '1');

// ── Test quarantine — marking known-flaky tests ───────────────────────────
// Install: dotnet add package Xunit.SkippableFact
[SkippableFact]
public async Task FlakySignalRTest()
{
    Skip.If(Environment.GetEnvironmentVariable("CI") == "true",
        "Skipped in CI — known flaky, tracked in issue #123");
    // ... test body
}

// ── Detecting hung tests ───────────────────────────────────────────────────
// Run tests with a blame hang timeout to catch deadlocks:
// dotnet test --blame-hang-timeout 30s --blame-hang-dump-type mini
// Generates a mini dump when a test hangs for 30+ seconds

Common Mistakes #

Mistake 1 — Using DateTime.Now directly in production code (non-deterministic tests) #

❌ Wrong — post.CreatedAt = DateTime.UtcNow; test assertions on timestamps fail intermittently due to clock ticks.

✅ Correct — inject IClock; use FrozenClock in tests; exact timestamp assertions always pass.

Mistake 2 — Ignoring flaky tests instead of fixing or quarantining them #

❌ Wrong — “just re-run it” becomes the team norm; CI is unreliable; green CI means nothing.

✅ Correct — fix, quarantine with tracking issue, or remove; never ignore; green CI must mean “code is correct.”