Log Filtering, Sinks and Production Configuration

Production logging requires careful configuration โ€” the wrong settings can flood storage with noise (every EF Core SQL statement, every HTTP framework log), hide real problems (log levels too high), or hurt performance (synchronous file writes on every request). Good production log configuration filters out noise, elevates important events, and keeps the logging pipeline non-blocking. Log hygiene โ€” knowing what to log, what to filter, and how to store logs efficiently โ€” is an operational skill as important as writing the logs in the first place.

Log Level Filtering for Production

// โ”€โ”€ appsettings.Production.json โ€” production log levels โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
// {
//   "Serilog": {
//     "MinimumLevel": {
//       "Default": "Warning",         // only warnings and above by default
//       "Override": {
//         "Microsoft":               "Warning",
//         "Microsoft.AspNetCore":    "Warning",
//         "Microsoft.Hosting.Lifetime": "Information",  // startup/shutdown visible
//         "Microsoft.EntityFrameworkCore": "Warning",   // suppress EF SQL logs
//         "BlogApp":                 "Information"      // your code at Information
//       }
//     }
//   }
// }

// โ”€โ”€ appsettings.Development.json โ€” verbose for local development โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
// {
//   "Serilog": {
//     "MinimumLevel": {
//       "Default": "Debug",
//       "Override": {
//         "Microsoft.EntityFrameworkCore.Database.Command": "Information",  // show SQL
//         "BlogApp": "Trace"           // maximum verbosity for your code
//       }
//     }
//   }
// }

// โ”€โ”€ Programmatic filtering โ€” suppress specific sources โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
builder.Host.UseSerilog((ctx, cfg) => cfg
    .ReadFrom.Configuration(ctx.Configuration)
    .Filter.ByExcluding(Matching.FromSource("Microsoft.AspNetCore.StaticFiles"))
    .Filter.ByExcluding(logEvent =>
        logEvent.Properties.ContainsKey("RequestPath") &&
        logEvent.Properties["RequestPath"].ToString().Contains("/health")));
Note: Health check endpoints are typically polled every 10โ€“30 seconds by load balancers and Kubernetes liveness probes. At Information level, this generates 2โ€“6 log entries per health check call โ€” 120โ€“360 entries per minute of pure noise. Filtering out health endpoint requests (as shown above) significantly reduces log volume without losing any meaningful observability. Similarly, filter /metrics, /favicon.ico, and static file requests that generate noise without value.
Tip: Use async sinks in production to prevent logging from blocking request threads. Serilog’s Serilog.Sinks.Async wraps any sink in an asynchronous buffer: .WriteTo.Async(a => a.File("logs/app.log")). Log entries are queued to a background channel and written by a dedicated thread, keeping request threads free. For high-throughput APIs, this is essential โ€” synchronous file writes (especially over network shares or cloud storage) add measurable latency to every request.
Warning: Rolling file sinks without retention limits will eventually fill the disk. Always configure retainedFileCountLimit (daily rolling) or retainedFileSizeLimit (size-based rolling). A production server with unbounded log files is a time bomb โ€” disk-full conditions cause complete service outages, often at the worst possible time. Set retention to match your compliance requirements (typically 30โ€“90 days for most applications) and monitor disk usage as a health metric.

Rolling File Sink with Retention

// โ”€โ”€ Serilog file sink with rolling, async, and retention โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
builder.Host.UseSerilog((ctx, cfg) => cfg
    .ReadFrom.Configuration(ctx.Configuration)
    .Enrich.FromLogContext()
    .Enrich.WithMachineName()
    .WriteTo.Async(a => a.Console(
        new CompactJsonFormatter()))   // JSON format for log aggregator ingestion
    .WriteTo.Async(a => a.File(
        formatter: new CompactJsonFormatter(),
        path: "logs/blogapp-.log",
        rollingInterval: RollingInterval.Day,
        retainedFileCountLimit: 30,    // keep 30 days of logs
        fileSizeLimitBytes: 100 * 1024 * 1024,  // 100MB per file
        rollOnFileSizeLimit: true)));

// โ”€โ”€ Seq sink for local development (structured log viewer) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
if (ctx.HostingEnvironment.IsDevelopment())
{
    cfg.WriteTo.Seq("http://localhost:5341");
    // dotnet tool install --global seqcli
    // seqcli run  (or docker run -p 5341:5341 datalust/seq)
}

Logging Health Check

// โ”€โ”€ Verify logging pipeline is operational at startup โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
public class LoggingHealthCheck(ILogger<LoggingHealthCheck> logger) : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken ct = default)
    {
        try
        {
            logger.LogInformation("Logging health check: pipeline operational.");
            return Task.FromResult(HealthCheckResult.Healthy("Logging pipeline operational."));
        }
        catch (Exception ex)
        {
            return Task.FromResult(
                HealthCheckResult.Unhealthy("Logging pipeline failure.", ex));
        }
    }
}

// builder.Services.AddHealthChecks().AddCheck<LoggingHealthCheck>("logging");

Common Mistakes

Mistake 1 โ€” No retention limit on rolling file sink (disk fills up)

โŒ Wrong โ€” unlimited logs accumulate and fill the disk, causing service outage:

.WriteTo.File("logs/app.log", rollingInterval: RollingInterval.Day)  // no retainedFileCountLimit!

โœ… Correct โ€” always set retainedFileCountLimit and fileSizeLimitBytes.

Mistake 2 โ€” Synchronous sinks in production (blocking request threads)

โŒ Wrong โ€” direct file/network sink writes block request threads during heavy load.

โœ… Correct โ€” wrap all production sinks in .WriteTo.Async(a => a.SinkName(...)).

🧠 Test Yourself

A Kubernetes liveness probe calls /health every 15 seconds. At Information level, each call generates 3 log entries. Over 24 hours, how many health-check log entries are created, and why should they be filtered?