Building a Logging Middleware with Stack Trace and Request ID in ASP.NET

Software teams across Australia, from fintechs in Sydney to resource platforms in Perth, depend on detailed observability to diagnose production issues. A well-instrumented ASP.NET application captures every layer of a request, from the incoming URL to the deepest database call, and ties them together with a single identifier. Logging middleware sits at the front of the request pipeline, decorating each call with context that survives across services and threads. Without this context, debugging a slow checkout or a broken integration resembles finding a needle in a haystack.

This guide walks through the design and implementation of a custom logging middleware that emits structured events, captures request correlation identifiers, and records stack traces when exceptions occur. The approach works with the standard ASP.NET Core pipeline and integrates with popular sinks such as Serilog, Application Insights, and Seq. Along the way, the article considers regulatory realities such as the Notifiable Data Breaches scheme under the Privacy Act 1988 and APRA CPS 234, both of which influence log design choices in production systems.

The role of structured logging in distributed ASP.NET systems

Structured logging replaces human-readable sentences with key-value pairs that machines can parse, index, and aggregate. In a typical ASP.NET Core application, the default logger writes plain text to the console or debug window, which is adequate for local development but limiting once the system scales. A request that travels through a web server, an API gateway, several microservices, and a database leaves a wake of disconnected log entries. Operators need a way to pull every event for a single user action into a coherent story.

Request identifiers solve this problem. A request ID, sometimes called a correlation ID, is a short token attached to the inbound HTTP request and copied into every log entry, metric, and outgoing call. When a customer in Adelaide reports that their insurance claim failed, the support team can search the logs for the identifier, and the entire journey appears in chronological sequence. Australian financial institutions regulated under APRA CPS 234 must demonstrate that they can reconstruct events leading to an incident, making correlation identifiers a practical requirement rather than a luxury.

Stack traces complement request IDs by showing the call path that produced an error. A captured stack trace reveals which method, file, and line number raised the exception, dramatically reducing the time engineers spend reproducing defects. When combined with structured properties such as UserId, TenantId, and Environment, the resulting log entry becomes a rich diagnostic record suitable for triage and post-incident review.

Designing the middleware pipeline

Middleware in ASP.NET Core forms a chain where each component can inspect, modify, short-circuit, or pass through the HTTP context. Logging middleware typically appears early in the chain so that subsequent components run inside its context. The standard pattern involves three responsibilities wrapped around the next delegate: pre-execution logging, exception capture, and post-execution logging.

Pre-execution logging records the start of the request, including the HTTP method, path, query string, and client IP. Exception capture wraps the next delegate in a try-catch block, records the exception with its stack trace, and rethrows or translates the error into an appropriate HTTP response. Post-execution logging records the outcome, including the status code and any computed metrics such as elapsed milliseconds.

Developers building SaaS products for the Australian market often need to honour state-level health data rules, particularly when serving customers in New South Wales and Victoria. Logging middleware provides a natural place to enrich requests with compliance metadata, such as a flag indicating whether Personally Identifiable Information appears in the payload. Adding this enrichment at the boundary ensures that downstream components inherit the context without needing to recompute it, and a clear correlation strategy pays dividends during incident response.

Capturing stack traces and request IDs

The implementation of the middleware uses three primary building blocks: a request ID generator, an exception capture routine, and a structured logger. The request ID generator reads the X-Request-ID header if the caller provided it, or generates a new GUID if the header is missing. Storing the identifier on the HttpContext.Items collection makes it available to downstream components without polluting the global log context.

The exception capture routine wraps the next delegate in a try-catch block. When an exception occurs, the middleware logs the event with the exception object, which includes the stack trace, and then either rethrows the exception or returns a sanitised error response. The choice depends on the application's error handling conventions and regulatory obligations. Healthcare applications subject to the My Health Records Act must avoid leaking patient identifiers into logs, so sanitisation becomes a critical step during exception capture.

Common logging libraries used in Australia-hosted ASP.NET applications differ in configuration surface, ecosystem, and performance characteristics:

After selecting a library, the middleware uses the configured logger to emit events at appropriate levels. Information-level logging suits routine request flow, Warning-level logging handles recoverable errors, and Error-level logging captures unhandled exceptions. Including the request ID and stack trace in every event ensures that operators can correlate events across services during root cause analysis.

Integrating with Application Insights and distributed traces

Application Insights is a popular choice for teams operating in Australian regions, including the Azure Australia East and Australia Southeast data centres. The Application Insights SDK automatically captures request telemetry, dependency calls, and exceptions, but custom properties such as a business-specific tenant identifier require manual enrichment. The logging middleware reads the tenant identifier from the route or claims, attaches it to the current Activity, and includes it in every log entry.

Distributed tracing builds on the Activity class, which represents a unit of work in the OpenTelemetry ecosystem. ASP.NET Core automatically creates an Activity for each inbound request and propagates the trace context through HTTP headers. The middleware enriches the current Activity by setting tags such as request.path, request.method, and custom.application. These tags appear in the Application Insights transaction view and provide rich context for debugging multi-step operations.

For teams seeking further reading on related performance topics, the optimizing-linq-queries-for-large-datasets-in-c article explores techniques for reducing memory pressure when querying big data sets. Efficient queries reduce log volume because fewer exceptions occur, which keeps the logging pipeline responsive under load. The connection between query optimisation and observability is often direct in high-traffic systems.

Performance considerations and production best practices

Logging introduces overhead, and high-throughput applications must balance observability against latency. Asynchronous sinks buffer log events in memory and flush them to disk or the network in the background, reducing the time spent on I/O during the request. Serilog's async wrapper and NLog's async targets both provide this capability with minimal configuration, making them suitable for latency-critical services hosted in Australian regions.

Sampling provides another lever for controlling log volume. Instead of recording every event in detail, the middleware can log a percentage of requests in full and a summary of the remainder. Application Insights offers adaptive sampling that automatically adjusts the volume based on the application's traffic, ensuring that telemetry stays within budget while preserving diagnostic value for unusual events such as failed payments.

Security considerations deserve attention as well. Logs frequently contain sensitive data, including authentication tokens, personal identifiers, and payment details. The middleware should sanitise sensitive fields before logging, and access to log storage must follow the principle of least privilege. Australian government systems subject to the Protective Security Policy Framework require documented controls over log access, making sanitisation a compliance matter rather than a stylistic choice. Production deployments spanning multiple regions for disaster recovery rely on centralised log aggregation using tools such as Seq, Elasticsearch, or Azure Log Analytics, with retention policies aligned to the seven-year obligation for certain financial records under the Corporations Act 2001.