Building Resilient .NET Microservices with Polly

Microservices give Australian organisations flexibility to release features independently, scale selected workloads and choose suitable cloud services. They also introduce a network between operations that once ran inside a single process. A payment API can become slow, a stock service can return a temporary error, or an Azure dependency can be unavailable in one region while the rest of the application continues operating.

Polly is a resilience and transient-fault-handling library for .NET. It provides policies that control how an application responds to failures, including retries, timeouts, circuit breakers, fallbacks and isolation. Used carefully, these policies help services remain responsive without hiding serious faults or creating a wave of repeated requests across the system.

Why Resilience Matters in Microservices

A method call inside a monolith usually has predictable process-level behaviour. A microservice call crosses a network, passes through load balancers and authentication layers, and may depend on a database or another external provider. Each boundary adds possible failure modes: connection resets, DNS problems, HTTP 429 responses, slow queries and brief service interruptions.

A useful resilience strategy distinguishes between transient and permanent faults. A short-lived timeout may be worth retrying, while a 400 Bad Request generally requires a code or data correction. Retrying every failure can increase pressure on an already struggling dependency, especially when dozens of application instances make the same decision at the same time.

Australian systems often need to account for geographic distance. An application serving customers in Perth may communicate with resources hosted in the Sydney Azure region, while users in Melbourne or Brisbane experience different network paths and latency. Polly cannot fix poor architecture or an undersized service, but it can make expected network uncertainty manageable.

Retry Policies Without Creating More Failure

A retry policy repeats an operation after a failure, usually with a delay between attempts. It works well for temporary HTTP 408, 429, 502, 503 and 504 responses, provided the request is safe to repeat. Exponential backoff increases the wait after each attempt, reducing the chance that all clients will retry simultaneously.

Jitter adds a small random variation to each delay. Without jitter, thousands of instances can follow the same retry schedule and send another burst at exactly the same moment. This “thundering herd” effect is particularly harmful during a regional incident or after a popular Australian retail website releases a limited product drop.

Retries must respect operation semantics. A GET request is normally easier to repeat than a POST that creates an order, unless the receiving API supports idempotency keys. A practical policy should set a maximum number of attempts, inspect the response status, record the reason for each retry and avoid retrying authentication failures or validation errors.

In Polly, a retry can be composed with a timeout and a circuit breaker. The order matters: a timeout limits how long an individual attempt can run, while the retry policy determines whether another attempt is allowed. The total request budget should include all attempts, so a caller does not wait far longer than the user interface or upstream gateway permits.

Timeouts, Circuit Breakers and Fallbacks

A timeout is one of the most important resilience controls because an unbounded request can consume a thread, socket or connection-pool entry. Every outbound call should have a deliberate limit based on its purpose. A search request may need a shorter budget than a batch operation, while a health check should usually fail quickly.

A circuit breaker stops calls to a dependency after a defined pattern of failures. In the open state, calls fail immediately for a cooling-off period. The breaker then permits limited test calls in a half-open state. If the dependency has recovered, normal traffic resumes; if not, the circuit remains open.

Fallbacks provide a controlled alternative when the preferred operation fails. A product catalogue might return recently cached data, and a notification service might place an event on a durable queue. A fallback should be honest about freshness and availability. Returning stale stock information as if it were current can be more damaging than displaying a temporary error.

Bulkhead isolation limits how many concurrent operations can consume a particular resource. Separate limits for payment, reporting and catalogue calls can prevent a slow reporting provider from exhausting all available worker capacity. This is useful for SaaS platforms serving customers across Australian time zones, where a large batch workload can otherwise compete with daytime transactions.

Choosing Policies for Different Calls

Resilience policies should be designed around the dependency and business operation rather than applied as a universal template. A read-only catalogue request may use a short timeout, a few jittered retries and a circuit breaker. A payment command may use a single carefully bounded attempt, an idempotency key and a reconciliation process rather than automatic repetition.

The following combinations provide a practical starting point. Values must be tested against real latency, provider limits and service-level objectives rather than copied unchanged into production.

In a .NET service, policies can be attached to an HttpClient through IHttpClientFactory. Named or typed clients make it possible to give an inventory provider a different policy from an identity provider. This arrangement also manages handler lifetimes more safely than creating a new HttpClient for every request.

Polly’s classic policy model commonly uses constructs such as WaitAndRetryAsync, CircuitBreakerAsync, TimeoutAsync and FallbackAsync. Newer Polly releases use a resilience pipeline approach, while the underlying engineering principles remain the same. Teams maintaining an older .NET application should check package versions and migration guidance before mixing configuration styles.

Composing Policies in a .NET Service

A policy pipeline should have a clear responsibility at each stage. A typical outbound call might apply a timeout around the attempt, retry selected transient failures with backoff, and use a circuit breaker to stop traffic when repeated attempts fail. The exact ordering should be documented because it changes how many calls are made and how failures are measured.

For example, a retry policy might allow three attempts with delays of roughly 200 milliseconds, 500 milliseconds and one second, plus jitter. A circuit breaker could open after a defined number of failures within a window and remain open for 30 seconds. These figures are illustrative; a high-volume API may need stricter limits, while a batch process could tolerate a longer budget.

Policy registration belongs close to dependency configuration, while business-specific fallback behaviour belongs in the application layer. A service should avoid catching every exception and returning a generic success response. That approach makes monitoring misleading and may cause downstream systems to believe an operation completed when it did not.

Configuration should be externalised so operators can tune limits without rebuilding the service. Australian organisations may run workloads in Azure Australia East or Australia Southeast and maintain separate production and disaster-recovery environments. Timeouts and failure thresholds should be tested in each environment because network distance, provider quotas and traffic patterns differ.

Observability and Operational Safety

A resilience policy is effective only when engineers can see what it is doing. Log the dependency name, operation, attempt number, delay, status code and correlation identifier. Metrics should distinguish original failures from retry attempts, and should record circuit-breaker state changes, timeout counts and fallback usage.

Distributed tracing helps connect a customer request with calls across several services. A trace showing one incoming request followed by four retries reveals a very different problem from a trace containing one failed attempt. OpenTelemetry can provide a consistent approach across .NET services, while dashboards can show dependency latency and error rates by region.

Alerting should focus on sustained customer impact rather than every individual retry. A small number of retries may be normal during network variation. A rising retry rate combined with increased latency, open circuits or fallback responses indicates a dependency problem that needs investigation.

Privacy and data handling also belong in operational design. Under Australia’s Privacy Act 1988 and the Australian Privacy Principles, logs should avoid unnecessary personal information, payment details and authentication tokens. A trace identifier is generally safer than copying a customer’s email address into every log entry. Teams should also understand where telemetry is stored and whether cross-border transfer arrangements meet organisational and regulatory requirements.

Testing Failure Behaviour Before Production

Resilience code should be tested as deliberately as ordinary business logic. Unit tests can verify that a 503 is retried, a 400 is not retried, a breaker opens after the configured threshold and a fallback returns the expected degraded result. Tests should also confirm that cancellation tokens and request deadlines are respected.

Integration tests can use a local stub server to introduce delays, dropped connections, malformed responses and sequences of failures followed by recovery. Injecting a 2.5-second delay into an API call is a simple way to confirm that the timeout behaves as expected. Testing a 429 response verifies that the service respects rate limits rather than increasing traffic.

Load testing is essential because retry behaviour multiplies traffic. If 100 requests fail and each is attempted three times, the dependency may receive up to 300 calls. A chaos test can combine high latency with partial errors to reveal whether connection pools, queues and thread resources remain stable.

Teams should also test graceful recovery. When a circuit closes, the first successful requests should not cause a sudden overload. When a fallback serves cached data, the cache should have a defined age and invalidation policy. These checks matter for services used during Australian peak periods such as end-of-financial-year processing, major sales events and morning commuter traffic.

Building a Sustainable Resilience Strategy

Polly should support a broader reliability practice that includes sensible service boundaries, health checks, capacity planning and clear ownership of dependencies. It cannot compensate for a service that has no database indexes, a gateway with an unsuitable timeout or an API contract that does not define idempotency.

Every policy needs an operational owner and a reason for its settings. Document which errors are retryable, what the maximum end-to-end duration is, how a circuit opens and what users see during fallback. Review these decisions when a provider changes its quotas or when traffic moves between Sydney, Melbourne and other Australian locations.

Resilience also needs to align with business priorities. A banking workflow may prefer a visible failure and later reconciliation over duplicate processing. An online media service may accept slightly stale content to preserve availability. A government or healthcare platform may need additional audit controls, data residency checks and strict handling of sensitive information.

Used with discipline, Polly gives .NET teams a practical vocabulary for handling unreliable dependencies. Retries address brief faults, timeouts enforce boundaries, circuit breakers prevent cascading failure, fallbacks preserve selected user experiences and bulkheads contain resource pressure. The result is a service that fails in a controlled, observable way rather than allowing a small dependency problem to spread throughout the platform.