Implement resilient .NET clients with Polly.Contrib

Cloud APIs fail in ordinary ways: sockets time out, gateways return 503 responses, and a dependency slows just enough to exhaust a thread pool. In a .NET application, those faults become user-facing errors unless the call site decides how to recover. Polly turns that decision into reusable policies, while Polly.Contrib provides contributed extensions for common integration scenarios.

Resilience does not mean hiding every error. It means bounding retries, preventing one broken dependency from consuming all capacity, and returning a predictable response when recovery is not possible. The useful resilience policies are deliberately small: retry transient faults, time out slow work, open a circuit after sustained failure, and limit concurrency or request rates.

The examples use ASP.NET Core’s typed HttpClient and Polly 7-style APIs, making the subject directly relevant to the C# Corner Annual Conference 2020 audience. They are aimed at Australian teams operating services that call local banks, payment providers, government systems or overseas partners, where latency and failure patterns change with geography and time of day.

Put resilience at the outbound boundary

Put resilience at the outbound boundary, not in every controller. A single policy around a typed HttpClient gives all callers the same retry budget, timeout and circuit state. It also keeps recovery logic out of business rules, where a payment calculation might accidentally retry a non-idempotent write.

Each mechanism addresses a different problem. Retrying can recover a dropped connection, but unbounded retries create a retry storm and increase tail latency. A timeout releases scarce resources, while a circuit breaker stops traffic to a dependency that is already failing. Treat these as coordinated controls rather than interchangeable switches.

Prepare the packages and client boundary

Use the core Polly package and Polly.Contrib for the contributed extensions. ASP.NET Core also needs the HttpClient policy integration; current Microsoft.Extensions.Http.Polly supplies AddPolicyHandler. If Polly.Contrib does not bring it in transitively, install it directly and align all Polly dependencies on one major version.

dotnet add package Polly
dotnet add package Polly.Contrib
dotnet add package Microsoft.Extensions.Http.Polly

Create policies in a dedicated factory and inject the resulting IAsyncPolicy<HttpResponseMessage> into registration code. Keep HttpClient configuration separate from business logic. Also decide whether HttpClient owns the overall timeout; when Polly applies a timeout per attempt, set the client timeout to infinite so it cannot unexpectedly terminate a retry sequence.

Build retry behaviour that limits harm

Retry only faults that are plausibly transient. HttpPolicyExtensions.HandleTransientHttpError covers common network failures and selected HTTP responses such as 408, 500, 502, 503 and 504. Extend that set deliberately, and keep permanent 4xx responses out of the retry path.

private static IAsyncPolicy<HttpResponseMessage> BuildRetryPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .Or<Polly.Timeout.TimeoutRejectedException>()
        .WaitAndRetryAsync(
            retryCount: 3,
            sleepDurationProvider: attempt =>
            {
                var ceiling = Math.Min(4000, 250 * Math.Pow(2, attempt));
                var delay = 50 + Random.Shared.NextDouble() * (ceiling - 50);
                return TimeSpan.FromMilliseconds(delay);
            });
}

The delay function uses capped exponential backoff with jitter. Without jitter, clients that failed together can retry together and recreate the outage. Randomising the delay spreads attempts across a small window while the ceiling prevents an excessive wait.

Bound attempts and respect operation semantics. GET and HEAD are naturally repeatable; PUT and DELETE need idempotent behaviour; POST should retry only when the endpoint supplies an idempotency key and the same key is reused. This matters for Australian payment and banking integrations, including BPAY and PayID flows, where a lost response must not create a second payment.

Add timeouts and circuit breaking

A timeout is a latency budget, not a retry setting. Give each attempt enough time for a healthy dependency and less time than the caller’s remaining budget. A two-second upstream timeout inside a fifteen-second API request leaves room for controlled retries and a useful fallback response.

var timeout = Policy.TimeoutAsync<HttpResponseMessage>(
    TimeSpan.FromSeconds(2));

A circuit breaker sits outside the retry sequence. It counts consecutive failures, opens after the configured threshold, rejects calls quickly during the break, then permits a limited probe in the half-open state. Opening after sustained failure protects local resources and gives the dependency room to recover.

Choose the failure predicate carefully. Repeated 5xx responses and transport exceptions are strong candidates, while a 429 usually represents a quota that needs rate limiting. A breaker should not trip because one client sent a malformed request or because a business validation returned 400.

Compose and register the policy

Composition determines semantics. The desired path is circuit breaker, then retry, then per-attempt timeout: the breaker observes the final outcome of the whole recovery sequence, the retry policy gets a fresh timeout for each attempt, and the timeout protects every individual call.

private static IAsyncPolicy<HttpResponseMessage> BuildPolicy()
{
    var retry = BuildRetryPolicy();
    var timeout = Policy.TimeoutAsync<HttpResponseMessage>(
        TimeSpan.FromSeconds(2));

    var breaker = Policy
        .HandleResult<HttpResponseMessage>(
            response => (int)response.StatusCode >= 500)
        .Or<HttpRequestException>()
        .Or<Polly.Timeout.TimeoutRejectedException>()
        .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30));

    return Policy.WrapAsync(
        breaker,
        Policy.WrapAsync(retry, timeout));
}

Register the combined policy once with the typed client. Microsoft.Extensions.Http.Polly provides AddPolicyHandler for this purpose; Polly.Contrib-era registrations use the same policy model. A singleton policy is suitable when it contains no per-request mutable state.

services.AddHttpClient<LedgerClient>(client =>
{
    client.BaseAddress = new Uri("https://api.example.com/");
    client.Timeout = System.Threading.Timeout.InfiniteTimeSpan;
})
.AddPolicyHandler(BuildPolicy());

Do not create a new HttpClient for every request. The factory pools handlers and connections, while the policy controls failures around them. For each endpoint, document the retry count, total latency budget, breaker threshold and fallback, because these values are operational decisions rather than universal defaults.

Observe failures in an Australian operating context

Record attempts, final outcomes, delay, elapsed time, timeout cause and circuit transitions with a correlation identifier. Teams can then distinguish one slow request from a dependency-wide pattern. Australian incident chat is often friendly and informal, using phrases such as no worries or give it a go, but an alert still needs exact status codes, regions and timestamps.

Store timestamps in UTC while retaining local context. Sydney and Melbourne normally use Australian Eastern Standard Time and shift to UTC+11 during daylight saving; Brisbane remains on UTC+10, and Perth uses UTC+8. A single local clock can therefore hide the fact that failures are clustered in one state or during a specific business window.

The NBN and other access networks do not produce identical latency or packet-loss profiles. Fixed-line, regional and mobile paths can behave differently, so use realistic timeout and jitter assumptions. Likewise, an Azure Australia East or Australia Southeast deployment may still depend on an overseas SaaS endpoint, and failover across Australian regions is not instant.

The commercial context matters too. Melbourne is a strong fintech hub, Sydney hosts large enterprise and health workloads, Canberra’s government sector has distinct procurement and security requirements, and Western Australia’s mining and resources teams often operate remote sites. Where the Privacy Act 1988 and Australian Privacy Principles apply, organisations must keep payment and health details out of resilience telemetry.

Match policies to failure modes and test them

Match each control to the symptom it solves. A concurrency limiter protects a local service from excess parallel work; a rate limiter enforces provider quotas; and a fallback supplies a reduced response when recovery is impossible. These controls complement retry, timeout and circuit-breaker behaviour instead of replacing it.

The main mechanisms differ in scope and operational cost:

Test policies with a deterministic fake HttpMessageHandler that returns timeouts, 429s and 503s, then assert the number of attempts and elapsed budget. Inject the delay provider when exact timing matters, and add fault injection around the circuit’s half-open transition. A small load test can expose synchronised retries that unit tests miss.

A production policy is successful when it limits blast radius, preserves caller budgets and emits evidence operators can act on. The right combination matches the dependency’s failure model rather than a generic retry count.