Building resilient .NET APIs with Polly and HttpClient

Modern applications rarely fail because every component is permanently unavailable. They fail through short network interruptions, overloaded services, expired connections, slow DNS lookups, deployment restarts, or a dependency that responds with a temporary 503. A well-designed client must recognise these conditions without turning a small incident into a larger outage.

The circuit breaker pattern gives an application a controlled way to stop calling an unhealthy dependency. Polly provides the resilience policies, while HttpClient and IHttpClientFactory provide the managed HTTP communication layer in .NET. Used together, they can reduce latency spikes, protect downstream systems, and give users a predictable response when a service is unavailable.

This approach is relevant to Australian software teams building systems across Sydney, Melbourne, Brisbane and regional locations. Network paths over the NBN, mobile connections and cloud regions can behave differently, while privacy obligations and customer expectations make uncontrolled retries especially risky. The goal is a measured recovery strategy rather than simply sending the same request again.

Why transient failures need protection

A transient failure is temporary and may succeed if the request is attempted later. Common examples include a gateway timeout, a 429 rate-limit response, a connection reset, or a service returning HTTP 503 during a rolling deployment. Retrying can help in these cases, but retries consume threads, sockets, CPU and downstream capacity. If hundreds of requests retry at the same time, a minor failure can become a retry storm.

A circuit breaker adds a stateful decision around the call. In the closed state, requests flow normally and failures are recorded. When failures exceed a configured threshold, the breaker changes to open, immediately rejecting calls for a defined period. After that period, it enters half-open and permits a limited trial request. A successful trial closes the circuit; another failure opens it again.

This behaviour is different from a timeout. A timeout limits how long one operation can wait, while a breaker limits how often an unhealthy dependency is contacted. Both are important. A timeout without a breaker can still generate a large number of slow requests. A breaker without a timeout may never receive a result because each pending call can wait indefinitely.

The policy should also distinguish failures that deserve a retry from those that do not. A malformed request, a 400 response, a failed authentication attempt or a business validation error will not be fixed by repeating the call. Retrying those responses adds load and can create duplicate side effects.

Designing a Polly policy

Polly supports common resilience strategies including retry, timeout, circuit breaking, fallback and bulkhead-style isolation. For an HTTP client, a sensible policy order is usually timeout, retry and circuit breaker, although the exact composition depends on the Polly version and the desired behaviour. The important detail is that each policy has a clear responsibility.

A retry should use exponential back-off and jitter. Exponential back-off increases the delay after each failed attempt, while jitter adds a small random variation so that many clients do not retry simultaneously. A typical sequence might wait 200 milliseconds, then 500 milliseconds, then one second. The values should reflect the dependency’s latency and service-level agreement rather than being copied blindly.

The following example uses the familiar Polly integration available with IHttpClientFactory:

static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .OrResult(response => response.StatusCode == HttpStatusCode.TooManyRequests)
        .WaitAndRetryAsync(
            retryCount: 3,
            sleepDurationProvider: attempt =>
                TimeSpan.FromMilliseconds(200 * Math.Pow(2, attempt)));
}

static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(
            exceptionsAllowedBeforeBreaking: 5,
            durationOfBreak: TimeSpan.FromSeconds(30));
}

HandleTransientHttpError covers connection errors, HTTP 5xx responses and HTTP 408. A 429 response is added explicitly because rate limiting is common in public APIs. In production, a Retry-After header may be more useful than a fixed delay. Polly versions and package APIs change over time, so teams using newer .NET releases should also review Polly v8 resilience pipelines and the current Microsoft.Extensions.Http.Resilience integration.

The breaker threshold needs careful interpretation. Five failures in a row may be appropriate for a low-volume internal service, but a high-volume API may need a failure ratio over a sampling window. A single global breaker can also hide differences between tenants or regions. Configure policies around the actual dependency boundary, and avoid making a circuit so sensitive that one isolated failure blocks healthy traffic.

Wiring HttpClient safely

HttpClient should generally be created through IHttpClientFactory. Creating and disposing a new client for every request can exhaust available sockets, while keeping one unmanaged client forever can leave stale DNS information. The factory manages handler lifetimes and allows resilience policies to be applied to a named or typed client.

A named client can be registered in a .NET application like this:

services.AddHttpClient("Payments", client =>
{
    client.BaseAddress = new Uri("https://payments.example.com/");
    client.Timeout = TimeSpan.FromSeconds(10);
})
.AddPolicyHandler(GetRetryPolicy())
.AddPolicyHandler(GetCircuitBreakerPolicy());

A service can then request the configured client from IHttpClientFactory. A typed client is often easier to test because its dependency and operations are expressed through a dedicated class:

public sealed class PaymentClient
{
    private readonly HttpClient _httpClient;

    public PaymentClient(HttpClient httpClient) => _httpClient = httpClient;

    public async Task<HttpResponseMessage> GetStatusAsync(
        string paymentId,
        CancellationToken cancellationToken)
    {
        return await _httpClient.GetAsync(
            $"payments/{paymentId}/status", cancellationToken);
    }
}

The HttpClient.Timeout property and Polly timeout policy should not accidentally create confusing nested limits. Define which layer owns the timeout, then document the total time available to the caller. A three-attempt retry policy with a ten-second timeout per attempt can occupy a request for roughly thirty seconds, excluding connection overhead. That may be unacceptable for an interactive checkout flow.

Retries also need an idempotency decision. A GET request is normally safe to retry when the server follows HTTP semantics. A POST that creates an order may create duplicates if the first response is lost after the server commits the transaction. For payment, booking and order workflows, use an idempotency key supported by the downstream API, or avoid automatic retries for non-idempotent operations.

Handling fallback and observability

A circuit breaker should not simply turn an exception into an unexplained 500 response. The calling service needs a deliberate fallback. For a product catalogue, cached data may be suitable. For an exchange-rate service, a recently verified value might be acceptable under a documented business rule. For a payment authorisation, pretending that the dependency succeeded is unsafe; the correct result may be a pending state that can be reconciled later.

Fallback behaviour should be visible to the caller and to operations staff. A response can carry a correlation ID and a clear status such as “temporarily unavailable”, while internal logs record the dependency name, policy event, elapsed time and outcome. Avoid logging access tokens, payment details or personal information. This matters for organisations covered by the Australian Privacy Act 1988 and its Australian Privacy Principles, especially when logs are replicated across cloud regions or external monitoring platforms.

Polly exposes events for retries and circuit transitions. Connect these events to structured logging, metrics and distributed tracing. Useful measures include retry count, timeout count, breaker-open duration, rejected calls, dependency latency and the proportion of responses served by a fallback. A breaker that never opens may be too lenient; one that opens constantly may indicate a bad threshold, a broken dependency or a network problem between an Australian application region and its upstream service.

The following pattern illustrates a guarded call without hiding the business outcome:

try
{
    var response = await paymentClient.GetStatusAsync(paymentId, token);
    response.EnsureSuccessStatusCode();
    return await response.Content.ReadFromJsonAsync<PaymentStatus>(token);
}
catch (BrokenCircuitException)
{
    logger.LogWarning("Payment dependency circuit is open");
    return PaymentStatus.Pending();
}
catch (HttpRequestException exception)
{
    logger.LogError(exception, "Payment dependency request failed");
    throw;
}

The fallback should be designed with the product owner and compliance team. A cached search result may be fine for a retail site, but stale health information or an incorrect financial balance has different consequences. Circuit breaking is an engineering control, not a substitute for business continuity planning.

Operational checklist for production

Australian traffic patterns can expose weaknesses that are less obvious in a local development environment. An application hosted in Sydney may call a dependency in another region, while customers in Perth, Darwin or regional Queensland experience different network paths and latency. Measure from the same cloud regions and connectivity types used by real customers, including mobile access where it forms a meaningful part of the market.

Cloud location also affects governance. Australian organisations may require customer information to remain in an Australian region because of contractual terms, sector rules or internal policy. The Privacy Act and Australian Privacy Principles require careful handling of personal information, and regulated organisations may have additional obligations. Resilience telemetry should therefore be classified, retained and accessed under the same controls as other operational data.

Use this production checklist when reviewing a Polly and HttpClient implementation:

• Set a finite timeout for every outbound call and pass cancellation tokens from the incoming request.
• Retry only transient, safe-to-repeat operations, with exponential back-off, jitter and a bounded attempt count.
• Configure circuit thresholds using measured failure rates, traffic volume and dependency service limits.
• Add idempotency keys or durable reconciliation for payments, orders and other non-idempotent writes.
• Publish breaker, retry, timeout and fallback metrics without exposing personal or confidential customer data.

Testing should include more than a simulated 500 response. Inject DNS failures, connection resets, slow responses, 429 rate limits, malformed payloads and a dependency that recovers after the circuit opens. Verify that cancellation stops work, that half-open calls are limited, and that the application returns a useful result when the dependency remains offline.

Load testing is particularly valuable for services used during Australian business peaks, major retail promotions and end-of-financial-year activity. Test whether retries increase downstream traffic beyond the provider’s quota, whether a breaker opens quickly enough, and whether recovery causes a sudden wave of requests. A resilient client should fail in a controlled way, recover gradually and leave enough capacity for the rest of the application.