Building reliable background job queues with Hangfire and SQL Server in C#

Software teams across Australia are quietly running millions of background jobs every day. From the nightly reconciliation that a big-four bank kicks off after the ASX closes, to the document indexing pipelines that keep myGov responsive during tax time, the pattern is everywhere. In most of those codebases, the job queue is a piece of plumbing that nobody wants to write from scratch, and rightly so.

C# developers in Melbourne, Sydney, Brisbane and Perth have largely settled on a small set of tools for this kind of work. Hangfire is one of the more popular choices because it gives you a SQL Server backed queue, a friendly dashboard, retries, scheduling and continuations, all without forcing you to host a separate service. You add a NuGet package, point it at a database, and you have a working job system before lunch.

If you are running on Azure, you also benefit from the local regions. Azure Australia East (Sydney) and Australia Southeast (Victoria) let you keep the queue, the worker and your application database inside the country, which matters for organisations bound by the Privacy Act and the Notifiable Data Breaches scheme. That single decision often rules out hosted SaaS queue products hosted overseas.

This walkthrough builds a small but production-shaped job queue in C# using Hangfire and SQL Server. We will look at the package setup, the database schema, job definitions, dashboard security and a few of the localisations that make the experience smoother in Australian deployments.

Why Australian .NET shops reach for background jobs

A surprising amount of work in line-of-business applications is not user-facing. Generating end-of-month statements, pushing reconciliations to a finance system, sending SMS reminders through a provider, or pulling telemetry from a fleet of IoT devices in a Pilbara mine site are all jobs that take seconds or minutes and do not belong on a request thread.

In many Australian businesses, the trigger is not just performance but compliance. The ATO's Single Touch Payroll requirements, superannuation clearinghouse batches, and ASIC reporting windows all run on rigid cut-off times. Teams at organisations like the NDIA, Services Australia, the big-four banks and large insurers need work that completes before 9 am Sydney time, even if it starts at midnight. That is a scheduling problem dressed up as a business problem, and a queue is the right tool.

Another local factor is the way cloud regions are structured. Azure Australia East and Australia Southeast are paired, and a lot of organisations keep their primary compute in Sydney with failover to Victoria. If your queue is in the same region as your workers, latency is low and the data stays onshore. Hangfire's use of SQL Server fits this naturally because most teams already have an Azure SQL or managed SQL Server instance in the region for their main application data.

Adding Hangfire to a C# project

The simplest entry point is the Hangfire.Core package, plus Hangfire.SqlServer for the storage backend. In a modern .NET 8 or .NET 9 web API, you wire it up in Program.cs with a few lines. The pattern looks like this in spirit: register the services, supply a connection string, and let the middleware extension add the dashboard route.

builder.Services.AddHangfire(config => config
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSqlServerStorage(builder.Configuration.GetConnectionString("HangfireDb"),
        new SqlServerStorageOptions
        {
            CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
            SlidingInvisibilityTimeout = TimeSpan.FromMinutes(15),
            QueuePollInterval = TimeSpan.Zero,
            UseRecommendedIsolationLevel = true,
            DisableGlobalLocks = true
        }));

builder.Services.AddHangfireServer();

A common Australian touch is to use an environment-specific configuration file so that your local dev box points at a developer SQL Server on localhost, your CI pipeline points at a container, and production points at an Azure SQL elastic pool in Australia East. The connection string itself can sit in Key Vault or in appsettings, depending on how the rest of the platform team prefers secrets to be managed.

You will also want to add authorisation for the dashboard. Out of the box Hangfire lets anyone with the URL see the jobs. Even for an internal tool that is rarely a good idea, and in regulated environments it is a non-starter. A simple authorisation filter that requires an authenticated user with a specific role, often an Entra ID group, is the minimum bar.

Wiring SQL Server as the job store

Hangfire creates its schema automatically the first time the application starts, but you will usually want to run the create script in source control. The schema is small and includes tables for jobs, state, parameters, history and a set of locks. Once the tables are in place, the rest of the setup is just connection management.

For Australian workloads, the most common pattern is a single Azure SQL database used only by Hangfire, sized for the number of jobs rather than the volume of data. Because the job table grows over time, a retention job that prunes completed records is worth scheduling. Many teams at fintechs in Sydney and Melbourne run a nightly cleanup at 2 am AEST to keep the table from eating through the elastic pool's storage.

A quick mental model of the lifecycle is helpful. A job moves through the states Enqueued, Processing, Succeeded, Failed, Scheduled, Awaiting, Deleted and AwaitingRetry. Each transition writes to the Job table and the State table, which is what makes the dashboard tick. If you have ever lost a job at 2 am and needed to know whether it ran, was retried, or just vanished, that history is a lifesaver.

Defining and enqueuing jobs

Jobs in Hangfire are ordinary C# methods. You write them as static or instance methods on services you already have, mark them with an interface, and call BackgroundJob.Enqueue from a controller, an event handler or another job. Continuations and scheduled jobs are first-class features.

A realistic example is a retail team running price drops across a national catalogue. When a marketing user clicks "Apply promo" in a Sydney office, the API enqueues a job that fans out to dozens of store servers across the country. Each fan-out job is itself enqueued, and a final job aggregates the results and writes a CSV into SharePoint for the buying team.

public class PriceUpdateService
{
    private readonly IBackgroundJobClient _jobs;

    public PriceUpdateService(IBackgroundJobClient jobs) => _jobs = jobs;

    public string SchedulePriceDrop(int promoId, DateTime whenAest)
    {
        var utcStart = TimeZoneInfo.ConvertTimeToUtc(whenAest,
            TimeZoneInfo.FindSystemTimeZoneById("AUS Eastern Standard Time"));
        return _jobs.Schedule<IPromoWorker>(w => w.ApplyPromo(promoId), utcStart);
    }
}

Notice the timezone handling. AEST becomes relevant again because business stakeholders will hand you dates in local time, but the worker runs in UTC. Converting at the boundary keeps the rest of the code clean and avoids the classic "the job ran an hour late" bug that hits teams every October and April when daylight saving flips.

Securing the dashboard and managing retries

The dashboard is half the reason teams pick Hangfire. It lets you requeue failed jobs, view the exception stack trace, see how long a job has been processing, and trigger a one-off run of a method. It also needs to be locked down, especially in Australian government and financial-services environments.

The standard approach is a custom IAuthorizationFilter that checks the user against an allow-list or an Entra ID group. In a typical setup, the operations team has full read/write access, developers have read access in non-production environments only, and everyone else sees a 403. The filter integrates with whatever authentication your app already uses, so single sign-on through Entra ID is straightforward.

Retries are handled automatically. By default Hangfire will retry a failed job up to 10 times with an exponential back-off. For jobs that call flaky external systems such as Australia Post, the ATO's SBR gateway, or a third-party payment processor, you usually want to wrap the call in your own retry policy with Polly, then let Hangfire handle the outer loop. This keeps each layer simple and gives you a single place to adjust behaviour when a provider has a bad day.

Production hardening and migration paths

A mature Hangfire deployment on Azure Australia East is rarely a single instance. The dashboard is usually deployed to an internal admin app, while the workers run as a separate App Service or a container app, scaled out by the queue length. Diagnostic settings push Hangfire's logs into Log Analytics so that a sudden spike in failed jobs lights up an alert in PagerDuty or Opsgenie.

Many Australian teams are also working through a migration from .NET Framework to modern .NET, and Hangfire is a useful citizen in that journey. If you are weighing the broader move, the legacy .NET Framework migration path outlines a realistic order of operations for a lift-and-shift onto Azure. Once the application is on .NET 8 or 9, Hangfire runs as a hosted service without any of the System.Web quirks that used to make background work painful on the old framework.

You will also want to think about how the queue database is patched and backed up. Azure SQL handles most of this for you, but job history is data you may want to keep for auditing. A weekly export of the Job and State tables to a storage account in the same region is a small addition that pays off the first time someone asks why a customer's invoice was sent twice.

Picking the right tool for the job

Hangfire is not the only option in the .NET ecosystem, and the right choice depends on what you already have running.

If you already run SQL Server, Hangfire is usually the path of least resistance. If you are greenfield and fully on Azure, Functions and a Storage queue may be a better cultural fit. Quartz is the right tool when scheduling is the main problem. MassTransit shines when the queue is part of a larger service-to-service conversation.

Jobs worth offloading from the request thread

• Nightly reconciliation and statement generation that must complete before the Sydney trading day opens.
• Outbound webhooks to partners like Australia Post, Stripe or a CRM provider where latency is unimportant.
• Image and PDF processing for uploads from a mobile workforce out in the field.
• Periodic cache warmups and search index rebuilds that do not need a request to trigger them.

Common production gotchas

• Forgetting to scale workers, leaving a single instance to chew through a backlog after a deploy.
• Leaving the dashboard open to anonymous users, which leaks job names and parameter values.
• Not pruning completed jobs, which can balloon the Hangfire database within months.
• Treating job parameters as trusted, even though they are persisted in clear text in SQL Server.