Background file processing pipelines with C# channels

Modern applications rarely process every file inline. A team in Brisbane might receive scanned invoice PDFs from a regional office every morning, a logistics firm in Melbourne may drop shipment manifests into a shared drive, and a Perth-based accounting practice needs to reconcile thousands of CSVs each week before quarter-end. In all of these scenarios the .NET runtime offers a tidy answer: a long-running hosted service that consumes a producer-consumer pipeline built on Channels. The Channel API in System.Threading.Channels delivers a lock-free, asynchronous queue that scales from a single developer laptop to a multi-node deployment without dragging in a heavy message broker.

This article walks through a complete implementation, from project scaffolding in Visual Studio or JetBrains Rider to wiring up a FileSystemWatcher that feeds the channel, and finally to graceful shutdown that survives SIGINT in containers. We will compare Channels with several alternatives so the trade-offs are visible, and touch on local realities such as working with the Australian Privacy Principles when log files contain personal information, or routing ATO-mandated reports through a queue that can be audited end to end.

Understanding System.Threading.Channels in .NET

Channels were introduced in .NET Core 3.0 and have since become the recommended approach for in-process producer-consumer scenarios. A channel is a ring-buffer-like structure that pairs a writer with one or more readers and supports both bounded and unbounded capacity. The bounded variant is particularly useful for file processing because it lets the pipeline apply natural backpressure: when the consumer falls behind, the writer awaits instead of allocating gigabytes of queue memory on a Sydney developer's workstation.

At the heart of the API are ChannelWriter<T> and ChannelReader<T>. The writer exposes TryWrite, WriteAsync, and Complete, while the reader offers ReadAsync, TryRead, and WaitToReadAsync. Async semantics mean a single ASP.NET Core process can host a worker that processes thousands of files per hour without consuming a thread per item. Compared with ConcurrentQueue<T>, the channel signals completion cleanly, and unlike BlockingCollection<T>, it integrates with the modern Task-based world without locking wrappers. When teams in Adelaide pick Channels, they typically do so because the API feels native to async/await and removes the legacy baggage of older collections.

Configuring the IHostedService in an ASP.NET Core application

ASP.NET Core ships with a generic host that already knows how to manage long-running services through IHostedService. A background worker that reads from a channel is just a class implementing IHostedService or, more commonly, inheriting BackgroundService. The host takes care of starting the service when the web app boots, calling StopAsync when a Ctrl+C arrives, and forwarding cancellation tokens that reflect the configured shutdown timeout.

In a typical solution the worker is added inside the shared service file as either AddHostedService<FileProcessorWorker>() or, when multiple queues are involved, AddSingleton<IFileQueue, ChannelFileQueue>(). The hosted service receives an IServiceProvider so it can resolve scoped dependencies such as DbContext or HttpClient on a per-message basis. This pattern is the same one used in the related walkthrough, where minimal hosting reduces the boilerplate to a Program.cs file that fits on a single screen. For a real workload, though, the file-processing pipeline usually needs more configuration than a minimal API, so Program.cs often ends up declaring options binding, logging, and the channel factory alongside the host registration.

Defining the channel and message contract

Before any code is written, it helps to decide what a "file message" actually looks like. A record type called FileWorkItem with a string Path, a string CorrelationId, and a DateTimeOffset EnqueuedAtUtc gives every consumer enough context to log meaningful diagnostics. Including the correlation id is essential when audit logs must travel back to a source system under the Notifiable Data Breaches scheme that operates alongside the Privacy Act 1988.

The channel itself is created with Channel.CreateBounded<FileWorkItem>(new BoundedChannelOptions(capacity) { FullMode = BoundedChannelFullMode.Wait, SingleReader = false, SingleWriter = false }). Setting FullMode to Wait means writers block gracefully when the queue fills, which is the safest default for file ingestion. A bounded capacity of a few hundred items is usually enough to absorb spikes from overnight batch uploads without risking an out-of-memory exception on a virtual machine with limited RAM, a common constraint for small businesses hosting in Australian data centres.

Implementing the background worker consumer

The worker overrides ExecuteAsync, receives a singleton ChannelReader<FileWorkItem> through dependency injection, and enters a loop that awaits WaitToReadAsync followed by ReadAsync. Each item is handed to a private ProcessAsync method that opens the file, streams it into a parser, and writes the extracted records into a sink such as a SQL Server table or a Cosmos DB container. Wrapping the body in a try-catch-finally block ensures a single malformed file does not kill the entire worker — a critical property when the pipeline runs unattended through an Australian night.

Cancellation tokens from the host are observed at the top of the loop, allowing the service to drain the remaining queue during the shutdown grace period rather than abandoning work. Logging is done through ILogger<T> with structured properties so entries can be filtered by correlation id in Application Insights or Seq. For organisations regulated by the ACCC or APRA, structured logs are also easier to redact before they leave Australian shores.

Hooking up the producer with FileSystemWatcher

The producer side is usually a thin file wrapper around FileSystemWatcher, configured to watch a configurable directory such as C:\uploads or /var/incoming. The watcher fires Created and Renamed events, and the handler debounces them — a feature important on Windows network shares where a single file save may emit multiple events — before pushing the path onto the channel. Debouncing is rarely needed on Linux but is helpful when the watcher is attached to a path mounted via SMB from a Windows file server in a Melbourne branch office.

To avoid losing events when the watcher buffer fills, the Filter property restricts file extensions, the InternalBufferSize is increased, and the EnableRaisingEvents flag is set only after the channel writer is fully initialised. A common Australian pattern is to point the watcher at a OneDrive for Business sync folder or a SharePoint Online library using the Microsoft Graph SDK, which means the pipeline must tolerate authentication throttling when many files arrive simultaneously.

Managing backpressure, errors, and graceful shutdown

Backpressure is what separates a robust pipeline from one that silently drops work. With a bounded channel and FullMode.Wait, the producer naturally pauses when consumers fall behind, which is exactly the behaviour a finance team needs during end-of-month reconciliation. Errors are handled in two layers: per-item exceptions are logged and the item is moved to a dead-letter subdirectory, while exceptions thrown outside the per-item scope are caught and trigger a host shutdown orchestrated by IHostApplicationLifetime.

Graceful shutdown is tested by sending SIGTERM to a container, which is how Kubernetes on services such as AKS or EKS requests termination. The worker observes the cancellation token, completes its current item, drains the channel up to the configured timeout, and then exits. Files left in the dead-letter directory can be inspected with the same parser in a console mode, allowing an operator to replay them after fixing the underlying schema error. When the system processes ATO-mandated reports or payment summaries, this dead-letter discipline is what keeps an organisation compliant with both internal audit requirements and external regulators such as AUSTRAC.

Verifying the pipeline on Windows, macOS, and Linux agents

A pipeline that only runs on a developer's machine is not really a pipeline. The same project should target net8.0 (or the current LTS) and run unchanged on Windows for local debugging, on macOS for CI on GitHub Actions runners, and on a Linux container for production deployment. The worker does not rely on Windows-only APIs, so FileSystemWatcher is abstracted behind an IFileSource interface with two implementations: one for the operating system file watcher and one for polling, useful in container environments where inotify events may be unreliable on mounted volumes.

Tests fall into three buckets. Unit tests fake the channel and verify the consumer reads every message in order. Integration tests spin up the host in a WebApplicationFactory, drop real files into a temporary directory, and assert that the sink receives the expected records. End-to-end tests run the worker as a separate process and exercise SIGINT to confirm graceful shutdown completes within the budget. With this layered test approach, a small team in Hobart or Darwin can ship file-processing features with the same confidence as a much larger organisation, and the channel-based architecture remains easy to reason about long after the original developer has moved on.