Building Event-Driven Systems with Azure Cosmos DB Change Feed in .NET

Azure Cosmos DB has become a familiar workhorse for Australian teams building globally distributed applications, and the change feed feature turns the service into more than just a low-latency document store. When paired with the .NET SDK, the change feed offers a managed, scalable way to react to data modifications without writing complex polling logic or maintaining side tables. For organisations running workloads in Sydney, Melbourne or Brisbane data centres, the pattern helps bridge operational data with downstream event-driven pipelines while keeping read traffic close to home and latency predictable for end users across the country.

This article walks through the design and implementation of a change feed processor, the configuration choices that matter in production, and a few patterns that appear repeatedly in real systems across finance, retail and government projects. If you are attending a developer event such as C# Corner Annual Conference, expect to see this technology featured in several sessions on cloud-native development, and the examples below should give you a useful starting point regardless of the event track you choose to follow on the day.

Understanding the Change Feed Pattern

The change feed is a persistent, append-only log of inserts and updates that occur against a Cosmos DB container. The feed is exposed in chunks, ordered by the partition key range, and clients iterate through it using a continuation model. There is no explicit enablement step, no schema, and no cost beyond the request units already consumed by the writes that produced it. Reads from the change feed count as normal point or query operations against the configured consistency level, which is worth keeping in mind when modelling throughput budgets for shared containers that already serve interactive traffic.

For Australian developers, the most common misconception is that the feed behaves like a message queue. It does not. Items remain in the feed for as long as the consumer keeps the associated lease, and there is no built-in replay window, dead-letter queue or retention policy beyond what your code and lease container define. Treat it as a stateful, partitioned stream of mutations rather than a competing-consumer broker, and design any replay tooling you need to keep around for incidents or migrations.

Because the feed is partitioned, the processor library in the .NET SDK parallelises work across partition key ranges. Each range is leased to one host at a time, and the host persists its progress in a separate leases container. This design is what allows teams to add or remove worker instances without re-engineering consumer logic, even when those workers live in different Azure regions serving a Sydney-based retail platform or a Melbourne banking workload. The same machinery also underpins most replication and audit pipelines that Australian Cyber Security Centre accredited organisations use to keep secondary stores in sync, including the read-only replicas that several federal agencies maintain for reporting purposes.

Setting Up the .NET SDK and Leases

A typical project starts by referencing the Microsoft.Azure.Cosmos NuGet package and creating a CosmosClient instance configured for the workload. The client should be registered as a singleton, since it pools connections and manages internal transport state. Connection policy options such as ApplicationPreferredRegions let you set Australia East as the primary region and Australia Southeast as a failover, which aligns with the Australian Prudential Regulation Authority expectations for resilience in financial services workloads. Teams that operate outside the finance sector still tend to follow the same region-pinning pattern because it keeps the request path short and predictable for customers connecting from Australian ISPs.

The leases container is where most configuration mistakes happen. It should live in the same Cosmos DB account so that writes and leases share the same partition topology, but it is normally placed in a separate database for clarity. The container must be created with a default TTL that matches your recovery time objective, because the processor relies on lease documents expiring to recover from a worker crash. Setting TTL too low causes unnecessary handovers in chatty workloads; setting it too high delays recovery when a process dies in production. A 60-second default is a sensible starting point for most consumer-grade services, with adjustments driven by observed health rather than initial guesswork, and a small number of teams in Adelaide and Perth have reported better outcomes with 90 seconds on a slower network path.

The processor itself is constructed with GetChangeFeedProcessorBuilder<T>, where T is the type that the delegate will receive. The builder lets you configure the instance name, which is how multiple worker hosts cooperate; the poll interval, which defaults to five seconds and controls how quickly the processor detects new changes; and a start-from-beginning flag, which is useful for backfills when a downstream system has just been created. Australian teams regularly use this flag when seeding an analytics warehouse for the first time after a major migration, particularly for retail and loyalty platforms that are rebuilt every few years to support new marketing campaigns.

Implementing the Change Feed Processor

The handler delegate is the heart of the system. It receives a ChangeFeedProcessorContext and a stream of changes, and it is responsible for both deserialisation and any business logic that should run in response. A common pattern is to keep the handler lean, perform idempotent work, and forward a typed event to a downstream bus such as Azure Service Bus or Event Grid. Heavy processing should be offloaded, because the lease stays held until the delegate returns, and any latency there will block every other change in the same partition key range from being delivered.

var processor = client.GetContainer("orders", "live")
    .GetChangeFeedProcessorBuilder<OrderEvent>("order-events", HandleAsync)
    .WithInstanceName(Environment.MachineName)
    .WithLeaseContainer(() => client.GetContainer("ops", "leases"))
    .WithStartTime(DateTime.UtcNow.AddMinutes(-5))
    .Build();

await processor.StartAsync();

The HandleAsync method should treat each batch as a unit. If the work for a batch is transactional in nature, wrap it in a try-catch and decide whether to throw, log and continue, or move the failing document to a dead-letter container. Decisions like this are often governed by the Australian Privacy Principles when the data being processed includes personal information; if a failure risks exposing data through a partial update, the delegate should fail loudly and let the lease roll over to another host for retry. Storing failed payloads in a dedicated container gives you a clean target for rehydration jobs and forensic review under the Notifiable Data Breaches scheme.

Consider also the throughput of the underlying container. The change feed itself does not consume extra request units, but the leases container does, and the host application needs enough memory and CPU to keep up with the chosen poll interval. In a small team in Brisbane running a two-instance processor on Basic tier VMs, monitoring the lease container's request unit consumption has caught several production incidents that would otherwise have surfaced as silent lag. The lesson generalises: the lease container is part of your real-time data path and deserves the same operational treatment as any hot table.

Scaling, Reliability and Failure Handling

Scaling the change feed processor is largely a matter of adding instances, because the leases container mediates the work. A common misstep is to add a worker per logical service without considering how partition key ranges line up. If a container has only a handful of physical partitions, throwing more workers at it yields diminishing returns, and the cost of each host may outweigh the throughput benefit. Teams that run containers on Azure Kubernetes Service in the Australia East region often cap their processor count at the partition count observed in the metrics, which keeps the unit economics healthy without hurting responsiveness.

For high-availability deployments, place processor instances in different availability zones. In Australian regions, zone-redundant configuration has become a default ask from regulators and large customers, particularly for the four major banks. Cosmos DB itself offers zone-redundant storage, and the processor library can run across zones as long as no two instances share the same machine name and they all share the leases container. A simple approach is to derive the instance name from the pod name or VM name, which Kubernetes and Azure VM scale sets guarantee to be unique within the cluster.

Observability should not be an afterthought. The processor exposes a ChangeFeedProcessorState from a separate query, and metrics such as estimated lag, work performed, and exceptions are available through Application Insights when you wire the SDK to the standard Microsoft.Azure.Cosmos event source. Australian teams that have adopted the Australian Cyber Security Centre's Essential Eight often map these signals to a central SIEM, since the change feed is frequently the backbone of audit and event-replication pipelines. Lag-based alerts are especially valuable because they catch problems before customers notice downstream effects, and the same alerts feed nicely into a runbook that walks an on-call engineer through a region failover.

Comparing Common Change Feed Approaches

The processor library is the right starting point for most production systems. Manual reads are best reserved for backfill scripts and one-off migrations, where the cost of standing up a leases container is not worth it. The Functions trigger is convenient but couples the change feed to a Functions Premium or Container Apps hosting plan, which is something Australian teams often debate when the workload is part of a cost-sensitive internal platform serving government agencies subject to whole-of-government cloud procurement rules. The table above is meant as a quick reference, not a strict rulebook; some teams mix approaches, running a Functions trigger for spiky feeds and a processor library for the steady core traffic.

Practical Recommendations for Production Rollouts

• Always create the leases container in the same account as the source container, but in a separate database, with a default TTL aligned to your recovery objectives.
• Keep the change feed handler small and idempotent, and forward events to a durable bus such as Service Bus or Event Grid for any non-trivial downstream work.
• Run at least two processor instances in different availability zones, and use ApplicationPreferredRegions to keep traffic within Australia where latency and data residency matter.
• Treat the feed as a stateful stream rather than a queue, and build a separate retention strategy if your auditors expect long replay windows under the Privacy Act 1988.
• Monitor the lease container's request units, the work performed metric, and the estimated lag, and alert on sustained deviation from the baseline.
• Add a synthetic write that exercises the full handler path during smoke tests, so a silent deserialisation failure does not go unnoticed for weeks.
• When backfilling historical data, set the start time explicitly and use a dedicated processor instance so that ongoing traffic is not affected.