Resumable file uploads with Azure Blob Storage

A multi-gigabyte file sent through one HTTP request is vulnerable to timeouts, browser restarts, mobile network changes and proxy limits. It also forces web servers to receive data they do not need to interpret. Chunking replaces that fragile request with bounded transfers that can be retried independently, giving users a resumable file upload service without holding an entire payload in memory.

Building a file upload service with chunking and Azure Blob Storage follows this pattern: an ASP.NET Core API coordinates the upload while Azure Blob Storage keeps the bytes in a scalable block blob. The service tracks the session, validates each part and commits the finished object. For attendees following the C# and cloud tracks at C# Corner Annual Conference 2020, this demonstrates how API design, retry policies and cloud primitives combine in a production-oriented C# solution.

Why direct uploads become fragile

Large uploads are part of everyday Australian workflows: engineering teams in Sydney exchange project archives with offices in Melbourne and Brisbane, while field users may upload from regional centres over fixed wireless, DSL or mobile connections. Distance does not guarantee failure, but variable bandwidth and changing IP addresses make one long-lived request a poor assumption. The service should treat interruption as normal and preserve completed work.

Chunked transfer improves resilience because only the failed part needs another attempt. It also enables progress reporting, per-part checksums and controlled parallelism. After each request, the API can report the accepted indexes, allowing a client to resume from a server-confirmed position. If an application node restarts, already staged blocks remain available to the completion step.

Chunk size is a trade-off, not a magic number. Smaller parts reduce retry cost but increase request counts; larger parts improve throughput but waste more data when a connection drops. An 8 MiB chunk is a practical starting point, while high-volume ingestion may use 16–32 MiB parts. Keep the selected size consistent for the entire session.

Designing a resumable upload protocol

Expose three operations: create a session, upload a chunk and complete the session. The create request carries the declared file name, size, media type and optional checksum. The server authenticates the caller, applies quotas, generates an opaque upload identifier, and returns the chunk size and endpoint. Each chunk request carries that identifier, an index and the binary body; completion returns the final blob reference and status.

Persist the session before accepting data. Its metadata can contain the tenant, user, expected size, chunk size, accepted indexes, expiry and state. Generate deterministic Azure block IDs from the upload identifier and index, then Base64-encode them; all block IDs in a blob must have the same length. Re-uploading an index replaces the block with that ID, making retries safe once the bytes have been checked.

To finish, read the recorded indexes, place them in ascending order and call CommitBlockListAsync. Azure then exposes the block blob as one object. Treat completion as idempotent: if the database update fails after Azure commits, a retry should reconcile the existing object instead of creating another. A distributed lock per upload prevents concurrent completion requests from racing.

Configuring Azure Blob Storage

Use a private storage account and disable public blob access. Australian workloads should choose a region that matches their users and governance needs: Australia East in Sydney and Australia Southeast in Melbourne are common choices, while Australia Central can suit some central-government scenarios. An Australian copy can reduce latency and avoid unnecessary cross-border movement, but residency does not replace contractual or regulatory controls.

Target a block blob and stage each part with StageBlockAsync. Current Azure Storage APIs permit much larger blocks than 8 MiB, but a blob can contain at most 50,000 blocks. Choose the chunk size so the largest valid file fits that ceiling; 8 MiB parts support roughly 390 GiB, while 32 MiB parts support about 1.5 TiB. Azure discards uncommitted blocks that receive no activity for seven days, so abandoned sessions naturally expire.

Give each tenant a separate container or validated prefix, and create the real blob name on the server rather than trusting the uploaded name. Let the application use a narrow managed identity instead of embedding an account key. Enable diagnostic logs, metrics and, where required, soft delete or versioning. For direct browser-to-Azure transfers, configure CORS carefully and issue short-lived, user-delegated SAS tokens with only the required permissions.

Building the service with ASP.NET Core

Keep the ASP.NET Core layer thin. A typical .NET implementation obtains BlobServiceClient through DefaultAzureCredential, resolves the destination container and streams the request into Azure:

var blob = container.GetBlobClient(blobName);
var blockId = Convert.ToBase64String(
    Encoding.UTF8.GetBytes($"upload:{uploadId}:{index:D6}"));

await blob.StageBlockAsync(blockId, request.Body, cancellationToken);

The Azure.Storage.Blobs SDK handles streaming and transient retries. Configure request limits for the chosen chunk plus protocol overhead, and never read Request.Body into a byte array. Validate the declared length and reject indexes outside the agreed range before staging, preventing clients from creating invalid blocks. Attach a correlation identifier to every request and retain it with the upload session for support and audit work.

At completion, reconstruct the ordered block-ID list and call CommitBlockListAsync, optionally with an If-None-Match condition so an existing object is not overwritten. Update the session only after Azure confirms the commit. If the process stops between those actions, background reconciliation can compare session state, the committed list and the blob ETag. This covers partial business operations as well as network faults.

Securing files and meeting Australian obligations

Treat the upload endpoint as an authenticated business operation. Require a tenant-scoped identity, confirm that the caller may write to the destination, and apply per-user or per-tenant storage quotas. Never derive authorisation from a client-supplied file name. For direct Azure uploads, a user-delegated SAS can be limited to one blob, a short expiry and write-only actions, while the application issues final download links.

Validate file names and extensions, enforce size and media-type policies, and scan content for malware before other users can access it. A declared media type is useful metadata but does not prove the contents. Store a server-calculated checksum and compare it with the completed payload. Rate limits should protect both the API and Azure transactions, and audit records should identify who initiated, changed or downloaded an object.

Privacy duties depend on the information in the files. Organisations handling Australian personal information may need to follow the Australian Privacy Principles under the Privacy Act 1988, including security and accountability, and assess whether a serious data breach is notifiable. Health information can also trigger Victorian or Queensland health-records legislation. Define the purpose, retention period and deletion rules, encrypt data in transit and at rest, keep secrets in Azure Key Vault, and exclude sensitive data from logs. Residency, contracts and technical controls must be reviewed together.

Running the service in production

Production telemetry should join the API request, upload session and Azure operation. Track chunk acceptance rate, retry count, time to first byte, completion duration, abandoned sessions and storage transactions. Azure Monitor and Application Insights can reveal gateway throttling, client disconnects and failing regions. Correlate these signals with deployment versions so a regression can be identified before users abandon large uploads.

Capacity and cost need separate models. Application compute grows with concurrent chunks, object storage grows with retained bytes, and transaction and egress charges can vary with traffic. For an Australian business case, express those costs in Australian dollars, show GST separately and compare peak regional demand rather than average throughput. Teams using a local managed-service provider still need the same identity, monitoring and recovery controls.

Three deployment patterns make the operational trade-offs visible, with each shifting validation and network load to a different layer: