Handling environment variables in Azure DevOps for .NET pipelines

Environment variables are the connective tissue of any non-trivial CI/CD pipeline. In an Azure DevOps workflow that targets .NET, they decide which connection string the build picks up, which feature flag stays active in staging, and which telemetry endpoint gets compiled into the production artefact. A misnamed variable can quietly ship a test database URL into a customer-facing API, and the resulting incident postmortem usually traces back to a single overlooked setting in the variable group.

For Australian development teams, the stakes are a little higher than usual. Privacy obligations under the Notifiable Data Breaches scheme, the ACSC Essential Eight maturity expectations that auditors now reference, and the AU-NZ Azure region pair that most local enterprises standardise on all influence how environment configuration should be designed. The patterns below are written for engineers shipping .NET workloads from Sydney, Melbourne, Brisbane, and Adelaide who need consistent behaviour across dev, test, and production without leaking secrets into logs or pull request output.

Why environment variables matter in .NET pipelines

.NET has always read configuration from a layered stack of providers. The host builder walks through environment variables, JSON files, command-line arguments, and key vaults in a defined order, and the last writer usually wins. Once an Azure DevOps pipeline writes a value into the process environment, ASP.NET Core's default configuration provider picks it up automatically, with the double underscore replacing the JSON hierarchy separator. That means ConnectionStrings__PrimaryDb in the pipeline becomes ConnectionStrings:PrimaryDb inside IConfiguration, with no custom code required.

The same mechanism applies to worker services, Azure Functions running in-process, and console applications targeting .NET 8 or later. Anything the framework exposes through IConfiguration is reachable from an environment variable, which makes the variable group the central nervous system for an entire solution. A team in Brisbane consolidating ten micro-services under one pipeline often ends up with hundreds of variables, and the naming discipline applied on day one determines whether that catalogue grows into a manageable system or becomes a permanent source of outages.

A subtler reason to care is build reproducibility. When the same commit produces the same artefact on a developer's laptop in Perth, a build agent in the AU-NZ region, and a release pipeline running in production, the variables feeding the build must be deterministic and traceable. An internal page that lists every variable, its source, and its scope pays for itself the first time a hotfix has to be shipped on a Sunday night.

Setting up variable groups in Azure DevOps

A variable group is the natural unit of organisation for related settings. Most teams create at least three: one for shared, non-secret values such as the target .NET SDK version and the SonarQube project key; one per environment covering connection strings, feature flags, and service URLs; and one per release ring covering deployment windows and approval contacts. Variables are added through Library in the Pipelines hub, and each entry accepts a name, a value, an optional secret flag, and a comment that survives rename operations.

Linking a group to a pipeline is done through a variables clause in the YAML or through the classic editor's variable group selector. Once linked, the variables become available to every job and step in the pipeline unless the link is scoped narrower. For a multi-stage YAML pipeline targeting azure-pipelines.yml, the typical pattern is to link the shared group globally and the environment-specific groups per stage, which lets a single template file deploy to an integration slot in Sydney and a production slot somewhere else without duplicating variables. Resource owners can also be configured to gate changes, so a DBA in Melbourne can approve a connection string update before it reaches production.

Variable groups are pure files, and Azure DevOps stores their values in an encrypted store tied to the project. They are not meant for long-term key storage, however, and that is where Azure Key Vault enters the picture.

Quick checks before linking a variable group:

• Confirm every secret in the group is sourced from Key Vault rather than typed in by hand, and that the vault has soft delete and purge protection enabled.
• Decide the naming convention up front, including how double underscores will replace dots in section names that ASP.NET Core reads.
• Restrict the group's role-based access to the security groups that genuinely own the values, and review the membership every quarter.

Securing secrets with Azure Key Vault

Secrets should rarely live in a variable group for any length of time. The recommended pattern is to store them in an Azure Key Vault and link the vault to the variable group through an Azure subscription service principal. When the group is refreshed, every secret name becomes a pipeline variable, and the redacted string shown in the UI hides the actual value from anyone reading the variable list.

This arrangement also unlocks the ACSC-aligned controls that Australian enterprises usually need. Customer-managed keys, soft delete with a 90-day retention, and purge protection can all be enabled at the vault level, giving auditors a clear answer when they ask where a connection string lives. The pipeline then uses the variable as it would any other, typically through $(ConnectionStrings.PrimaryDb) in a task or through the environment variable that ASP.NET Core reads automatically. The vault access policy still applies, so a junior developer in Adelaide cannot read production secrets even with pipeline access.

Key Vault also integrates with managed identities at deployment time, which removes the need to ship connection strings to the target App Service or AKS cluster at all. The pipeline fetches the value, the target workload fetches it again at runtime through its own managed identity, and the secret never crosses a network boundary in plaintext. Many Australian financial-services firms have standardised on this arrangement because it removes a whole category of audit findings around secrets in transit, and the same pattern satisfies APRA CPS 234 obligations for material service providers.

Variable scope and precedence

Precedence in Azure DevOps pipelines follows a well-defined order, and getting it wrong is the single most common cause of "works on my machine" pipeline incidents. Pipeline-level variables override variable groups, stage-level variables override pipeline-level variables, and job-level variables override stage-level. A variable set explicitly through a script (##vso[task.setvariable]) at runtime wins over every static declaration. The same precedence applies to secrets, with the additional rule that secret variables cannot be expanded into other variables or script arguments in a way that exposes them in logs.

Practical scoping often comes down to naming. A team in Sydney that runs integration tests in a CI pipeline and a release into UAT from a CD pipeline usually splits variables by lifecycle. CI variables feed build and test stages, while CD variables feed only the deployment stages. The same variable name can then have different values per ring without naming gymnastics, and a release gate in UAT can override the value entirely. The audit trail in the pipeline run view makes it easy to confirm which scope actually won, which is invaluable when a deployment appears to use the wrong value.

Variable templates are the other half of the story. A variables.yml file committed alongside azure-pipelines.yml defines a reusable parameter set that branches can override through parameters. Combining templates with variable groups gives teams a clean way to balance central control with local flexibility, a balance that Australian engineering managers often need when product squads in different cities share a common platform.

Mapping variables to appsettings across environments

The classic mistake is to assume that the same JSON hierarchy works for every environment. appsettings.Development.json lives in source control, appsettings.Production.json does not, and the production values must come from somewhere. In an Azure DevOps pipeline, that somewhere is the variable group, which exposes values through environment variables that ASP.NET Core reads with the __ separator.

The same mapping handles the configuration sections that .NET 8 introduced, such as Kestrel__Endpoints__Http__Url and Logging__LogLevel__Default. A common pattern in Australian teams is to keep environment-specific knobs in a single naming convention: ASPNETCORE_ENVIRONMENT selects the profile, ServiceBus__ConnectionString carries the namespace key, and Telemetry__ConnectionString feeds Application Insights. Treating these names as a published contract lets frontend, backend, and platform engineers share a single reference page, often kept in an internal wiki that links back to the pipeline definition.

The mapping also matters for build-time configuration. Source generators in modern .NET, for example, can read configuration values to embed constants into compiled output. A pipeline that sets Build__Version, Build__Commit, and Build__Branch through variables gives the source generator everything it needs to produce artefacts stamped with the right metadata, and the resulting DLLs are immediately traceable back to the commit that produced them.

Common pitfalls, hardening, and the catalogue lifecycle

A few recurring issues appear in almost every Azure DevOps engagement involving .NET pipelines. The comparison below summarises the most common pitfalls, their symptoms, and the configuration that prevents them. Engineers who internalise it can usually diagnose a broken pipeline within minutes rather than hours.

A related issue is variable expansion across pipelines. When a multi-repo project in Melbourne shares a variable group with another project, the project collection scope of the group matters. A project-scoped group only resolves within its own project, while a project collection group is visible everywhere. The latter is convenient but expands the blast radius of a misconfigured value, so most platform teams default to project scope and use Azure Key Vault for any genuinely shared secret.

Habits that keep the catalogue maintainable:

• Treat variable names as part of the public API of the pipeline and document every change in a changelog entry tied to the release.
• Rotate secrets through Key Vault on a fixed schedule and let the pipeline pick up the new value rather than editing the variable group by hand.
• Use the Azure DevOps Wiki or a similar internal page to publish the canonical list of variables and their owners, and link it from the pipeline YAML.

For teams that want a deeper dive into the documentation side of the equation, a walkthrough of the Azure DevOps Wiki guide pattern sits alongside the patterns above and complements the variable catalogue approach described here.