Creating a sitemap generator for ASP.NET Core MVC sites

Search engines discover pages through links, external references and previously known URLs, but a well-formed XML sitemap gives crawlers a direct map of a website’s preferred public pages. For an ASP.NET Core MVC application, generating that map in code is often more reliable than maintaining a static file whenever products, articles, properties or other records change frequently.

A sitemap generator also provides a useful discipline for URL design. It encourages developers to separate indexable pages from account screens, search results, duplicate routes and private content. This matters for Australian websites serving customers in Sydney, Melbourne, Brisbane, Perth and regional areas, where a clear site structure can support both national visibility and location-specific searches.

Why an XML sitemap matters

An XML sitemap is a machine-readable document, normally available at /sitemap.xml, containing canonical URLs that a search engine may crawl. It is a discovery aid rather than a ranking guarantee. Listing a URL does not force Google or Bing to index it, and a sitemap should never be used to hide poor internal linking or duplicate content.

A conventional sitemap uses the urlset root element and the Sitemap protocol namespace. Each <url> entry can contain a <loc> value and optional metadata such as <lastmod>, <changefreq> and <priority>. In practice, lastmod is the most useful optional field when it accurately reflects a meaningful content update. Search engines may ignore the other two fields, so invented frequencies and priorities add little value.

The generator should include pages that are public, canonical and useful to visitors. Typical entries include the home page, category pages, published blog posts, product details and service locations. It should exclude login and registration pages, shopping cart states, filtered search URLs, administrative routes, preview pages and any content blocked by authorisation.

Australian businesses should also decide whether the sitemap contains one domain or several. A site using example.com.au should generally emit URLs from that host, while a separate staging domain must never appear in the production sitemap. If the organisation serves customers in multiple states, location pages can be included when they provide genuine local information rather than thin copies with only a suburb name changed.

Designing the URL inventory

A maintainable implementation begins with a URL inventory rather than a controller that manually concatenates strings. Static MVC actions can be represented by named route values, while database-backed pages can be projected into a common record containing a URL and its last modification date.

For example, an application might collect the home page, an about page, published posts and active services:

public sealed record SitemapEntry(string Location, DateTimeOffset? LastModified);

public interface ISitemapSource
{
    Task<IReadOnlyCollection<SitemapEntry>> GetEntriesAsync(
        CancellationToken cancellationToken);
}

The source can use LinkGenerator to create paths from route names. This is safer than hard-coding /blog/{slug} in several places, especially when an MVC application has areas, culture prefixes or custom route conventions.

public sealed class SitemapSource : ISitemapSource
{
    private readonly LinkGenerator _links;
    private readonly AppDbContext _db;
    private readonly IHttpContextAccessor _httpContext;

    public SitemapSource(
        LinkGenerator links,
        AppDbContext db,
        IHttpContextAccessor httpContext)
    {
        _links = links;
        _db = db;
        _httpContext = httpContext;
    }

    public async Task<IReadOnlyCollection<SitemapEntry>> GetEntriesAsync(
        CancellationToken cancellationToken)
    {
        var request = _httpContext.HttpContext!.Request;
        var entries = new List<SitemapEntry>
        {
            new (AbsoluteUrl("home", null), null),
            new (AbsoluteUrl("about", null), null)
        };

        var posts = await _db.Posts
            .Where(post => post.IsPublished)
            .Select(post => new
            {
                post.Slug,
                post.UpdatedUtc
            })
            .ToListAsync(cancellationToken);

        entries.AddRange(posts.Select(post =>
            new SitemapEntry(
                AbsoluteUrl("post-details", new { slug = post.Slug }),
                post.UpdatedUtc)));

        return entries
            .GroupBy(entry => entry.Location, StringComparer.OrdinalIgnoreCase)
            .Select(group => group.OrderByDescending(x => x.LastModified).First())
            .ToArray();
    }

    private string AbsoluteUrl(string routeName, object? values)
    {
        var request = _httpContext.HttpContext!.Request;
        var path = _links.GetPathByName(request.HttpContext, routeName, values)
            ?? throw new InvalidOperationException($"Route '{routeName}' was not found.");

        return $"{request.Scheme}://{request.Host}{path}";
    }
}

The host and scheme should come from trusted forwarded-header configuration when the application runs behind Azure App Service, a reverse proxy or a load balancer. Otherwise, the generator may publish internal hostnames, an HTTP scheme or a port that is unavailable to search engines. Configure UseForwardedHeaders carefully and restrict trusted proxies rather than accepting arbitrary header values.

Serving the sitemap from MVC

The XML response can be produced by an MVC controller using XDocument, XmlWriter or a dedicated sitemap library. Explicit XML construction gives the application control over escaping, encoding and response caching.

[ApiController]
public sealed class SitemapController : ControllerBase
{
    private static readonly XNamespace Sitemap =
        "http://www.sitemaps.org/schemas/sitemap/0.9";

    private readonly ISitemapSource _source;

    public SitemapController(ISitemapSource source) => _source = source;

    [HttpGet("/sitemap.xml", Name = "sitemap")]
    [ResponseCache(Duration = 3600, Location = ResponseCacheLocation.Any)]
    public async Task<IActionResult> Get(CancellationToken cancellationToken)
    {
        var entries = await _source.GetEntriesAsync(cancellationToken);

        var document = new XDocument(
            new XDeclaration("1.0", "utf-8", null),
            new XElement(Sitemap + "urlset",
                entries.Select(entry =>
                    new XElement(Sitemap + "url",
                        new XElement(Sitemap + "loc", entry.Location),
                        entry.LastModified is { } date
                            ? new XElement(Sitemap + "lastmod",
                                date.UtcDateTime.ToString("yyyy-MM-dd"))
                            : null))));

        return Content(document.ToString(SaveOptions.DisableFormatting),
            "application/xml; charset=utf-8");
    }
}

Register the service in Program.cs:

builder.Services.AddHttpContextAccessor();
builder.Services.AddScoped<ISitemapSource, SitemapSource>();

The URL must be absolute, properly XML-escaped and free from session identifiers or tracking parameters. An ampersand in a query string becomes & in XML. Avoid query-string URLs wherever a clean route is available. If the site deliberately supports multiple language variants, use separate language URLs and apply consistent canonical and alternate-link policies.

A single sitemap supports up to 50,000 URLs or 50 MB uncompressed. Larger sites need a sitemap index that points to multiple files, such as sitemap-posts-1.xml and sitemap-products-1.xml. Splitting by content type can improve cache behaviour and lets a large catalogue update independently from static pages.

Handling dynamic and editorial content

Dynamic pages should enter the sitemap only after they meet the same publication rules used by the application. A post scheduled for next month, a soft-deleted property or a product unavailable in every market should not appear simply because a database row exists. Apply filters for publication status, visibility, tenant ownership and canonical slug before generating entries.

LastModified should come from a trusted domain field such as UpdatedUtc, stored in UTC. Do not use the current time for every URL on every request; that tells crawlers that the whole site changed continuously and makes the metadata meaningless. If an editor changes a headline without changing the page’s substantive content, the update policy should determine whether the timestamp changes.

Some websites have location-based content for Australian states and territories. A page for Victorian building services can be valuable when it includes relevant regulations, contact information and service coverage; a collection of near-identical pages for every suburb may create low-quality duplication. Keep the canonical URL stable and ensure the sitemap reflects the preferred version.

Privacy also affects the URL inventory. Under the Australian Privacy Act and Australian Privacy Principles, URLs should not expose personal information such as email addresses, customer names or insecure token values. A sitemap is public by design and may be downloaded by anyone. Private customer portals, appointment links and account-specific pages belong behind authentication and must be absent from the feed.

Caching, validation and deployment

Generating a sitemap on every request can create unnecessary database load, particularly for a busy commerce site or an application hosted on Azure with multiple instances. Response caching is suitable for smaller sites, while a larger system can create the document after a content update and store it in Azure Blob Storage, a distributed cache or a database-backed snapshot.

When using output caching, invalidate the relevant sitemap after a published item changes. A background worker can rebuild the document, but it should use a cancellation token, handle transient database failures and avoid publishing a partial result. If the last valid snapshot is available, serving it briefly is safer than returning malformed XML or a server error.

Validation belongs in automated tests and deployment checks. Test that every URL is absolute, uses the production host, returns a successful response, has no fragment identifier and maps to a canonical public page. Parse the response as XML rather than checking strings, and verify the expected namespace.

[Fact]
public async Task Sitemap_contains_only_public_absolute_urls()
{
    var response = await _client.GetAsync("/sitemap.xml");
    response.EnsureSuccessStatusCode();

    var xml = XDocument.Parse(await response.Content.ReadAsStringAsync());
    var locations = xml.Descendants(
        XNamespace.Get("http://www.sitemaps.org/schemas/sitemap/0.9") + "loc");

    Assert.NotEmpty(locations);
    Assert.All(locations, node =>
    {
        var uri = new Uri(node.Value);
        Assert.Equal("https", uri.Scheme);
        Assert.Equal("example.com.au", uri.Host);
        Assert.DoesNotContain("#", uri.ToString());
    });
}

A production deployment should also expose the sitemap in robots.txt:

User-agent: *
Disallow: /admin/
Disallow: /account/
Sitemap: https://example.com.au/sitemap.xml

Submit the sitemap URL in Google Search Console and Bing Webmaster Tools, then inspect crawl and indexing reports. For Australian teams, this can be part of a release checklist shared between developers, content editors and SEO specialists in offices across Sydney or Melbourne, including after a domain migration or major route change.

Comparing implementation choices

The right approach depends on catalogue size, update frequency and how much control the development team needs. A static file is simple for a brochure site, while a generated index is more appropriate when content changes throughout the day.

A practical ASP.NET Core MVC solution often combines a shared URL source with a cached XML endpoint. That design keeps route generation close to the application’s routing system, avoids exposing unpublished records and leaves room to move from one file to a sitemap index later. With accurate canonical URLs, careful privacy filtering and regular validation, the sitemap becomes a dependable part of the site’s publishing pipeline rather than a neglected SEO file.