Real-World Scenarios: An SEO Audit Found Every noindex Page Still Listed in the Sitemap

Background reading: Sitemap Generation Edge Cases: Multi-Value Properties, Vanity URLs, and Template-Aware Alternates covers the mechanics behind this incident.

Problem Statement

A quarterly SEO audit compared the site's actual live sitemap.xml against a list of pages marked noindex in the CMS via the standard Robots Tags checkbox field. The expectation was zero overlap — every noindex page should have been excluded from the sitemap by the custom generator's addResource override, which had a dedicated check for exactly this. Instead, every single noindex page in the sample was present in the sitemap, with no exceptions.

That "no exceptions" detail mattered. A partial leak — a handful of pages slipping through — would point at an edge case in a specific content type or workflow. All of them failing pointed at something structurally wrong with the check itself, not a gap in its coverage.

Approach and Why

Rather than guessing at the fix, the first step was reproducing the check in isolation against a page configured exactly the way the CMS's Robots Tags field actually stores its value — a multi-value property, since the field is a checkbox group supporting combinations like noindex plus nofollow, not a single free-text value. The existing unit test for the noindex check had mocked the property read as a plain string, which is exactly why it had never caught this: the mock never exercised the real multi-value type the field actually produces in production.

Once the check was tested against a genuinely multi-value property, it failed immediately and made the root cause obvious: the code called the untyped, single-argument ValueMap.get(name), got back a String[], and called .toString() directly on it — producing Java's default array-identity string instead of a comparable value. The equality check downstream had never once matched against a real page's Robots Tags value; it had been comparing "noindex" against something like [Ljava.lang.String;@6bc7c054 since the day it shipped.

The same audit pass also surfaced two smaller, related issues in the same generator worth fixing together rather than filing as separate follow-ups: alternate-language links that pointed at unresolved vanity paths instead of real content paths, and hreflang links being attached to pages that didn't actually share a template family with the current page.

POC

class RegionalSitemapGeneratorTest {

    @Test
    void multiValueNoIndexTagExcludesPageFromSitemap() {
        Page page = mockPageWithRobotsTags(new String[] { "noindex", "nofollow" });
        InMemorySitemap sitemap = new InMemorySitemap();

        generator.addResource("audited-page", sitemap, page.adaptTo(Resource.class));

        assertThat(sitemap.getUrls())
            .as("a page with noindex among its robots tags must never appear in the sitemap")
            .isEmpty();
    }

    @Test
    void singleValueRobotsTagStillWorks() {
        // guards against a fix that only handles the multi-value case and
        // regresses the (less common) single-string legacy property shape
        Page page = mockPageWithRobotsTags(new String[] { "noindex" });
        InMemorySitemap sitemap = new InMemorySitemap();

        generator.addResource("audited-page", sitemap, page.adaptTo(Resource.class));

        assertThat(sitemap.getUrls()).isEmpty();
    }

    @Test
    void pageWithOnlyNofollowIsStillIndexable() {
        // nofollow alone should not be treated as noindex - a real distinction
        // the check must preserve, not collapse into "any tag present = excluded"
        Page page = mockPageWithRobotsTags(new String[] { "nofollow" });
        InMemorySitemap sitemap = new InMemorySitemap();

        generator.addResource("audited-page", sitemap, page.adaptTo(Resource.class));

        assertThat(sitemap.getUrls()).hasSize(1);
    }

    @Test
    void vanityAlternatePathResolvesToRealContentBeforeExternalizing() {
        Page page = mockPageWithAlternates(Map.of(Locale.GERMANY, "/de/produkte/widget"));
        InMemorySitemap sitemap = new InMemorySitemap();

        generator.addResource("widget", sitemap, page.adaptTo(Resource.class));

        Url entry = sitemap.getUrls().get(0);
        assertThat(entry.getExtensions(AlternateLanguageExtension.class))
            .as("vanity-path alternate should resolve to the real content path, not a broken link")
            .hasSize(1);
    }

    @Test
    void templateMismatchedAlternateIsNotAttached() {
        Page page = mockPageWithAlternates(Map.of(Locale.FRANCE, "/content/site/fr/campaigns/microsite"));
        mockTemplateForPath("/content/site/fr/campaigns/microsite", "/conf/site/templates/promo-microsite");
        InMemorySitemap sitemap = new InMemorySitemap();

        generator.addResource("widget", sitemap, page.adaptTo(Resource.class));

        Url entry = sitemap.getUrls().get(0);
        assertThat(entry.getExtensions(AlternateLanguageExtension.class))
            .as("a page on a different template family is not a real translation and must not get an hreflang link")
            .isEmpty();
    }
}

Running the first test against the original, unfixed code was the confirming step — it failed exactly as the audit predicted, with the sitemap containing the noindex page's URL despite the check supposedly excluding it.

Implementation

The fix to the property read, changing from an untyped single-object read to the typed multi-value overload:

// BEFORE: untyped read + toString() on what is actually a String[] for
// any page using the standard multi-select Robots Tags field
private boolean isNoIndex(Page page) {
    String robotsTags = page.getProperties().containsKey(SeoProperties.PN_ROBOTS_TAGS)
            ? page.getProperties().get(SeoProperties.PN_ROBOTS_TAGS).toString()
            : StringUtils.EMPTY;
    return StringUtils.equalsIgnoreCase("noindex", robotsTags);
}

// AFTER: typed multi-value read, checked by membership not string equality
private boolean isNoIndex(Page page) {
    String[] robotsTags = page.getProperties().get(SeoProperties.PN_ROBOTS_TAGS, new String[0]);
    return Arrays.stream(robotsTags).anyMatch(tag -> "noindex".equalsIgnoreCase(tag));
}

The vanity-path resolution and template-compatibility check were added as two new, independently-named private methods rather than folded into the existing alternate-link loop, so each has its own clear failure mode and its own log line when it skips a link:

for (Map.Entry<Locale, String> alt : alternates.entrySet()) {
    Optional<String> resolvedPath = resolveAlternateContentPath(alt.getValue(), siteRoot, resolver);
    if (resolvedPath.isEmpty()) {
        continue;
    }
    Resource alternateResource = resolver.getResource(resolvedPath.get());
    if (alternateResource == null || !isTemplateCompatible(alternateResource)) {
        LOG.debug("Skipping hreflang alternate at '{}' - template family does not match", resolvedPath.get());
        continue;
    }
    String externalAlt = externalizer.externalize(alternateResource);
    url.addExtension(AlternateLanguageExtension.class).setHref(externalAlt).setLocale(alt.getKey());
}

Rollout Steps

1. Reproduced the bug with a unit test against a real multi-value property shape, confirming the exact failure mode before writing the fix, so the fix could be verified against the same test that first caught the gap.
2. Fixed the property read, then ran the full sitemap generator's existing test suite plus the new tests above, all green.
3. Deployed to staging, regenerated the sitemap, and diffed it against the previous production sitemap.xml — confirmed every known noindex page had dropped out of the new one and no unrelated pages had disappeared.
4. Deployed to production on the next scheduled sitemap regeneration, resubmitted the sitemap in Search Console, and asked the SEO team to re-run their audit query two weeks later against the freshly re-crawled index.
5. Added a short note to the team's code review checklist: any ValueMap.get(name) call without an explicit type argument, followed immediately by .toString(), gets a second look for whether the underlying property can ever be multi-valued.

Why This Approach Held Up

The follow-up audit two weeks after resubmission showed zero noindex pages remaining in the live sitemap, and the vanity-path and template-mismatch fixes eliminated a handful of hreflang warnings that had been sitting unaddressed in Search Console for longer than anyone had connected back to the sitemap generator specifically. The most useful outcome wasn't the fix itself — it was the review-checklist addition, since the same untyped-read-then-toString pattern was found and corrected in two unrelated services during the following month's code reviews, before either one reached production.