How We Run Sitemap Audits at 137Foundry
We run sitemap indexation audits as the first check on almost every new SEO engagement. It is fast, it is quantitative, and it produces a scoping decision within the first day of the engagement. Here is the workflow, top to bottom, as we actually execute it.
Before anything else, we need three things from the client:
Read access to Google Search Console for the property
The sitemap URL (usually /sitemap.xml or /sitemap_index.xml)
Read access to the CMS or at least a screenshot of the SEO plugin configuration
The first two are the audit inputs. The third tells us whether the fixes we will recommend are executable at the CMS level or will require developer time. That distinction shapes the scope-of-work document.
Step 2: pull the sitemap and count it
Fetch the sitemap. If it is a sitemap index that references child sitemaps, recursively fetch every child. Deduplicate URLs across children (some CMSes emit the same URL in multiple children).
Break the URL list down by pattern:
Posts (usually /blog/*, /articles/*, or similar)
Pages (top-level marketing pages, service pages)
Tag or taxonomy archives (/tag/*, /category/*)
Author archives (/author/*)
Date archives (/YYYY/*, /YYYY/MM/*)
Paginated versions of any of the above (?page=2, /page/2/)
Legacy patterns (URLs from a previous site structure that are still generated)
The URL pattern breakdown is one of the most useful artifacts of the audit. It tells us at a glance where the noise is concentrated.
Step 3: pull the indexed URLs from Search Console
Search Console has a Pages report that can be filtered to a specific sitemap. Export the "All Pages" data as CSV. This gives us Google's view of what is indexed vs excluded, by URL, along with the specific exclusion reason for each.
The Google Search Central developer docs cover the API endpoints for URL inspection that can accelerate this on large sites, but the API is not necessary for most audits.
Step 4: diff sitemap against indexed
Set operations produce three URL groups:
Both (in sitemap and indexed): healthy. No action.
Sitemap only (in sitemap, not indexed): candidates for fix or removal.
Indexed only (indexed, not in sitemap): usually a canonicalization mistake or legacy URLs that are still being served.
Most of the interesting work is in the "Sitemap only" group. That is where the Search Console exclusion reason matters.
Step 5: categorize the "Sitemap only" URLs
For each URL in the "Sitemap only" set, we bucket by the Search Console exclusion reason:
"Duplicate without user-selected canonical": canonical fix needed
"Alternate page with proper canonical tag": remove from sitemap (why include a canonical target's alternate)
"Crawled - currently not indexed": Google decided the content is not worth indexing - either fix content or remove
"Discovered - currently not indexed": Google chose not to crawl - usually a crawl-budget or discovery-authority issue
"Excluded by 'noindex' tag": remove from sitemap and question why the sitemap generator includes noindexed URLs
"Blocked by robots.txt": either unblock or remove from sitemap - blocking crawl is different from blocking indexation
"Soft 404" or "Not found (404)": remove from sitemap and either restore the URL or 301 it
"Server error (5xx)": fix the server error before doing anything else
Each bucket has a different recommended action. The scope-of-work document is built from the aggregate.
Step 6: sample the largest bucket
The largest bucket almost always tells the site's story. For most WordPress-based content sites, the largest bucket is "Crawled - currently not indexed" and the URLs in it are tag archives, paginated archives, and thin auto-generated pages. For most Shopify-based e-commerce sites, the largest bucket is "Duplicate without user-selected canonical" and the URLs are variant product pages.
We open 10 URLs from the largest bucket and look at:
Actual on-page word count
Canonical tag (does it exist, does it point where we expect)
Internal link count to the URL from the rest of the site
Whether the URL is genuinely useful to a visitor
This pattern-match usually takes 15 minutes and produces a clear picture of what is going on. From there, the recommendation writes itself.
The full walkthrough of this sampling and categorization step is in the sitemap audit article we publish for teams doing their own audits.
Step 7: write the scope document
Every audit produces a scope-of-work document that lists:
URL patterns to remove from the sitemap
URL patterns to noindex at the CMS level
URL patterns to 301 redirect
URL patterns that need actual content improvement
URL patterns to leave alone
Each row includes an approximate URL count and a difficulty rating (CMS-side change vs developer change). The client gets a proposal with clear scope and estimated impact.
Step 8: implement and monitor
Implementation is either the client's team executing our scope document or we handle it under a fixed-price scope. Either way, we monitor Search Console weekly for the first month to catch any indexation drops on URLs the client actually wanted ranked.
The two metrics we track:
Indexation rate (indexed / submitted)
Absolute indexed count of "kept" URLs (URLs we did not remove)
The first should climb steadily. The second should climb slowly or stay flat.
If the first is climbing but the second is dropping, we are deindexing URLs the client actually wanted ranked. The scope document was too aggressive. Back off.
If both are flat, the CMS configuration change did not stick. Check that noindex tags are actually rendering and that robots.txt is not overriding.
If both are climbing, the audit worked.
Photo by cottonbro studio on Pexels
The audit engagement typically ends with a scheduled 90-day check-in and a delivered scope document, a fresh sitemap-diff script the client can re-run themselves, and a written playbook for what to do quarterly to prevent regressions.
Most clients see meaningful indexation and organic-traffic improvement within 6 to 10 weeks of implementation. A minority see nothing meaningful, which usually means the audit was not the highest-leverage intervention and we redirect the engagement elsewhere (content, internal linking, or backlink work).
Why this is the first thing we run
The sitemap audit is the first check because:
It is fast (one day of analysis for a mid-sized site)
It is quantitative (Search Console gives real Google data, not third-party approximations)
The scope decision is clean (either the numbers indicate leverage or they do not)
The fix is usually cheap (config-level changes, not content rewrites)
The results are measurable (indexation rate is a hard number)
Compare that to a content audit or a backlink audit - both are slower, more subjective, and produce less clearly measurable outcomes in the first six weeks. Not that content and backlinks do not matter, but the sitemap audit either produces a fast win or eliminates itself as the problem, and both outcomes are useful.
The Google Search Central documentation is the canonical reference on how sitemaps and Search Console reports work. The Ahrefs blog has good coverage of specific indexation issues and how to interpret Search Console patterns. Moz has some of the clearer general-audience writeups on technical SEO fundamentals.
The 137Foundry technical SEO service is where we run this workflow end-to-end for clients. For sites doing it themselves, the detailed audit walkthrough covers the mechanics.
Audit the sitemap. Diagnose the largest bucket. Fix at the CMS. Watch the numbers move. That is the whole workflow, and it has held up across every engagement we have run it on.