How To Find All Existing And Archived Urls On A Website

There are galore reasons you mightiness petition to find each nan URLs connected a website, but your nonstop extremity will find what you’re searching for. For instance, you whitethorn want to:

• Identify every indexed URL to analyse issues for illustration cannibalization aliases standard bloat
• Collect current and humanities URLs Google has seen, peculiarly for tract migrations
• Find all 404 URLs to retrieve from post-migration errors

In each scenario, a azygous instrumentality won’t springiness you everything you need. Unfortunately, Google Search Console isn’t exhaustive, and a “site:example.com” hunt is constricted and difficult to extract accusation from.

In this post, I’ll locomotion you done immoderate devices to build your URL database and earlier deduplicating nan accusation utilizing a spreadsheet aliases Jupyter Notebook, depending connected your website’s size.

Archive.org is an invaluable instrumentality for SEO tasks, funded by donations. If you hunt for a domain and premier nan “URLs” option, you tin entree up to 10,000 listed URLs.

However, location are a less limitations:

• URL limit: You tin only retrieve up to 10,000 URLs, which is insufficient for larger sites.
• Quality: Many URLs whitethorn beryllium malformed aliases reference assets files (e.g., images aliases scripts).
• No export option: There isn’t a built-in measurement to export nan list.

To bypass nan deficiency of an export button, usage a browser scraping plugin like Dataminer.io. However, these limitations mean Archive.org whitethorn not proviso a complete solution for larger sites. Also, Archive.org doesn’t bespeak whether Google indexed a URL—but if Archive.org recovered it, there’s a bully chance Google did, too.

How to usage it:
Export your inbound links in Moz Pro to get a speedy and easy database of target URLs from your site. If you’re dealing pinch a monolithic website, spot utilizing the Moz API to export accusation beyond what’s manageable successful Excel aliases Google Sheets.

It’s important to connection that Moz Pro doesn’t corroborate if URLs are indexed aliases discovered by Google. However, since astir sites usage nan same robots.txt rules to Moz’s bots arsenic they do to Google’s, this method mostly useful bully arsenic a proxy for Googlebot’s discoverability.

Similar to Moz Pro, nan Links conception provides exportable lists of target URLs. Unfortunately, these exports are capped at 1,000 URLs each. You tin usage filters for circumstantial pages, but since filters don’t usage to nan export, you mightiness petition to spot connected browser scraping tools—limited to 500 filtered URLs astatine a time. Not ideal.

Performance → Search Results:

This export gives you a database of pages receiving hunt impressions. While nan export is limited, you tin use Google Search Console API for larger datasets. There are too free Google Sheets plugins that simplify pulling overmuch extended data.

Indexing → Pages report:

Even better, you tin usage filters to create different URL lists, efficaciously surpassing nan 100k limit. For example, if you want to export only blog URLs, recreation these steps:

Step 1: Add a conception to nan report

Step 2: Click “Create a caller segment.”

Server aliases CDN log files are perchance nan eventual instrumentality astatine your disposal. These logs seizure an exhaustive database of each URL measurement queried by users, Googlebot, aliases different bots during nan recorded period.

Considerations:

• Data size: Log files tin beryllium massive, truthful galore sites only clasp nan past 2 weeks of data.
• Complexity: Analyzing log files tin beryllium challenging, but various devices are disposable to simplify nan process.

Once you’ve gathered URLs from each these sources, it’s clip to harvester them. If your tract is mini enough, usage Excel or, for larger datasets, devices for illustration Google Sheets aliases Jupyter Notebook. Ensure each URLs are consistently formatted, past deduplicate nan list.

And voilà—you now personification a wide database of current, old, and archived URLs. Good luck!