Site Crawler — Bulk SEO Audit Tool

Crawl Mode

Spider follows links automatically; List/Sitemap crawl exactly the URLs you give it (optionally expanding further).

Start URL

The crawler starts here and follows internal links found on each page.

Max Pages

Crawl stops once this many pages are fetched.

Include / Exclude URL Filters

Include only URLs matching (optional)

One pattern per line. Plain text = substring match, * = wildcard, or wrap in /…/ for regex. Leave blank to include everything in scope.

Exclude URLs matching (optional)

Any URL matching one of these is skipped entirely — checked before Include.

Advanced crawl settings (scope, depth, speed, robots.txt)

Max Depth

Concurrent Requests

Delay Between Requests (ms)

Crawl Scope

Crawl As (User-Agent)

Also crawl links found on these pages
In List/Sitemap mode, follow internal links discovered on each fetched page too

Respect robots.txt
Skip URLs disallowed for * user-agent (recommended)

Treat URLs with different query strings as duplicates
e.g. /page?sort=asc and /page?sort=desc count as one URL

Check external links for broken status (not crawled, just verified)
Adds a HEAD request per unique external link found

Check images for broken links
Adds a lightweight HEAD request per image (up to 15/page, 300 total) — slower on image-heavy sites

Retry failed / 429 / 5xx requests
Retries up to 2 times with backoff (honours Retry-After when the server sends it)

Thin Content Threshold (words)

Pages with fewer words than this are flagged "Thin Content".

Slow Page Threshold (ms)

Pages slower than this are flagged "Slow".

Target Keyword (optional)

Checked in title, H1, URL, meta description, first 100 words.

Sitemap URL for Orphan-Page Check (optional)

After the crawl, compares this sitemap against the pages actually reachable by internal links and lists any sitemap URLs that were never linked to (orphans).

Pulls a value from a matching element on every crawled page — e.g. price, SKU, author. Leave the selector blank to skip a row. Results appear in each row's detail view, and as extra columns in the CSV/XLSX export.

Flags whether each page's raw HTML contains a given string or regex — e.g. find pages missing a GA tag, or containing old branding. After the crawl: filter results using the "Found"/"Missing" chips above the table, click any row to see the match per rule, or check the CSV/XLSX export.

Regex

Import Crawl (.json)

Crawling… 0%

Initializing…

Crawl History (saved in this browser, last 20 crawls)

Compare Two Crawls

Load two JSON exports from previous crawls (e.g. before/after a migration or redesign) to see what changed — new pages, removed pages, status code changes, title changes, and redirect changes. Works independently of the crawler above.

Crawl A (baseline / older)

No file loaded.

Crawl B (newer / current)

No file loaded.

What This Crawler Checks

The Site Crawler walks a website in Spider, List or Sitemap mode, up to your configured page limit and depth. For every page it captures the response status code, full redirect chain, title and length, meta description and length, H1 count, word count, canonical tag, meta robots / X-Robots-Tag directives, hreflang tags, JSON-LD structured data types, pagination rel=prev/next, viewport tag, and internal/external/inlink counts — the same core signals Screaming Frog surfaces in its overview crawl, plus a few it charges for.

🕷️ Spider / List / Sitemap Modes

Follow links automatically, paste an exact URL list, or seed the crawl straight from a sitemap.xml (including sitemap indexes).

🔀 Full Redirect Chains

Every hop of a redirect is followed manually and logged, not just the final destination — so 3-hop redirect chains are visible, not hidden.

🏷️ Titles & Metas

Flags missing, duplicate, too-short or too-long titles and meta descriptions, plus an estimated SERP pixel width for both.

🧬 Duplicate & Thin Content

Hashes each page's visible text to catch exact-duplicate content, and flags pages under a configurable word-count threshold.

🌍 Hreflang & Structured Data

Extracts every hreflang alternate and every JSON-LD @type on the page for an at-a-glance international-SEO and schema inventory.

🕵️ Orphan Page Detection

Compares a sitemap.xml against the pages actually reachable by internal links, and lists sitemap URLs no page links to.

🔗 Inlinks & Site Structure

Tracks which pages link to each URL ("found on") and renders a collapsible folder-style tree of the whole crawled site.

🧩 Custom Extraction & Search

Pull a CSS-selected value (price, SKU, author…) from every page, or flag pages containing/missing a given string or regex.

🤖 robots.txt & Selectable User-Agent

Optionally respects Disallow rules for the crawling user-agent, which can be switched to Googlebot, Bingbot, mobile Chrome or a custom string.

♻️ Retry & Backoff

Automatically retries failed, 429 or 5xx requests with backoff, honouring a server's Retry-After header when present.

🎯 Include / Exclude Filters

Restrict the crawl to a subfolder, skip parameterised or admin URLs, or focus on a specific content type using substring, wildcard or regex patterns.

📥 CSV / XLSX / JSON / Sitemap Export

Export a multi-tab XLSX workbook (overview, pages, redirects, duplicates, images, schema, hreflang), a re-importable JSON snapshot, or a fresh sitemap.xml built from the crawl's healthy indexable pages.

Good to Know

This is a browser-based crawler: it reads raw server-rendered HTML through crawler-proxy.php and does not execute JavaScript, so content injected client-side (heavy SPA frameworks) won't be seen — same limitation as curl-based crawlers. Large sites should narrow scope with Include/Exclude filters or a lower Max Pages to keep crawl time reasonable, since pages are fetched sequentially in small concurrent batches to stay polite to the target server. Custom extraction and search rules run against the raw HTML/DOM of each page, not a rendered browser view.