Website Analysis
Run and Configure the Website Crawler
Configure the crawler, run a full-site audit, review the collected page data, and resolve common crawl or integration problems.
What the crawler does
The SEO Core crawler visits every page on your website and collects a detailed picture of its technical health. It follows internal links, reads your sitemap, respects robots.txt rules, and surfaces every issue that could be limiting your search visibility.
Full-site coverage
Crawls every discovered URL — not just your homepage. The crawler follows internal links across your entire domain to build a complete page inventory.
Issue detection
Identifies missing titles, broken meta tags, duplicate content, slow redirect chains, missing canonical tags, schema errors, and dozens of other technical issues.
Page structure analysis
Reviews heading hierarchy, content structure, internal link patterns, and page metadata to give you a complete picture of each URL.
Scheduled re-crawls
Set a crawl schedule so your site is audited automatically at regular intervals — keeping your issue list current without manual effort.
Running your first crawl
- 1
Create or select a project
Every crawl is associated with a project. Go to Projects and select the project for the website you want to audit, or create a new one if this is your first time.
The project domain determines which URLs the crawler considers "internal" — only pages within your domain are crawled.
- 2
Open the crawler
Select Website Analysis from the sidebar. You'll land on the crawler dashboard for your active project.
- 3
Configure crawl settings (optional)
Before starting, you can adjust the crawl configuration to match your site's needs:
- Max pages — limits how many pages are crawled in one run. Useful for very large sites.
- Max depth — controls how many link levels the crawler follows from your homepage.
- Include/exclude paths — focus the crawl on specific sections of your site, or skip areas you don't need.
- Include sitemap — automatically seeds the crawler with all URLs listed in your XML sitemap.
- Crawl speed — adjust the number of requests per second to avoid overloading your server.
- 4
Start the crawl
Click Start Crawl. The crawler begins working immediately in the background. Large sites may take several minutes to complete — you don't need to stay on the page.
Tip
For your first crawl, start with the default settings and your sitemap enabled. You can always narrow the scope in future crawls once you've seen the initial results.
- 5
Review the results
When the crawl is complete, the results are organized into sections: Issues, Pages, Structure, and Redirects.
Start with Issues — they're sorted by impact so the most important fixes appear first.
Scheduled crawls
Instead of running crawls manually, you can set a schedule so your site is audited automatically at regular intervals. This keeps your issue list fresh and alerts you to regressions — like a page losing its title tag after a CMS update.
Setting a schedule
Go to Projects → Edit (or the project settings) and enable Scheduled crawls. Choose a frequency: daily, every 3 days, or weekly. The crawler will run automatically using the same settings as your last manual crawl.
Note
Scheduled crawls count toward your plan's monthly crawl allowance. Check your plan limits if you're running frequent crawls on large sites.
What the crawler analyzes
Metadata
- Title tag — presence, length (30–60 characters), and uniqueness across pages.
- Meta description — presence, length (120–160 characters), and uniqueness.
- Canonical tag — whether one is set, and whether it points to the correct URL.
- Robots meta tag — noindex, nofollow, and other directives.
- Open Graph and Twitter card tags.
Page structure
- Heading hierarchy — H1 presence, H1 uniqueness, and heading order consistency.
- Internal links — anchor text quality, broken internal links, and orphaned pages.
- Page depth — how many clicks from the homepage each URL requires.
- Content — pages with thin or duplicate content.
Technical
- Redirect chains and loops.
- Status codes — 4xx and 5xx errors discovered during the crawl.
- Schema markup — JSON-LD presence and basic validation.
- Sitemap coverage — URLs in your sitemap vs. URLs discovered by crawling.
Deploying improvements directly from crawl results
Once you've identified an issue on a specific page — like a missing or poor-quality title tag — you can fix it directly from the crawler view without touching your CMS.
- 1
Find the page in crawler results
Filter by issue type or browse the pages list to find the URL you want to fix.
- 2
Edit the value
Click on the page row and find the field you want to change — title, description, heading, or others. Enter your improved value.
- 3
Deploy the change
Click Deploy. SEO Core will push the new value to your live site within seconds, using whichever deployment method you've set up (JavaScript snippet, WordPress plugin, or Node.js middleware).
Important
The deploy script or plugin must be installed and active on your site before you can deploy changes from the crawler. See the Deployments Overview to set this up.
Google Search Console integration
Connecting Google Search Console to your project enriches your crawler data with real-world search performance metrics — impressions, clicks, and ranking positions — for each crawled URL.
- See which crawled pages are generating impressions in Google Search.
- Identify pages with high impressions but low CTR — strong candidates for title and description improvements.
- Prioritize fixes on pages that already have search visibility.
See the Google Search Console guide for connection steps.