Oncrawl's crawler is designed to crawl real websites. We provide modifiable crawl parameters that will help adapt our crawler to your specific case.
Here's how to set up a crawl. We'll cover the following points:
Where to find the crawl settings
From your project page, select a crawl configuration in the Crawler menu, then:
click Edit configuration to modify it, or
click Create configuration to create a new one.
The crawl settings are organized in four steps: Configuration, Analysis, Automate and Summary. Use Next step and Previous step to move between them, and Save to save your changes.
Crawl configurations
A crawl configuration is a set of settings saved with a name, so you can reuse it for later crawls. Each crawl in the project list shows the name of the configuration used.
When you click Create configuration, you can start from a predefined configuration tailored to a use case, or from scratch:
Crawl Discovery: website audit in spider mode, monthly crawl, JS rendering.
Content: content audit with AI analysis.
Sanity Check: quick checks on a list of key URLs, daily crawl, alerting.
Performance: Core Web Vitals and load times on a list of URLs, daily crawl.
Accessibility: automated accessibility checks mapped to WCAG 2.2.
Google Discover: in-depth Google Discover audit using Google Search Console data.
Custom configuration: a new crawl with no predefined settings. Use Copy config from to base it on an existing configuration; you can change all of the parameters afterwards.
Learn more about these predefined configurations in Oncrawl Lenses.
A good name might, for example, state what is being crawled, or the type of additional data included:
A "Rankings" crawl that includes GSC data
A "Blog without parameters" crawl of https://blog.example.com that excludes query strings
A "Daily sanity check" that checks key pages every day
A "Monthly full crawl (JS)" that crawls the full site every month with JS enabled
Step 1: Configuration
This step sets the crawl's technical boundaries: speed, depth, and the scope of your site to explore.
Crawler settings
Start URL
Choose how the crawler discovers your URLs:
Spider mode: the crawler starts from the start URL and follows all followed links it encounters, until it reaches the maximum number of URLs or the maximum depth. You can add Alternate start URLs.
List of URLs: the crawler examines the URLs in a list you provide, but does not follow the links it finds on those pages.
Sitemap mode: Oncrawl uses the URLs listed in your sitemaps to build the list of URLs to crawl.
Crawl limits
In spider mode, the crawl limits tell the crawler when to stop:
Max URLs: the crawler stops once it has discovered this number of URLs, unless it has reached another limit first.
Max depth: the start URL(s) are depth 1, pages linked from them are depth 2, and so on. With a max depth of 3, the crawler stops after discovering all pages at depth 3.
Your remaining monthly token quota is displayed in this section.
You can also modify the maximum depth while the crawl is running: go to Crawl Monitoring, click the Pause crawl drop-down and choose Change max depth.
Crawl JS
Enable JavaScript crawling (x3 crawl tokens): renders pages before analysis, so Oncrawl can discover content and links loaded by JavaScript.
Enable Shadow DOM: includes Shadow DOM content when the site exposes it in open mode. Closed Shadow DOM content cannot be accessed.
JavaScript Audits (x5 crawl tokens in total): Run Core Web Vitals analysis and Run Accessibility audit. JavaScript crawling is required to run JavaScript audits.
Crawl bot
You need to verify the domain in the Verify domain ownership section of the workspace settings if you want to use a full custom user-agent.
Crawl as Desktop bot or Crawl as Mobile bot: the default Oncrawl user-agents. You can change the Bot name.
Crawl with custom user-agent: provide a Bot name and a Full user agent string. This lets you see how your site renders for a given user-agent, or whitelist a unique user-agent approved by your IT team.
Max crawl speed
A crawl speed higher than 1 URL per second requires you to verify ownership of the domains in your start URLs.
Set the number of URLs crawled per second. Enter an estimated number of URLs to see the approximate crawl time. The crawl may be slower if the site does not respond fast enough.
The ideal speed is the speed your server can handle best. You can also change the max speed while the crawl is running: go to Crawl Monitoring, click the Pause crawl drop-down and choose Change max speed.
Crawler scope
Subdomains: by default, subdomains encountered during the crawl (such as https://store.example.com when crawling https://www.example.com) are treated as a different site. Tick Crawl encountered subdomains to follow them.
URL pattern filtering: tick Enable these filters to use regex rules to include or exclude URLs. Pages excluded by these filters are not crawled and do not appear in the crawl results.
Crawler behavior
Follow links
Follow HTTP redirects (3xx): follows and analyzes redirect destinations, including redirect chains.
Follow alternates
Follow canonicals
Ignore nofollow: links with nofollow are treated as followed links. The Data Explorer column "Processing warning" shows which links were modified.
Ignore noindex
Ignore links from 4xx and 5xx pages: links found on pages returning a 4xx or 5xx status are not included in your internal linking data.
URL parameters and normalization
Normalize URL encoding: normalizes encoded and decoded URLs to reduce duplicates.
Rewrite // to /: normalizes repeated slashes. Disable it to identify malformed URLs.
Crawl URLs with parameters: crawls URLs with query strings, such as example.com?q=search.
Reorder query params: normalizes the order of query parameters to reduce quota usage and false duplicates.
Filter URL parameters: include or exclude specific parameter names, such as utm_source, utm_medium, utm_term.
Note that Oncrawl (and Google) count each URL with different parameters as a separate URL.
Virtual robots.txt
You will need to verify the domain in the Verify domain ownership section of the workspace settings before you can override a website's robots.txt.
Tick Enable virtual robots.txt to provide a temporary robots.txt that applies to the Oncrawl bot only, for example to crawl blocked pages, crawl only part of the site, or crawl faster than the crawl delay. Your actual robots.txt is not modified. Learn more about virtual robots.txt files.
DNS override
Assign a different IP to a given host, for example to crawl a site hosted on a pre-production server that uses your usual domain name. DNS overrides are not compatible with JS crawling.
HTTP headers
Add HTTP headers to the crawler's requests, one per line:
Header-Name: Header-Value
To set multiple cookies, use the following format:
Cookie: Cookie1_Name=Cookie1_Value; Cookie2_Name=Cookie2_Value
Authentication
Tick Enable HTTP authentication to provide a username and password, for example to crawl a password-protected pre-production site. Basic, Digest and NTLM are supported; you can also indicate the realm (blog, wiki, cms, admin…).
Step 2: Analysis
This step enriches your crawl with additional analyses and external data sources.
Content analysis:
Exclude elements: exclude boilerplate content (header, footer, navigation, cookie banners…) from content metrics such as word count and similarity, using CSS selectors (tags, ids, classes). Link discovery is not affected.
Link position:
Define CSS selectors for the Header, Footer, Navigation and Aside zones. Links outside these zones are classified as Main. Use Test link position to check your selectors.
Inrank settings:
Tick Use reasonable-surfer weighting to weight internal links by their position on the page, instead of giving all links the same weight.
Content Lens:
AI analysis of content quality. Each analyzed URL consumes Content Lens tokens, a quota separate from the crawl quota. You can limit the number of URLs or the depth analyzed, and include/exclude URLs with regex. Learn more.
Sitemaps:
Enable sitemaps cross-data analysis, Specify sitemaps URLs (otherwise, sitemaps declared in robots.txt or with a standard name are used), and Allow soft mode to analyze sitemaps regardless of the sitemaps protocol rules. Learn more about checking URLs in sitemaps.
Connectors:
Add data from Logs, Google Search Console, Google Discover, Google Analytics, Piano and more.
Cross Analysis Range:
Number of days of data used from connected sources (45 by default). Google Search Console data has a three-day delay.
Crawl over crawl:
Tick Generate crawl over crawl with previous crawl to compare this crawl with a previous one with the same start URL. Learn more about crawl over crawl.
Scraping:
Tick Enable scraping to build custom fields with regex or XPath extraction rules. Learn more.
Data ingestion:
Tick Enable data ingestion to add fields from a third-party CSV file to each URL. Learn more.
Step 3: Automate
Notifications
Add up to 1,000 email addresses to notify when a crawl ends or encounters a problem. We recommend notifying members of the workspace. The email states which crawl has ended and the reason for the end of the crawl.
Webhook
A webhook works like an HTTP callback link. When a crawl ends, Oncrawl sends an HTTP POST request to the URLs defined here, which you can use to trigger further actions, such as alerts via the Oncrawl API or Big Data exports. We recommend using a secret to verify the authenticity of the received payload.
The data provided contains information related to Workspace_Id and Project_Id, as well as the reason for the end of the crawl:
Coded reason | UI label | Email label |
| Monthly URL quota reached. | The crawl ended because it reached the maximum number of URLs in the monthly quota. |
| Configured maximum number of URLs reached. | The crawl ended because it reached the maximum number of URLs that was set in the crawl profile. |
| Configured maximum depth reached. | The crawl ended because it reached the maximum crawl depth that was set in the crawl profile. |
| Crawl canceled by workspace member. | The crawl was cancelled by a member of the workspace. |
| Crawl ended early by workspace member. | This crawl was ended early by a member of the workspace. |
| No URLs fetched. | The crawler could not retrieve any URLs. |
Oncrawl also provides reasons in the case of scheduled crawls that couldn’t start:
Coded reason | UI label | Email label |
| Cannot start a crawl (crawl profile contains an invalid element). | The scheduled crawl's profile contains an invalid element. |
| A crawl with the same profile was already running. | The scheduled crawl did not launch because another crawl with the same profile was already running. |
| Too many crawls currently running. | The scheduled crawl did not launch because other crawls were running at the same time. The workspace has reached its quota for concurrent crawls. |
| Cannot start a crawl with Google Analytics (feature is not available). | The scheduled crawl did not launch because it includes the Google Analytics feature that is not available in the workspace subscription. |
| Cannot start a crawl with Majestic (feature is not available). | The scheduled crawl did not launch because it includes the Majestic feature that is not available in the workspace subscription. |
| Cannot start a crawl with scraping (feature is not available). | The scheduled crawl did not launch because it includes the scraping feature that is not available in the workspace subscription. |
You can test whether your webhook URL works using a provided API endpoint.
Export to
Tick Enable export to Data Studio to export your metrics to Looker Studio (formerly Google Data Studio) at the end of the crawl.
Step 4: Summary
Review all the settings of your configuration, then click Launch crawl.