A crawl over crawl allows you to compare the results of a current crawl with previous crawls in the same project. This is essential when tracking changes over time, when comparing different versions of the same site (production/staging, mobile/desktop, different regions...) or when looking at the impact of corrections and modifications that you have implemented on your website.
Here's how to set up a crawl over crawl and what to expect when it's finished.
What you need to be able to run a crawl over crawl
A crawl over crawl compares two crawls in the same project. To get meaningful results, the two crawls should be comparable:
To compare the same site at two points in time (for example, before and after a release), both crawls should use the same settings: same start URLs, subdomain settings, depth, etc.
To compare two versions of the same site (for example, desktop vs mobile, or raw HTML vs JavaScript rendering), each crawl should cover the whole of one version of the site.
If the crawls you want to compare have multiple start URLs, it's possible that Oncrawl might not be able to determine how to compare your sites. If this is the case, you'll see an error message asking you to contact us for help setting up your comparison. You can reach us through the pink chat button at the bottom of any screen.
You can obtain comparable crawls in two different ways:
Launch a new crawl with the crawl over crawl option enabled: launch a new crawl with the same start URLs and subdomain settings as an existing crawl.
Compare two existing crawls: pick two completed crawls from your project and compare them.
Option 1: Crawl over crawl with a new crawl
For this option, you need to have selected an existing crawl to which you would like to compare the new crawl that you are about to set up and run.
1. First, verify the settings used for the existing crawl by clicking on the name of your configuration in the list of crawl reports on your project home page.
2. From the project home page, click Setup a new crawl to set the parameters for your new crawl.
3. Double check that the following settings for the new crawl are the same as the ones for the existing crawl. If you use the same crawl configuration, it is likely that this will already be the case.
If you want to look at two snapshots of the same site at different times, such as before and after a modification, you will need to provide the same start URL and any additional start URLs as used in your previous crawl. This can be done in the Start URL subsection.
4. Under the Crawl over crawl subsection of the Analysis section, tick the Generate crawl over crawl with previous crawl box.
This step in the configuration process will allow the bot to run the crawl over crawl analysis. This enables you to compare the two crawls once the current crawl has been completed.
5. You can now launch your new crawl.
Once the new crawl and analysis has finished, you will have access to the crawl over crawl report in the dashboard for both the new crawl and the existing crawl.
Note: crawl over crawls generated automatically this way are the only ones that can be used in alerts (Comparisons: Pages dataset). Comparisons created manually with option 2 do not trigger alerts. Learn more about alerts.
Option 2: Crawl over crawl with two existing crawls (a comparison)
If you didn't plan to compare two crawls when you launched them, you can compare two completed crawls at any time.
Reminder: You can view the settings used for any crawl in your project by hovering over the "information" icon next to the crawl settings, in the list of crawl reports, on your project home page.
1. On the project home page, click Crawl over crawl in the crawls section.
2. Click Set up a crawl over crawl.
3. Choose the type of comparison:
Raw HTML vs JavaScript: compares a crawl without JavaScript rendering with a crawl with JavaScript rendering, to detect how JavaScript alters your content and link discoverability, especially for LLM bots. Learn more about this lens.
Post-release SEO monitoring: compares crawls from before and after a release to detect SEO changes and regressions.
Custom config: set up your own comparison.
4. Select the two crawls you want to compare and launch the comparison.
Oncrawl will analyze the differences between the two crawls and add the crawl over crawl report to the analysis dashboard of both crawls. Since the crawls have already been completed, the crawl over crawl skips the crawling step and goes directly to the analysis.
Data from a crawl over crawl
Once you have completed a crawl with an available crawl over crawl analysis, you can view the crawl over crawl report.
All crawls with an available crawl over crawl report will show up in the list of crawls on the project home page with a grey crawl over crawl symbol in the Analysis column:
Click on Show analysis to access the crawl reports. In the sidebar, click on Crawl over crawl to view the different dashboards available in the crawl over crawl report.
If multiple crawls can be compared with the one you are looking at, you can use the Compared crawls menu at the top of the page to switch to the crawl against which you want to compare the current one.
The charts in the crawl over crawl report will indicate differences between the crawls in multiple ways:
By providing numbers for both crawls, often accompanied by the percent of change from one crawl to the next.
By showing results for both crawls on the same chart.
By presenting an analysis of the changes between the two crawls.
By comparing the breakdown from one crawl to another to show how URLs with given characteristics have been modified.
In the Data explorer
When looking at crawl over crawl data in the Data explorer, you'll see two columns per parameter: one for the results in Crawl 1 and one for the results in Crawl 2.
Data retention policy
Crawl over crawl data is kept for 30 days. After this period, the comparison is no longer available, but you can recreate it at any time by comparing the two crawls again (see Option 2).