A crawl over crawl allows you to compare the results of a current crawl with previous crawls in the same project. This is essential when tracking changes over time, when comparing different versions of the same site (production/staging, mobile/desktop, different regions...) or when looking at the impact of corrections and modifications that you have implemented on your website.
Here's how to set up a crawl over crawl and what to expect when it's finished.
What you need to be able to run a crawl over crawl
The crawl over crawl is a feature of advanced projects. Before beginning, you should make sure that:
Your plan includes advanced projects: Crawl over crawl is a feature of advanced projects.
The project in which you are going to create a crawl over crawl has been converted to an advanced project. If this isn't the case and you have remaining advanced projects, you should convert your regular project to an advanced project now.
A crawl over crawl compares two crawls in the same project. Ideally, these crawls should:
Either cover the same parameters of the same site at different points in time.
This means that both crawls should have the same start URLs, subdomains, depth, etc...
Or cover the whole of one single version of the same site.
For example, if you want to compare the mobile and desktop versions of a site, you should compare a crawl of the desktop version with a crawl of the mobile version.
If the crawls you want to compare have multiple start URLs, it's possible that Oncrawl might not be able to determine how to compare your sites. If this is the case, you'll see an error message during the next step. This message will ask you to contact us for help setting up your comparison. You can reach us through the pink chat button at the bottom of any screen.
You can obtain comparable crawls in two different ways:
Launch a new crawl with the crawl over crawl option enabled: launch a new crawl with the same start URLs and subdomain settings as an existing crawl.
Run a crawl over crawl analysis using two existing crawls: pick two crawls with the same start URL and the same subdomain settings. If there are multiple start URLs, all of them must also match in order to successfully launch the crawl over crawl analysis.
Option 1: Crawl over crawl with a new crawl
For this option, you need to have selected an existing crawl to which you would like to compare the new crawl that you are about to set up and run.
1. First, verify the settings used for the existing crawl by hovering over the "information" icon next to the crawl settings in the list of crawl reports on your project home page.
2. From the project home page, click Setup a new crawl to set the parameters for your new crawl.
3. Double check that the following settings for the new crawl are the same as the ones for the existing crawl. If you use the same crawl configuration, it is likely that this will already be the case.
If you want to look at two snapshots of the same site at different times, such as before and after a modification, you will need to provide the same start URL and any additional start URLs as used in your previous crawl. This can be done in the Start URL subsection.
4. Under the Crawl over crawl subsection of the Analysis section, tick the Generate crawl over crawl with previous crawl box.
This step in the configuration process will allow the bot to run the crawl over crawl analysis. This enables you to compare the two crawls once the current crawl has been completed.
5. You can now launch your new crawl.
Once the new crawl and analysis has finished, you will have access to the crawl over crawl report in the dashboard for both the new crawl and the existing crawl.
Option 2: Crawl over crawl with two existing crawls (a comparison)
If you didn't plan to compare two crawls, but their start URLs and subdomain settings are the same, you can add the crawl over crawl analysis at a later point.
Reminder: You can view the settings used for any crawl in your project by hovering over the "information" icon next to the crawl settings, in the list of crawl reports, on your project home page.
From the project home page, launch a crawl over crawl with the following steps:
1. In the Tasks section, click on the Running crawl over crawls tab.
2. Click Set up a comparison.
3. Select the two crawls you would like to compare.
4. Click Launch crawl over crawl.
When you launch the comparison, Oncrawl will analyze the differences between the two existing crawls and add the crawl over crawl report to the analysis dashboard of both crawls.
You can monitor the progress of this crawl over crawl in the Running crawl over crawls tab on the project home page. Since the crawls have already been completed, the crawl over crawl will skip the Crawling status and jump directly to the Analyzing step.
Data from a crawl over crawl
Once you have completed a crawl with an available crawl over crawl analysis, you can view the crawl over crawl report.
All crawls with an available crawl over crawl report will show up in the list of crawls on the project home page with a grey crawl over crawl symbol in the Analysis column:
Click on Show analysis to access the crawl reports. In the sidebar, click on Crawl over crawl to view the different dashboards available in the crawl over crawl report.
If multiple crawls can be compared with the one you are looking at, you can use the Compared crawls menu at the top of the page to switch to the crawl against which you want to compare the current one.
The charts in the crawl over crawl report will indicate differences between the crawls in multiple ways:
By providing numbers for both crawls, often accompanied by the percent of change from one crawl to the next.
By showing results for both crawls on the same chart.
By presenting an analysis of the changes between the two crawls.
By comparing the breakdown from one crawl to another to show how URLs with given characteristics have been modified.
In the Data explorer
When looking at crawl over crawl data in the Data explorer, you'll see two columns per parameter: one for the results in Crawl 1 and one for the results in Crawl 2.
