A Crawl over Crawl allows you to compare the results of a current crawl with previous crawls of the same website and in the same project. This is essential when tracking changes over time, or when looking at the impact of corrections and modifications that you have implemented on your website.

Here's how to set up a crawl over crawl and what to expect when it's finished.

What you need to be able to run a Crawl over Crawl

Crawl over Crawl is a feature of advanced projects. Before beginning, you should make sure that:

  • Your plan includes advanced projects. (Crawl over Crawl is a feature of advanced projects.)
  • The project you are going to create a crawl over crawl in has been converted to an advanced project. If this isn't the case and you have remaining advanced projects, you should convert your regular project to an advanced project now.

A Crawl over Crawl compares two crawls in the same project with the same start URL(s) and the same subdomain settings.

Therefore, to be able to compare two crawls, you need two crawls that:

  • are part of the same project.
  • have the same start URLs settings. This includes both the main start URL and any additional start URLs.
  • have identical "crawl encountered subdomains" settings. This option must be either enabled for both crawls or disabled for both crawls.

You can obtain comparable crawls in two different ways:

  1. Launch a new crawl with the Crawl over Crawl option enabled: launch a new crawl with the same start URLs and subdomain settings as an existing crawl.
  2. Run a Crawl over Crawl analysis using two existing crawls: pick two crawls with the same start URL and the same subdomain settings. If there are multiple start URLs, all of them must also match and run only a Crawl over Crawl analysis.

Option 1: Crawl over Crawl with a new crawl

For this option, you need to have selected an existing crawl you want to compare to a new crawl that you are about to set up and run.

First, verify the settings used for the existing crawl by hovering over the icon next to the crawl setting in the list of crawl reports on your project home page.

From the project home page, click "+ Setup new crawl" to set the parameters for your new crawl.

Double check that the following settings for the new crawl are the same as the ones for the existing crawl. If you use the same crawl configuration, it is likely that this will already be the case.

1 - Under "Start URL", provide the same start URL and any additional start URLs as used in your previous crawl.

2 - Under "Crawl subdomains", make sure that the option to crawl subdomains is set identically to the existing crawl.

3 - Under "Crawl over Crawl" in the "Analysis" section, tick the "Generate crawl over crawl with previous crawl" box.

This third step will run the crawl over crawl analysis to compare the two crawls when the bot has finished with the current crawl.

You can now launch your new crawl.

Once the new crawl and analysis have finished, you will have access to the Crawl over crawl report in the analysis results for both this crawl and the existing crawl.

Option 2: Crawl over Crawl with two existing crawls

If you didn't plan to compare two crawls, but their start URLs and subdomain settings are the same, you can add the crawl over crawl analysis at any later point.

Remember that you can view the settings used for any crawl in your project by hovering over the icon next to the crawl setting in the list of crawl reports on your project home page. 

From the project home page, launch a crawl over crawl:
1 - Under "Tasks", click on the "Running Crawl over Crawls" tab.

2 - Click "+ Launch crawl over crawl"
3 - Select the two crawls you would like to compare.

Note: If the start URLs or the subdomain configurations are incompatible, we'll let you know, and you won't be able to launch the crawl over crawl analysis.

When you click "+ Launch crawl over crawl", OnCrawl will analyze the differences between the two existing crawls and add the Crawl over Crawl report to the analysis results of both crawls.

You can monitor the progress of this crawl over crawl in the "Running Crawl over Crawls" tab on the project home page. Since the crawl has already been completed, the crawl over crawl will skip the "Crawling" status and begin directly by "Analyzing".

Data from a Crawl over Crawl

Once you have a crawl with an available crawl over crawl analysis, you can view the crawl over crawl report.

All crawls with an available Crawl over Crawl report will show up in the list of crawls on the project home page with a blue Crawl over Crawl symbol in the "Analysis" column:

Click on "Show analysis" to access the crawl reports. In the sidebar, click on "Crawl over Crawl" to view the different dashboards available in the Crawl over Crawl report.

If multiple crawls can be compared with the one you are looking at, you can use the "Compared with crawl" menu at the top of the page to switch the crawl you want to compare the current one with.

The charts in the Crawl over Crawl report will indicate differences between the crawls in multiple ways:

  • By providing numbers for both crawls, often accompanied by the percent of change from one crawl to the next.
  • By showing results for both crawls on the same chart.
  • By presenting an analysis of the changes between the two crawls. 
  • By comparing the breakdown from one crawl to another to show how URLs with given characteristics have been modified.

Going further

If you still have questions about running a crawl over crawl, drop us a line at @oncrawl_cs or click on the Intercom button at the bottom right of your screen to start a chat with us.

Happy crawling!

You can also find this article by searching for:
comparar dos crawls, diferencias entre dos rastreos
comparer deux crawls

Did this answer your question?