Crawl over Crawl

A Crawl over Crawl compares the results of two crawls to show you what the impact of your changes is or how two sites differ.

Updated over a week ago

A Crawl over Crawl allows you to compare the results of a current crawl with previous crawls in the same project. This is essential when tracking changes over time, when comparing different versions of the same site (production/staging, mobile/desktop, different regions...) or when looking at the impact of corrections and modifications that you have implemented on your website.

Here's how to set up a crawl over crawl and what to expect when it's finished.

What you need to be able to run a Crawl over Crawl

Crawl over Crawl is a feature of advanced projects. Before beginning, you should make sure that:

  • Your plan includes advanced projects. (Crawl over Crawl is a feature of advanced projects.)

  • The project you are going to create a crawl over crawl in has been converted to an advanced project. If this isn't the case and you have remaining advanced projects, you should convert your regular project to an advanced project now.

A Crawl over Crawl compares two crawls in the same project. Ideally, these crawls should:

  • Either cover the whole of the same site.
    (This means that both crawls should have the same Start URLs, subdomains, depth...)

  • Or cover the whole of one single version of the same site.
    (For example, if you want to compare the mobile and desktop versions of a site, you should compare a crawl of the desktop version only with a crawl of the mobile version only.)

If the crawls you want to compare have multiple start URLs, it's possible that OnCrawl might not be able to determine how to compare your sites. If this is the case, you'll see an error message during the next step. This message will asks you to contact us for help setting up your comparison. You can reach us through the blue chat button at the bottom of any screen. 

You can obtain comparable crawls in two different ways:

  1. Launch a new crawl with the Crawl over Crawl option enabled: launch a new crawl with the same start URLs and subdomain settings as an existing crawl.

  2. Run a Crawl over Crawl analysis using two existing crawls: pick two crawls with the same start URL and the same subdomain settings. If there are multiple start URLs, all of them must also match and run only a Crawl over Crawl analysis.

Option 1: Crawl over Crawl with a new crawl

For this option, you need to have selected an existing crawl you want to compare to a new crawl that you are about to set up and run.

First, verify the settings used for the existing crawl by hovering over the icon next to the crawl setting in the list of crawl reports on your project home page.

From the project home page, click "+ Setup new crawl" to set the parameters for your new crawl.

Double check that the following settings for the new crawl are the same as the ones for the existing crawl. If you use the same crawl configuration, it is likely that this will already be the case.

1 - If you want to look at two snapshots of the same site at different times, such as before and after a modification, you will need to provide the same start URL and any additional start URLs as used in your previous crawl. This can be done under "Start URL".

2 - Under "Crawl over Crawl" in the "Analysis" section, tick the "Generate crawl over crawl with previous crawl" box.

This third step will run the crawl over crawl analysis to compare the two crawls when the bot has finished with the current crawl.

You can now launch your new crawl.

Once the new crawl and analysis have finished, you will have access to the Crawl over crawl report in the analysis results for both this crawl and the existing crawl.

Option 2: Crawl over Crawl with two existing crawls

If you didn't plan to compare two crawls, but their start URLs and subdomain settings are the same, you can add the crawl over crawl analysis at any later point.

Remember that you can view the settings used for any crawl in your project by hovering over the icon next to the crawl setting in the list of crawl reports on your project home page. 

From the project home page, launch a crawl over crawl:
1 - Under "Tasks", click on the "Running Crawl over Crawls" tab.

2 - Click "+ Launch crawl over crawl"
3 - Select the two crawls you would like to compare.

When you click "+ Launch crawl over crawl", OnCrawl will analyze the differences between the two existing crawls and add the Crawl over Crawl report to the analysis results of both crawls.

You can monitor the progress of this crawl over crawl in the "Running Crawl over Crawls" tab on the project home page. Since the crawl has already been completed, the crawl over crawl will skip the "Crawling" status and begin directly by "Analyzing".

Data from a Crawl over Crawl

Once you have a crawl with an available crawl over crawl analysis, you can view the crawl over crawl report.

All crawls with an available Crawl over Crawl report will show up in the list of crawls on the project home page with a blue Crawl over Crawl symbol in the "Analysis" column:

Click on "Show analysis" to access the crawl reports. In the sidebar, click on "Crawl over Crawl" to view the different dashboards available in the Crawl over Crawl report.

If multiple crawls can be compared with the one you are looking at, you can use the "Compared with crawl" menu at the top of the page to switch the crawl you want to compare the current one with.

The charts in the Crawl over Crawl report will indicate differences between the crawls in multiple ways:

  • By providing numbers for both crawls, often accompanied by the percent of change from one crawl to the next.

  • By showing results for both crawls on the same chart.

  • By presenting an analysis of the changes between the two crawls. 

  • By comparing the breakdown from one crawl to another to show how URLs with given characteristics have been modified.

In the Data Explorer

When looking at Crawl over Crawl data in the Data Explorer, you'll see two URL columns: one for the URLs in Crawl 1 and one for the URLs in Crawl 2. If you've compared two versions of the same site, these URLs may be different.

Going further

If you still have questions about running a crawl over crawl, drop us a line at @oncrawl_cs or click on the Intercom button at the bottom right of your screen to start a chat with us.

Happy crawling!

You can also find this article by searching for:
comparar dos crawls, diferencias entre dos rastreos
comparer deux crawls

Did this answer your question?