All Collections
General information
How to modify crawl limits while crawling
How to modify crawl limits while crawling
How to change crawl speed and max crawl depth after launching your crawl
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

When you launch a crawl in OnCrawl, you set crawl limits including:

  • Max crawl speed: you tell the OnCrawl bot how many URLs to inspect in a period of time. 

  • Max crawl depth: you tell OnCrawl how many levels of links, or steps away from your Start URL the bot is allowed to explore.

After launching your crawl, you might want to modify these limits. This might occur for any number of reasons:

  • You realize after the fact that the settings were not what you intended.

  • Your site is large enough for the crawl to take several hours; you've now entered a low-traffic (or high-traffic) period on your site and you want to adjust the crawl to respect the available resources.

  • You're monitoring the crawl and you see a lot of 5xx errors: you may have over-estimated your server's capacity.

  • You're monitoring your crawl and you see a lot of unexpected 301s or 404s that you want to correct before crawling the whole site.

  • You're monitoring your crawl and you realize a deeper or a more shallow exploration is necessary as a first step.

How to modify crawl limits while crawling

From the project home page, open the crawl monitoring page for the crawl you want to modify. (If you've just launched the crawl, you're already on the monitoring page.)

Click on the drop-down menu next to the "Pause" button in the upper right.

Choose the limit you want to modify:

  • Change max speed

  • Change max depth

The changes can take a moment to take effect.

This does not modify the crawl profile you used for the crawl. Only the current crawl is affected.

How to end a crawl early

You can also stop a crawl that has not finished and analyze only the URLs that have already been inspected. This is different from aborting the crawl or pausing it.

  • Stop a crawl and start analysis: stops crawling and moves on to the Analyzing Data step in order to generate results.

  • Abort a crawl: cancel the crawl and do not keep or analyze any results.

  • Pause a crawl: stop crawling now, but pick up where you left off at a later time. (No results are generated until the crawl is unpaused and finished.)

From the crawl monitoring page, click on the drop-down menu next to the "Pause" button in the upper right.

Choose "Generate crawl result now".

Going further

If you still have questions about comparing sessions in Google Analytics with SEO visits in OnCrawl, drop us a line at @oncrawl_cs or click on the Intercom button at the bottom right of your screen to start a chat with us.

Happy crawling!

You can also find this article by searching for:
cambiar velocidad de rastreo, cambiar profundidad, finalizar un rastreo, terminar temprano
changer la vitesse de crawl, changer la profondeur, terminer un crawl tôt

Did this answer your question?