When you launch a crawl in Oncrawl, you set crawl limits including:
Max crawl speed: you tell the Oncrawl bot how many URLs to inspect in a period of time.
Max crawl depth: you tell Oncrawl how many levels of links, or steps away from your Start URL the bot is allowed to explore.
After launching your crawl, you might want to modify these limits. This might occur for any number of reasons:
You realize after the fact that the settings were not what you intended.
Your site is large enough for the crawl to take several hours; you've now entered a low-traffic (or high-traffic) period on your site and you want to adjust the crawl to respect the available resources.
You're monitoring the crawl and you see a lot of 5xx errors: you may have over-estimated your server's capacity.
You're monitoring your crawl and you see a lot of unexpected 301s or 404s that you want to correct before crawling the whole site.
You're monitoring your crawl and you realize a deeper or a more shallow exploration is necessary as a first step.
How to modify crawl limits while crawling
From the project home page, open the crawl monitoring page for the crawl you want to modify. (If you've just launched the crawl, you're already on the monitoring page.)
Click on the drop-down menu next to the Pause button in the upper right.
Choose the limit you want to modify:
Change max speed
Change max depth
The changes can take a moment to take effect.
This does not modify the crawl profile you used for the crawl. Only the current crawl is affected.
How to end a crawl early
You can also stop a crawl that has not finished and analyze only the URLs that have already been inspected. This is different from aborting the crawl or pausing it.
Stop a crawl and start analysis: stops crawling and moves on to the Analyzing Data step in order to generate results.
Abort a crawl: cancel the crawl and do not keep or analyze any results.
Pause a crawl: stop crawling now, but pick up where you left off at a later time. (No results are generated until the crawl is unpaused and finished.)
From the crawl monitoring page, click on the drop-down menu next to the Pause button in the upper right.
Choose Generate crawl result now.
