Video transcript

Welcome to OnCrawl's Getting Started Tutorials. Today's topic is crawl speed--or how to get fast results without bringing your server down.

Obviously, what your server can handle will vary from site to site.

It might even vary depending on the traffic on your site at different times of day. If that's the case, you may want to schedule your crawl instead of launching it manually.

To modify the crawl speed, we're going to make changes in the crawl settings for this. You can modify your default profile… or create a new one if you want to keep your current default as-is.

Use the cursor under Max crawl speed to set the crawl speed in URLs per second.

The calculator on the right lets you estimate how long the crawl will take at this speed, but of course this depends on how fast your server responds to the OnCrawl bot's requests.

The current max is 1 URL per second. If you have a small site, this is going to be a quick crawl.

For many sites, this might still be a bit much. I can switch to 1 URL every 2 seconds, which will leave you enough time to grab a coffee.

If you're sure of your server's capacity, you can also validate this project by to prove you have access to the account in order to increase the speed up to 10 URLs/second...

...which gives you a crawl time of about a minute.

OnCrawl can go faster--but most servers won't support so many consecutive requests--so we ask you to contact us if you need more speed.

Don't forget to save your settings!

Questions about crawl speed? Reach out to us from the OnCrawl interface by clicking on the blue chat button we just saw, or tweet to us at OnCrawl_CS.

See you next time!

Until then, happy crawling.
.

Note:

You can modify the maximum crawl speed while the crawl is running. To do so, go to the Crawl Monitoring and click on the blue "Pause crawl" drop-down button at the top right of the screen. Choose "Change max depth."