Sometimes you need to run a crawl at a time when you won’t be in front of your computer. Maybe you need to crawl a site every Sunday morning.
You might also be using Oncrawl for monitoring, and have set up automated alerts if a crawl finds something unexpected, or for tracking changes. In both cases, you’ll need crawls that run automatically to make it work.
Whatever the case may be, here’s how to schedule a crawl to run and, if you want, to repeat.
Scheduling a crawl
From the project page, click Schedule crawl in the Crawl reports section. This opens the Scheduled crawls panel, which lists the crawls already scheduled for this project.
Click Create scheduled crawl, then choose:
The crawl configuration you want to use for the future crawl.
When will the crawl start?: the date, time (24-hour format) and time zone. If this is a repeated crawl, this is when the first crawl will run.
Should this crawl repeat?: tick Repeat this crawl to run it on a regular basis.
By default, a scheduled crawl does not run if Oncrawl is already crawling your website. Tick Allow simultaneous crawls if you want it to run anyway. Note that running several crawls at once multiplies the number of requests made to your website.
Finally, click Create at the top right of the screen.
Managing scheduled crawls
Your crawl is now scheduled and will run at the date and time you indicated. The next scheduled crawl and its configuration are shown on the project page, under Next scheduled crawl.
To see all scheduled crawls, click Schedule crawl in the Crawl reports section. Each scheduled crawl shows its frequency (for example, “Every day” or “Every week”), the crawl configuration used, and the next Launch date. You can filter the list by Crawl configuration.
To make changes to a scheduled crawl, click Show/Edit details.
To completely cancel a scheduled crawl, click the trash icon to the right of Show/Edit details.
Running multiple crawls at once
If, for example, you’ve chosen a crawl speed that is close to the maximum that your server can handle, there is a risk that running two crawls at once will impact your website. This happens because, if two crawls are running at the same time, Oncrawl will make twice as many requests of your web server as for one crawl.
You can choose, for each scheduled crawl, whether or not it should run when other crawls are already running.
The number of crawls you can have running at the same time also applies to scheduled crawls. If you already have crawls running when the scheduled crawl is planned to start, the crawls already underway will continue.
If, when the scheduled crawl should start, it would make too many simultaneous crawls, the scheduled crawl will not run.
An alert in the crawl monitoring box will let you know what happened.