Skip to main content
Scheduling a crawl

If you would like to launch a crawl at a specific time or you wish to create a recurring crawl, here's how.

Updated yesterday

Sometimes you need to run a crawl at a time when you won't be in front of your computer. Maybe you need to crawl a site every Sunday morning.

You might also be using Oncrawl for monitoring, and have set up automated alerts if a crawl finds something unexpected -- or for tracking changes. In both cases, you'll need crawls that run automatically to make it work.

Whatever the case may be, and no matter the reason, here's how to schedule a crawl to run and, if you want, to repeat.

Scheduling a crawl

From the project page, click on the Scheduled crawls tab at the top of the crawl monitoring box.

If you already have scheduled crawls planned, they will be listed here:

Click on Schedule crawl to plan a new crawl.
โ€‹
You will then need to choose:

  • The crawl profile you want to use for the future crawl

  • The frequency with which to repeat the crawl. If you don't want the crawl to run regularly, choose Just once.

  • The date, time (in 24-hour format) and time zone (by city) when you want the crawl to run. If this is a repeated crawl, this is the date and time when you want the first crawl to run.

Click Schedule crawl.

Managing scheduled crawls

Your crawl is now schedule and will run at the time and date you indicated.

You can keep track of crawls that are scheduled to run in the Scheduled crawls tab in the crawl monitoring box on the project's home page.

If you need to cancel a scheduled crawl, you can delete it by clicking on the trash icon.

Reminder: the number of simultaneous crawls you can have running at the same time also applies to scheduled crawls. If you already have crawls running when the scheduled crawl is planned to start, the crawls already underway will continue. If, with the schedule crawl, that makes too many simultaneous crawls, the scheduled crawl will not run.

An alert in the crawl monitoring box will let you know what happened.

Did this answer your question?