Scheduling a crawl

If you would like to launch a crawl at a specific time or you wish to create a recurring crawl, here's how.

Updated over a week ago

Sometimes you need to run a crawl at a time when you won't be in front of your computer. Maybe you need to crawl a site every Sunday morning. Whatever the case may be, and no matter the reason, here's how to do it.

Scheduling a crawl

From the project page, click on the Scheduled crawls tab at the top of the crawl monitoring box.

If you already have scheduled crawls planned, they will be listed here:

Click on Schedule crawl to plan a new crawl.
โ€‹
You will then need to choose:

  • The crawl profile you want to use for the future crawl

  • The frequency with which to repeat the crawl. If you don't want the crawl to run regularly, choose Just once.

  • The date, time (in 24-hour format) and time zone (by city) when you want the crawl to run. If this is a repeated crawl, this is the date and time when you want the first crawl to run.

Click Schedule crawl.

Managing scheduled crawls

Your crawl is now schedule and will run at the time and date you indicated.

You can keep track of crawls that are scheduled to run in the Scheduled crawls tab in the crawl monitoring box on the project's home page.

If you need to cancel a scheduled crawl, you can delete it by clicking on the trash icon.

Reminder: your quota for simultaneous crawls applies to scheduled crawls, too. If your quota is used on crawls that are already running when the scheduled crawl is planned to start, they will continue. The scheduled crawl will not run.

An alert in the crawl monitoring box will let you know what happened.

Did this answer your question?