What are monthly crawl tokens?
Oncrawl plans function on the basis of monthly crawl quotas that contain a fixed number of tokens.
When you crawl a URL, one or more tokens are consumed from this quota, depending on the crawl actions you want to do for the URL.
What happens if you don't have enough tokens?
When you've consumed all of the tokens in your quota, the current crawls will end. If there are no tokens left in your quota, you won't be able to launch a crawl until your tokens are renewed.
As long as your subscription is active, the tokens in your monthly crawl quota are renewed every month.
How much does a crawl "cost"?
Crawling a website to analyze it consumes tokens from your crawl quota in the following way:
Crawl actions for each URL | URLs crawled | Tokens consumed |
Crawl ("fetch") the URL content only | 1 URL | 1 URL from your monthly quota |
Crawl ("fetch") the URL | 1 URL | 3 URL from your monthly quota |
Crawl ("fetch") the URL | 1 URL | 5 URL from your monthly quota |
For example, if you have a website with 50 000 pages, one crawl that renders JS and analyzes Core Web Vitals of your entire website will use up 250 000 tokens from your quota.
What happens if you use a URL limit in your crawl settings?
If you set a URL limit in your crawl settings, this will stop the crawl when you reach that number of URLs.
This will not limit the number of tokens used for the crawl to the number of URLs.
For example, if you have a website with 50 000 pages and you want to spend less than 100 000 tokens in this month's crawl quota, but you want to run a crawl that also renders JS, you'll need to set your crawl limit to 33 000 URLs:
33 000 URLs crawled x 3 (tokens to crawl + render JS) = 99 000 tokens from your quota