FAQ

Rebecca Berbel avatar Erle Alberton avatar Tanguy avatar
19 articles in this collection
Written by Rebecca Berbel, Erle Alberton, and Tanguy
Crawl

How to crawl some subdomains but not others

Sometimes you'll want to crawl only certain subdomains of your site, while excluding other subdomains from the crawl. Find out how.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to crawl a site with a robots.txt crawl delay greater than 1 second

Our bot supports the Crawl Delay directive of robots.txt file. If you have validated the project you can overcome this, learn how now!
Tanguy avatar
Written by Tanguy
Updated over a week ago

How to crawl a staging/pre-prod website

It's good practice to protect staging websites and pre-production websites from bots. Here's how to crawl one before it goes live.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to quickly scan and analyze a list of URLs?

Sometimes you may need to analyze a more or less important list of URLs without browsing the entire site. Here is how to proceed.
Erle Alberton avatar
Written by Erle Alberton
Updated over a week ago

How to explore all of the URLs in my sitemap

Sometimes you just want data on the URLs in your sitemap. It's possible to do this using a list of URLs.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

What IPs does OnCrawl use to crawl a website?

OnCrawl uses a dynamic range of IP addresses to crawl websites. If necessary, it's possible to restrict this range to a series of static IPs
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to use crawl results to find redirection loops

Your crawl results can help you to quickly produce a list of all redirected pages and the pages to which they are redirected.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago