Crawl setup
8 articles
Scheduling a crawlIf you would like to launch a crawl at a specific time or you wish to create a recurring crawl, here's how.
Crawl settingsSet crawl parameters, save crawl settings as a profile to reuse later, create cross-data reports, and launch your crawl.
Choosing your crawl mode: Spider mode vs. a URL list vs. Sitemaps modeUnderstand the different ways to crawl your site's pages with Oncrawl.
How (and why) to use a virtual robot.txt fileUse a virtual robot.txt file to tell the Oncrawl bot how to crawl your site
What IPs does Oncrawl use to crawl a website?Secure access by greenlisting Oncrawl's static IP addresses, ensuring smooth and uninterrupted website crawling.
Crawl part of a websiteUse a virtual robots.txt in order to limit the crawl to a subset of pages on a website
How to crawl some subdomains but not othersSometimes you'll want to crawl only certain subdomains of your site, while excluding other subdomains from the crawl. Find out how.
How to define rules of inclusion/exclusion for URL patterns in your crawl setupHow to tell the crawler what part of your site to analyze by including/excluding groups of URLs