Crawl strategy & optimization
8 articles
Crawl frequencyWhat is crawl frequency? Best practices to crawl frequency?
Crawl ratio & unique pagesHow to optimize crawl ratio? What are unique pages?
Crawl part of a websiteUse a virtual robots.txt in order to limit the crawl to a subset of pages on a website
How (and why) to use a virtual robot.txt fileUse a virtual robot.txt file to tell the Oncrawl bot how to crawl your site
How to quickly scan and analyze a list of URLs?Sometimes you may need to analyze a more or less important list of URLs without browsing the entire site. Here is how to proceed.
How to check URLs in a sitemapHere is how to make sure that your sitemaps will be taken into account during the analysis
Customize your user-agent in OncrawlLearn how to change the bot identity used to crawl your website.
How to explore all of the URLs in my sitemapSometimes you just want data on the URLs in your sitemap. It's possible to do this using a list of URLs with Oncrawl.