Crawl strategy & optimization
8 articles
- Crawl frequencyWhat is crawl frequency? Best practices to crawl frequency?
- Crawl ratio & unique pagesHow to optimize crawl ratio? What are unique pages?
- Crawl part of a websiteUse a virtual robots.txt in order to limit the crawl to a subset of pages on a website
- How (and why) to use a virtual robot.txt fileUse a virtual robot.txt file to tell the Oncrawl bot how to crawl your site
- How to quickly scan and analyze a list of URLs?Sometimes you may need to analyze a more or less important list of URLs without browsing the entire site. Here is how to proceed.
- How to check URLs in a sitemapHere is how to make sure that your sitemaps will be taken into account during the analysis
- Customize your user-agent in OncrawlLearn how to change the bot identity used to crawl your website.
- How to explore all of the URLs in my sitemapSometimes you just want data on the URLs in your sitemap. It's possible to do this using a list of URLs with Oncrawl.