Setting up your crawl
7 articles
Oncrawl LensesPredefined entry points tailored to your SEO use cases.
Choosing your crawl mode: Spider mode vs. a URL list vs. Sitemaps modeUnderstand the different ways to crawl your site's pages with Oncrawl.
How to crawl some subdomains but not othersSometimes you'll want to crawl only certain subdomains of your site, while excluding other subdomains from the crawl. Find out how.
How to crawl a staging/pre-prod websiteIt's good practice to protect staging websites and pre-production websites from bots. Here's how to crawl one before it goes live.
Crawl settingsSet crawl parameters, save crawl settings as a profile to reuse later, create cross-data reports, and launch your crawl.
How to define rules of inclusion/exclusion for URL patterns in your crawl setupHow to tell the crawler what part of your site to analyze by including/excluding groups of URLs
Optimize Content Analysis with the Exclude Content FeatureThis feature allows you to filter out dense, redundant, content from your analysis, enabling you to focus on what truly matters