Setting up your crawl
7 articles
- Oncrawl LensesPredefined entry points tailored to your SEO use cases.
- Choosing your crawl mode: Spider mode vs. a URL list vs. Sitemaps modeUnderstand the different ways to crawl your site's pages with Oncrawl.
- How to crawl some subdomains but not othersSometimes you'll want to crawl only certain subdomains of your site, while excluding other subdomains from the crawl. Find out how.
- How to crawl a staging/pre-prod websiteIt's good practice to protect staging websites and pre-production websites from bots. Here's how to crawl one before it goes live.
- Crawl settingsSet crawl parameters, save crawl settings as a profile to reuse later, create cross-data reports, and launch your crawl.
- How to define rules of inclusion/exclusion for URL patterns in your crawl setupHow to tell the crawler what part of your site to analyze by including/excluding groups of URLs
- Optimize Content Analysis with the Exclude Content FeatureThis feature allows you to filter out dense, redundant, content from your analysis, enabling you to focus on what truly matters