Oncrawl

When the crawler or log analyzer encounters a reference to certain types of files, it groups them as "resources". Here's the complete list.

What files are considered to be "resources" in Oncrawl?

Check whether everything's working as expected and receive maintenance notifications

OnCrawl archives old crawls. The data's still there. Here's how to unarchive all or part of an archived crawl.

Need to change the name of a project? It's easy.

A step by step article to understand how to share projects in different modes.

You've downloaded your Data Explorer results as a CSV file, but you usually work in Excel or Sheets. Now what?

Our bot supports the Crawl Delay directive of robots.txt file. If you have validated the project you can overcome this, learn how now!

How to change crawl speed and max crawl depth after launching your crawl

What is load time, and how does OnCrawl measure it?

Sometimes you may need to analyze a more or less important list of URLs without browsing the entire site. Here is how to proceed.

You can use sitemaps to compare and add data to a crawl. Here are the formats OnCrawl supports.

Sometimes you just want data on the URLs in your sitemap. It's possible to do this using a list of URLs.

Here is how to make sure that your sitemaps will be taken into account during the analysis

3xx status codes indicate a redirected page. Here’s how to find the page that was redirected, as well as the page it was redirected to.

Your crawl results can help you to quickly produce a list of all redirected pages and the pages to which they are redirected.

You have permanently redirected (301) URLs. Here's how to create a list of all the links, the URL, and the URL they are redirected to.

It's good practice to protect staging websites and pre-production websites from bots. Here's how to crawl one before it goes live.

If you follow your data closely you may have noticed volumes differences, here's why: OnCrawl is more reliable than GA over longer periods.

Crawl problems because your site redirects the Oncrawl bot because it doesn't have the right location-based cookie?  Here's how to fix it.

Data protection and privacy for personal data. Where is data hosted? What data is collected? Can you refuse to give OnCrawl personal data?