FAQ

Rebecca Berbel avatar Tanguy avatar Erle Alberton avatar
16 articles in this collection
Written by Rebecca Berbel, Tanguy, and Erle Alberton

How to recover a lost password

What to do if you forgot your password.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

OnCrawl and the GDPR

Data protection and privacy for personal data. Where is data hosted? What data is collected? Can you refuse to give OnCrawl personal data?
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated this week

How to crawl some subdomains but not others

Sometimes you'll want to crawl only certain subdomains of your site, while excluding other subdomains from the crawl. Find out how.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

How to use crawl results to find redirection loops

Your crawl results can help you to quickly produce a list of all redirected pages and the pages to which they are redirected.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

How to crawl a site with a robots.txt crawl delay greater than 1 second

Our bot supports the Crawl Delay directive of robots.txt file. If you have validated the project you can overcome this, learn how now!
Tanguy avatar
Written by Tanguy
Updated yesterday

How to crawl a staging/pre-prod website

It's good practice to protect staging websites and pre-production websites from bots. Here's how to crawl one before it goes live.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

What IPs does OnCrawl use to crawl a website?

OnCrawl uses a dynamic range of IP addresses to crawl websites. If necessary, it's possible to restrict this range to a series of static IPs
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

Lucene REGEX Cheat Sheet

This article is based on the Elastic Search Article
Erle Alberton avatar
Written by Erle Alberton
Updated yesterday

What format can I use for my sitemaps?

You can use sitemaps to compare and add data to a crawl. Here are the formats OnCrawl supports.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

How to check urls in sitemap?

Here is how to make sure that your sitemaps will be taken into account during the analysis
Erle Alberton avatar
Written by Erle Alberton
Updated yesterday

Why are there session volume differences between Google Analytics and my OnCrawl report?

If you follow your data closely you may have noticed volumes differences, here's why: OnCrawl is more reliable than GA over longer periods.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

How can I find the location of a 3xx (redirection) status code?

3xx status codes indicate a redirected page. Here’s how to find the page that was redirected, as well as the page it was redirected to.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

How to quickly scan and analyze a list of URLs?

Sometimes you may need to analyze a more or less important list of URLs without browsing the entire site. Here is how to proceed.
Erle Alberton avatar
Written by Erle Alberton
Updated yesterday

What to do when my graphs are empty

What to do if you ran a crawl, but your graphs and charts aren't showing any data
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

Why does my report only contain one URL?

What to do if your crawl results yield an analysis with only one URL.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday

How to create a file listing all links pointing to a 301 URL, the old URL, and the new URL

You have permanently redirected (301) URLs. Here's how to create a list of all the links, the URL, and the URL they are redirected to.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated yesterday