General information

General information about how to use OnCrawl

Francois Goube avatar Tanguy avatar Emma Labrador avatar +1
34 articles in this collection
Written by Francois Goube, Tanguy, Emma Labrador and 1 other
FAQ

How to check if OnCrawl services are up or down

Check whether everything's working as expected and receive maintenance notifications
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

Archived crawls and how to un-archive them

OnCrawl archives old crawls. The data's still there. Here's how to unarchive all or part of an archived crawl.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

Rename a project

Need to change the name of a project? It's easy.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

Share a project

A step by step article to understand how to share projects in different modes.
Emma Labrador avatar
Written by Emma Labrador
Updated over a week ago

How to open a export CSV file in Excel or Sheets

You've downloaded your Data Explorer results as a CSV file, but you usually work in Excel or Sheets. Now what?
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to crawl a site with a robots.txt crawl delay greater than 1 second

Our bot supports the Crawl Delay directive of robots.txt file. If you have validated the project you can overcome this, learn how now!
Tanguy avatar
Written by Tanguy
Updated over a week ago

How to modify crawl limits while crawling

How to change crawl speed and max crawl depth after launching your crawl
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How does OnCrawl calculate load time?

What is load time, and how does OnCrawl measure it?
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to quickly scan and analyze a list of URLs?

Sometimes you may need to analyze a more or less important list of URLs without browsing the entire site. Here is how to proceed.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

What format can I use for my sitemaps?

You can use sitemaps to compare and add data to a crawl. Here are the formats OnCrawl supports.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to explore all of the URLs in my sitemap

Sometimes you just want data on the URLs in your sitemap. It's possible to do this using a list of URLs.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to check urls in sitemap?

Here is how to make sure that your sitemaps will be taken into account during the analysis
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How can I find the target of a 3xx (redirection) status code?

3xx status codes indicate a redirected page. Here’s how to find the page that was redirected, as well as the page it was redirected to.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to use crawl results to find redirection loops

Your crawl results can help you to quickly produce a list of all redirected pages and the pages to which they are redirected.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to create a file listing all links pointing to a 301 URL, the old URL, and the new URL

You have permanently redirected (301) URLs. Here's how to create a list of all the links, the URL, and the URL they are redirected to.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to crawl a staging/pre-prod website

It's good practice to protect staging websites and pre-production websites from bots. Here's how to crawl one before it goes live.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

Why are there session volume differences between Google Analytics and my OnCrawl report?

If you follow your data closely you may have noticed volumes differences, here's why: OnCrawl is more reliable than GA over longer periods.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

How to bypass geographic redirects by crawling with HTTP headers

Crawl problems because your site redirects the OnCrawl bot because it doesn't have the right location-based cookie? Here's how to fix it.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

OnCrawl and the GDPR

Data protection and privacy for personal data. Where is data hosted? What data is collected? Can you refuse to give OnCrawl personal data?
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago
Advanced uses and settings

REGEX in OnCrawl

Use pattern detection in fields to get to the essentials faster. Use regular expressions to create filters (Data Explorer & Segmentations)
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

Lucene REGEX Cheat Sheet

This article is based on the Elastic Search Article
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

Custom dashboards

What are custom dashboards and how can you create and use them?
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

Crawl over Crawl

A Crawl over Crawl compares the results of two crawls to show you what the impact of your changes is or how two sites differ.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

Data scraping and custom fields

Data scraping is an option that allows you to analyse a portion of the source code extracted during a crawl using custom fields.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

JavaScript

If all or part of your website is built using JavaScript (JS), you may need to render pages in order to for a bot to be able to crawl them.
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

User-agent and bot name

Change the bot name to test rules aimed at different bots
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

OnCrawl API

OnCrawl is based on a platform built around an API. You can create your own application to request this API very easily, here's how to do it
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago

OnCrawl connector for Google Data Studio

Answers to your questions about the OnCrawl Data Studio connector
Rebecca Berbel avatar
Written by Rebecca Berbel
Updated over a week ago