General information

[Tutorial] How to use OnCrawl's Inrank

Inrank, or internal PageRank measures on-site page popularity and link equity. Here's how to interpret it.

Written by Rebecca Berbel
Updated over a week ago

[Tutorial] How to share a project in OnCrawl

You might want to share a project with coworkers or clients. Here's how.

Written by Rebecca Berbel
Updated over a week ago

[Tutorial] How to use the OnCrawl Chrome extension

OnCrawl's free chrome extension highlights on-page metrics directly in your browser, helping you to spot issues easily.

Written by Rebecca Berbel
Updated over a week ago

[Tutorial] How to monitor page speed with OnCrawl

Discover how measure page speed on your site with OnCrawl. But don't stop there: learn how OnCrawl helps you monitor page speed.

Written by Rebecca Berbel
Updated over a week ago

Tutorial: How to spot changing status codes with our Data Explorer

Written by Emma Labrador
Updated over a week ago

Tutorial: How to use the OnCrawl near duplicate detector

Written by Francois Goube
Updated over a week ago

Tutorial: How to monitor your main tags' performances with OnCrawl

Discover how to monitor your meta tags and Hn performances using OnCrawl. A complete guide about what OnCrawl is analyzing.

Written by Emma Labrador
Updated over a week ago

How to check if OnCrawl services are up or down

Check whether everything's working as expected and receive maintenance notifications

Written by Rebecca Berbel
Updated over a week ago

Archived crawls and how to un-archive them

OnCrawl archives old crawls. The data's still there. Here's how to unarchive all or part of an archived crawl.

Written by Rebecca Berbel
Updated over a week ago

Rename a project

Need to change the name of a project? It's easy.

Written by Rebecca Berbel
Updated over a week ago

Share a project

A step by step article to understand how to share projects in different modes.

Written by Emma Labrador
Updated over a week ago

How to open a export CSV file in Excel or Sheets

You've downloaded your Data Explorer results as a CSV file, but you usually work in Excel or Sheets. Now what?

Written by Rebecca Berbel
Updated over a week ago

How to crawl a site with a robots.txt crawl delay greater than 1 second

Our bot supports the Crawl Delay directive of robots.txt file. If you have validated the project you can overcome this, learn how now!

Written by Tanguy
Updated over a week ago

How to modify crawl limits while crawling

How to change crawl speed and max crawl depth after launching your crawl

Written by Rebecca Berbel
Updated over a week ago

How does OnCrawl calculate load time?

What is load time, and how does OnCrawl measure it?

Written by Rebecca Berbel
Updated over a week ago

How to quickly scan and analyze a list of URLs?

Sometimes you may need to analyze a more or less important list of URLs without browsing the entire site. Here is how to proceed.

Written by Rebecca Berbel
Updated over a week ago

What format can I use for my sitemaps?

You can use sitemaps to compare and add data to a crawl. Here are the formats OnCrawl supports.

Written by Rebecca Berbel
Updated over a week ago

How to explore all of the URLs in my sitemap

Sometimes you just want data on the URLs in your sitemap. It's possible to do this using a list of URLs.

Written by Rebecca Berbel
Updated over a week ago

How to check urls in sitemap?

Here is how to make sure that your sitemaps will be taken into account during the analysis

Written by Rebecca Berbel
Updated over a week ago

How can I find the target of a 3xx (redirection) status code?

3xx status codes indicate a redirected page. Here’s how to find the page that was redirected, as well as the page it was redirected to.

Written by Rebecca Berbel
Updated over a week ago

How to use crawl results to find redirection loops

Your crawl results can help you to quickly produce a list of all redirected pages and the pages to which they are redirected.

Written by Rebecca Berbel
Updated over a week ago

How to create a file listing all links pointing to a 301 URL, the old URL, and the new URL

You have permanently redirected (301) URLs. Here's how to create a list of all the links, the URL, and the URL they are redirected to.

Written by Rebecca Berbel
Updated over a week ago

How to crawl a staging/pre-prod website

It's good practice to protect staging websites and pre-production websites from bots. Here's how to crawl one before it goes live.

Written by Rebecca Berbel
Updated over a week ago

Why are there session volume differences between Google Analytics and my OnCrawl report?

If you follow your data closely you may have noticed volumes differences, here's why: OnCrawl is more reliable than GA over longer periods.

Written by Rebecca Berbel
Updated over a week ago

How to bypass geographic redirects by crawling with HTTP headers

Crawl problems because your site redirects the OnCrawl bot because it doesn't have the right location-based cookie? Here's how to fix it.

Written by Rebecca Berbel
Updated over a week ago

OnCrawl and the GDPR

Data protection and privacy for personal data. Where is data hosted? What data is collected? Can you refuse to give OnCrawl personal data?

Written by Rebecca Berbel
Updated over a week ago

REGEX in OnCrawl

Use pattern detection in fields to get to the essentials faster. Use regular expressions to create filters (Data Explorer & Segmentations)

Written by Rebecca Berbel
Updated over a week ago

Lucene REGEX Cheat Sheet

This article is based on the Elastic Search Article

Written by Rebecca Berbel
Updated over a week ago

Custom dashboards

What are custom dashboards and how can you create and use them?

Written by Rebecca Berbel
Updated over a week ago

Crawl over Crawl

A Crawl over Crawl compares the results of two crawls to show you what the impact of your changes is or how two sites differ.

Written by Rebecca Berbel
Updated over a week ago

Data scraping and custom fields

Data scraping is an option that allows you to analyse a portion of the source code extracted during a crawl using custom fields.

Written by Rebecca Berbel
Updated over a week ago

How to crawl a site that uses JavaScript

If all or part of your website is built using JavaScript (JS), you may need to render pages in order to for a bot to be able to crawl them.

Written by Rebecca Berbel
Updated over a week ago

User-agent and bot name

Change the bot name to test rules aimed at different bots

Written by Rebecca Berbel
Updated over a week ago

OnCrawl API

OnCrawl is based on a platform built around an API. You can create your own application to request this API very easily, here's how to do it

Written by Rebecca Berbel
Updated over a week ago

OnCrawl connector for Google Data Studio

Answers to your questions about the OnCrawl Data Studio connector

Written by Rebecca Berbel
Updated over a week ago