Oncrawl

What is a project in Oncrawl and how to manage projects

Projects

Find your way around the new look and feel of the Oncrawl platform.

Oncrawl's new interface

Core Web Vitals are a confirmed, official ranking signal. You can measure your site's performance on these metrics with Oncrawl

Auditing Core Web Vitals

Understanding your monthly URL quota

Rebecca Berbel

Emma Labrador

Francois Goube

General information about how to use Oncrawl

General information

Inrank, or internal PageRank measures on-site page popularity and link equity. Here's how to interpret it.

[Tutorial] How to use OnCrawl's Inrank

You might want to share a project with coworkers or clients. Here's how.

[Tutorial] How to share a project in OnCrawl

OnCrawl's free chrome extension highlights on-page metrics directly in your browser, helping you to spot issues easily.

[Tutorial] How to use the OnCrawl Chrome extension

Discover how measure page speed on your site with OnCrawl. But don't stop there: learn how OnCrawl helps you monitor page speed.

[Tutorial] How to monitor page speed with OnCrawl

Tutorial: How to spot changing status codes with our Data Explorer

Tutorial: How to use the OnCrawl near duplicate detector

Discover how to monitor your meta tags and Hn performances using OnCrawl. A complete guide about what OnCrawl is analyzing.

Tutorial: How to monitor your main tags' performances with OnCrawl

Video tutorials

When the crawler or log analyzer encounters a reference to certain types of files, it groups them as "resources". Here's the complete list.

What files are considered to be "resources" in Oncrawl?

Check whether everything's working as expected and receive maintenance notifications

How to check if OnCrawl services are up or down

OnCrawl archives old crawls. The data's still there. Here's how to unarchive all or part of an archived crawl.

Archived crawls and how to un-archive them

Need to change the name of a project? It's easy.

Rename a project

A step by step article to understand how to share projects in different modes.

Share a project

You've downloaded your Data Explorer results as a CSV file, but you usually work in Excel or Sheets. Now what?

How to open a export CSV file in Excel or Sheets

Our bot supports the Crawl Delay directive of robots.txt file. If you have validated the project you can overcome this, learn how now!

How to crawl a site with a robots.txt crawl delay greater than 1 second

How to change crawl speed and max crawl depth after launching your crawl

How to modify crawl limits while crawling

What is load time, and how does OnCrawl measure it?

How does OnCrawl calculate load time?

Sometimes you may need to analyze a more or less important list of URLs without browsing the entire site. Here is how to proceed.

How to quickly scan and analyze a list of URLs?

You can use sitemaps to compare and add data to a crawl. Here are the formats OnCrawl supports.

What format can I use for my sitemaps?

Sometimes you just want data on the URLs in your sitemap. It's possible to do this using a list of URLs.

How to explore all of the URLs in my sitemap

Here is how to make sure that your sitemaps will be taken into account during the analysis

How to check urls in sitemap?

3xx status codes indicate a redirected page. Here’s how to find the page that was redirected, as well as the page it was redirected to.

How can I find the target of a 3xx (redirection) status code?

Your crawl results can help you to quickly produce a list of all redirected pages and the pages to which they are redirected.

How to use crawl results to find redirection loops

You have permanently redirected (301) URLs. Here's how to create a list of all the links, the URL, and the URL they are redirected to.

How to create a file listing all links pointing to a 301 URL, the old URL, and the new URL

It's good practice to protect staging websites and pre-production websites from bots. Here's how to crawl one before it goes live.

How to crawl a staging/pre-prod website

If you follow your data closely you may have noticed volumes differences, here's why: OnCrawl is more reliable than GA over longer periods.

Why are there session volume differences between Google Analytics and my OnCrawl report?

Crawl problems because your site redirects the Oncrawl bot because it doesn't have the right location-based cookie?  Here's how to fix it.

How to bypass geographic redirects by crawling with HTTP headers

Data protection and privacy for personal data. Where is data hosted? What data is collected? Can you refuse to give OnCrawl personal data?

OnCrawl and the GDPR

Tanguy

Understanding the information collected by Oncrawl about a website's links

Link Dataset

Use pattern detection in fields to get to the essentials faster. Use regular expressions to create filters (Data Explorer & Segmentations)

REGEX in OnCrawl

This article is based on the Elastic Search Article

Lucene REGEX Cheat Sheet

What are custom dashboards and how can you create and use them?

Custom dashboards

A Crawl over Crawl compares the results of two crawls to show you what the impact of your changes is or how two sites differ.

Crawl over Crawl

Data scraping is an option that allows you to analyse a portion of the source code extracted during a crawl using custom fields.

Data scraping and custom fields

If all or part of your website is built using JavaScript (JS), you may need to render pages in order to for a bot to be able to crawl them.

How to crawl a site that uses JavaScript

Change the bot name to test rules aimed at different bots

User-agent and bot name

Oncrawl is based on a platform built around an API. You can create your own application to request this API very easily, here's how to do it

Oncrawl API

Answers to your questions about the Oncrawl Looker Studio (ex Data Studio) connector