When choosing a plan for log analysis, you will base your decision on the number of line you will ask OnCrawl to analyze. You will need to know approximate numbers in order to estimate this ahead of time.

Only the lines with useful information for OnCrawl analyses are taken into account (Google visits, user visits referred from Google). All other lines are ignored.

Note: the screen captures below show how to find values in the old version of Google Search Console. These values are not yet available in the new version. If you are using the new version of Google Search Console, you can still return to the old interface using the link at the bottom of the sidebar navigation panel:

Estimate the number of log entries per day and per month

You can estimate the number of lines OnCrawl will analyze by looking at two statistics:

  1. Daily average Googlebot hits. This is the sum of Googlebot hits for the desktop and mobile bots. You can find this information in Google Search Console. Use the "pages crawled per day" data under "Crawl" > "Crawl stats".

    Hits from all bots are totaled in Google Search console. Since OnCrawl only uses data from the indexing bots (googlebot-desktop and googlebot-mobile), the actual daily number of "bot hits" will be lower in OnCrawl than in Google Search Console.
  2. Daily average of organic visits from Google. This can be found in any analytics tool. If you use Google Analytics, this is reported as a total number of unique sessions and can be found under "Acquisition" > "All Traffic" > "Channels". You will need to switch the table's primary dimension to "Source" and click on "Google".

    Divide the number of sessions reported by the number of days in the period you are looking at in order to obtain a daily average.

    Multiple visits by the same user within a 30-minute period are considered to be a single session in Google Analytics. (If a user is inactive on your site for 30 minutes or more, any future activity is attributed to a new session.) Since OnCrawl's SEO visits does not group visits seen in your log files into sessions, the number of SEO visits in OnCrawl may be higher than the number of "Organic sessions" in Google Analytics.

Finding the daily average bot hits in Search Console

Finding the total number of sessions with Google as the source in Google Analytics

Add these two daily estimates together.

Multiply by 30 (days per month).

The result is the approximate number of lines of log analysis that your site requires per month.

Using log analysis with SEA and vertical search optimization bots

The SEA and vertical bots option adds tracking for the following bots:

This bot crawls pages for AdSense.

  • Mediapartners-Google (mobile)
  • Mediapartners-Google (desktop)

These bots check the quality and pertinence of pages used in paid ads campaigns.

  • AdsBot-Google-Mobile
  • AdsBot-Google (desktop)
  • AdsBot-Google-Mobile-Apps

These bots crawl your site looking for indexable media.

  • Googlebot-Image
  • Googlebot-Video

If you are setting up Log Monitoring for the first time, you don't have to do anything differently. Your estimated monthly value based on the average daily number of bot hits will be much closer to the actual value found in your logs.

If you are adding the SEA and vertical bots option to an existing Log Monitoring solution, it is a good idea to return to the old version of Google Search Console to look up the daily average number of bot hits for all bots.

Use this average daily value if you intend to track all bots, and add the daily average organic visits from Google. Then multiply by 30. This estimate will be close to the actual value found in your logs.

By comparing this estimate to the number of lines you currently process for SEO bots only you will be able to get an idea of the difference represented by the additional bots.

Going further

If you still have questions, drop us a line at @oncrawl_cs or click on the Intercom button at the bottom right of your screen to start a chat with us.

Happy crawling!

You can also find this article by searching for:
cuántas líneas de registros hay que enviar, ficheros registro, cómo saber cuántos recursos necesito
comment savoir combien de lignes de logs il faut télécharger

Did this answer your question?