Skip to main content
All CollectionsLog MonitoringLog monitoring
How to calculate the number of resources needed for log analysis
How to calculate the number of resources needed for log analysis

OnCrawl log analysis levels are based on the number of lines in your logs. Here's how to estimate how many lines you need.

Updated over 3 months ago

When determining the size of your log analysis quota for Oncrawl, you'll be measuring this in the number of "log lines", or times your site records a page request from an SEO visitor or from an SEO bot. You will need to know approximate numbers in order to estimate this ahead of time.

Only the lines with useful information for Oncrawl analyses are taken into account (Google visits, user visits referred from Google). All other lines are ignored.

Estimate the number of log entries per day and per month

You can estimate the number of lines Oncrawl will analyze by looking at two statistics:

  1. Daily average Googlebot hits. This is the sum of Googlebot hits for the desktop and mobile bots. You can find this information in Google Search Console under Settings (in the left-hand menu) > Crawl stats.

    The total number of "Crawl requests" and divide it by the number of days in your Search Console report. (This number is visible on the Settings page.) This data includes requests from all of the bots that Google uses, not just the requests from googlebot. This means that this number will give you a rough estimate, but will likely be greater than the real number of requests taken into account by OnCrawl.

    If you have multiple domains or subdomains for which your are setting up log monitoring, make sure you get this information for each.

    47.9K / 90 = 532

    Finding the daily average bot hits in Search Console

  2. Daily average of organic visits from Google. This can be found in any analytics tool. If you use Google Analytics (GA4), this is reported as a total number of unique sessions and can be found under Reports > Acquisition > Traffic acquisition. You will need to switch the table's primary dimension to Source / Medium and look for the line Google / Organic.

    In this example, there are 12,555 organic sessions from Google in the time period we're looking at:


    Divide the number of sessions reported by the number of days in the period you are looking at in order to obtain a daily average. If I'm looking at one week, that would mean: 12,555 sessions in a week / 7 days = 1794 sessions per day.

    Note: Multiple visits by the same user within the same period can be considered to be a single session in GA4. Since Oncrawl's SEO visits does not group visits seen in your log files into sessions, the number of SEO visits in Oncrawl may be higher than the number of organic sessions in GA4.

  3. Add these two daily estimates together: 532 crawl requests + 1794 visits = 2326 estimated lines per day.

  4. Multiply by 30 (days per month): 69,780 lines per month.

The result is the approximate minimum number of lines of log analysis that your site requires per month.

Using log analysis with SEA and vertical search optimization bots

The SEA and vertical bots option adds tracking for the following bots:

Adsense
This bot crawls pages for AdSense.

  • Mediapartners-Google (mobile)

  • Mediapartners-Google (desktop)

Adsbot
These bots check the quality and pertinence of pages used in paid ads campaigns.

  • AdsBot-Google-Mobile

  • AdsBot-Google (desktop)

  • AdsBot-Google-Mobile-Apps

Media
These bots crawl your site looking for indexable media.

  • Googlebot-Image

  • Googlebot-Video

AI

These bots crawl your site looking for usable material for AI LLMs.

  • ChatGPT bot

If you are setting up Log Monitoring for the first time, you don't have to do anything differently. Your estimated monthly value based on the average daily number of bot hits will be much closer to the actual value found in your logs.

If you are adding the SEA and vertical bots option to an existing Log Monitoring solution, it is a good idea to return to the old version of Google Search Console to look up the daily average number of bot hits for all bots.

Use this average daily value if you intend to track all bots, and add the daily average organic visits from Google. Then multiply by 30. This estimate will be close to the actual value found in your logs.

By comparing this estimate to the number of lines you currently process for SEO bots only you will be able to get an idea of the difference represented by the additional bots.

Did this answer your question?