You may want to use OnCrawl's SEO log monitoring solution but not want to share all of your server log data with OnCrawl. Not a problem! You can filter your log lines and provide OnCrawl with only the log lines that are useful for SEO analysis.

This also has the advantage of reducing the size of your log files, making it faster and easier to transfer them to OnCrawl and to process them.

Here's how it works.

Filtering the right lines for SEO analysis

SEO analysis concentrates on two types of log lines:

  1. Search engine bot hits. In this case, these are bots with google  or Googlebot  in the User Agent. (We'll take care of verifying that these are actually Googlebots.)
  2. Organic user hits (SEO visits). These lines contain google  (as in "https://www.google.com" or "https://www.google.ca") in the Referer field.

Make sure your filtered lines include BOTH of these type of lines.

You may want to use a broader filter than necessary, and keep any and all lines that contain google  anywhere in them.

Ideally, this filter should not be case sensitive.

Filtering the right lines for Ads bots and vertical bots

If you're including lines for SEA and vertical bots, use a filter that is not case sensitive. This is particularly important, as these bots have varied naming conventions.

Example image bot:
Googlebot-Image/1.0  contains "google", but uses an uppercase G.

Example "standard" (SEO) Googlebot:
Mozilla/5.0 (compatible ; Googlebot/2.1 ; +http://www.google.com/bot.html) contains both "Google" and "google".

Did this answer your question?