How to filter log lines

You may want to use Oncrawl's SEO log monitoring solution but not want to share all of your server log data with Oncrawl. Not a problem! You can filter your log lines and provide OnCrawl with only the log lines that are useful for SEO analysis.

This also has the advantage of reducing the size of your log files, making it faster and easier to transfer them to OnCrawl and to process them.

Here's how it works.

Filtering the right lines for SEO analysis

SEO analysis concentrates on two types of log lines:

Search engine bot hits. These are lines where the User Agent string matches one of the bots you want to monitor, for example, bots containing google or Googlebot. (We'll take care of verifying that these are actually Googlebots.)
Organic user hits (SEO visits). These lines contain google (as in "https://www.google.com" or "https://www.google.ca") in the Referer field.

Oncrawl monitors far more than just Google today. Bing, Yandex, Baidu, and a growing number of AI bots are also supported (see "Going further" below for the full list). Make sure your filtered lines include BOTH bot hits and organic hits for every engine you want to analyze.

Make sure your filtered lines include BOTH of these type of lines.

You may want to use a broader filter than necessary, and keep any and all lines that contain google anywhere in them.

Ideally, this filter should not be case sensitive.

Filtering the right lines for Ads bots and vertical bots

If you're also including lines for SEA and vertical bots, use a filter that is not case sensitive. This is particularly important, as these bots have varied naming conventions.

Example image bot:
Googlebot-Image/1.0 contains "google", but uses an uppercase G.

Example GoogleAds bot:
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) contains both "Google" and "google".

Example "standard" (SEO) Googlebot:
Mozilla/5.0 (compatible ; Googlebot/2.1 ; +http://www.google.com/bot.html) contains both "Google" and "google".

Going further

If you're interested in selecting certain bots only, you'll need to use more specific filters that directly target the specific bots you're interested in.

For an up-to-date reference covering all bots Oncrawl supports (search engines and AI bots), including exact user-agent strings, referrer patterns, and official IP ranges, see our supported bots list. For Google specifically, you can also refer to the user agent strings provided in Google's Search Console Help.