SEA & vertical bots

The SEA & Vertical bots dashboard provides information about hits by additional Google bots that is extracted from an analysis of your logs.

Updated over a week ago

Under the Logs Monitoring analysis report, SEA & Vertical bots presents information on Google bots not specifically dedicated to SEO. This includes SEA bots and vertical search optimization bots. By analyzing lines in your server logs, you can also follow the hits of these additional bots on your site.

How to get access to the SEA & vertical search optimization bots page

Because this option can change your log monitoring pricing due to the additional lines to be analyzed, you should contact us. (Use the blue Intercom button at the bottom right of this page.)

Once the option is activated, OnCrawl will analyze the next logs you upload for hits by additional Google bots.

The information for SEA and vertical search optimization bots is not retroactive: logs you have previously uploaded will not be automatically re-analyzed. If you are interested in retroactive information for SEA and vertical search optimization bots, you will need to analyze old logs for the period you are interested in. If this is the case, since you already have data for SEO bots and SEO visits, you can filter your logs before you re-upload them in order to limit the number of lines OnCrawl will analyze.

Monitored SEA and Vertical Search Optimization Bots

AdSense
These bots crawl pages for AdSense.

  • Mediapartners-Google (mobile)

  • Mediapartners-Google (desktop)

Adsbot
These bots check the quality and pertinence of pages used in paid ads campaigns.

  • AdsBot-Google-Mobile

  • AdsBot-Google (desktop)

  • AdsBot-Google-Mobile-Apps

Media
These bots crawl your site looking for indexable media.

  • Googlebot-Image

  • Googlebot-Video

Note: you can track the behavior of Google's SEO bots, Googlebot (mobile) and Googlebot (desktop), in the other reports under Logs Monitoring.

Our data

In the analysis results for a crawl with log monitoring and SEA data, you can find information on SEA and vertical search optimization bots under the Log Monitoring report, in the "SEA & vertical bots" dashboard tab. The charts on this page are each available for pages and for resources such as documents, images, JavaScript, css, and videos.

The information on SEA and vertical search optimization bots is also available in the Data Explorer. To create a report based on this information, begin by switching the dataset to "Log Monitoring: Events".

At this time, fields concerning these bots cannot be used for segmentation or cross-analysis.

Please note that, outside of the "SEA & vertical bots" dashboard, anywhere you see "All bots" indicated in the interface, this refers to all SEO bots, that is: only googlebot (desktop) and googlebot (mobile).

Chart: Crawl behavior

This chart presents the number of hits per bot per day, on pages or on resources such as images.

Chart: Status codes encountered by bots breakdown

Use this chart to evaluate the status of URLs hit by SEA and vertical search optimization bots. For each bot, the status codes returned by the server (200, 3xx, 4xx, 5xx…) is displayed. Pages and resources are analyzed separately, each with their own tab.

This chart presents the same type of information in the chart "Status codes encountered by bots breakdown" on the "Exploration sanity" page, but for SEA and vertical search optimization bots instead of SEO bots.

Chart: Sum of bytes by bots

Use this chart to evaluate the number of bytes sent to each bot when it requests URLs. As pages and resources are usually very different in size, they are analyzed separately.

This chart presents the same type of information in the chart "Sum of bytes breakdown" on the "Exploration sanity" page, but for SEA and vertical search optimization bots instead of SEO bots.

Chart: Average response time by bots

If your log files include response time logged by the server, this chart will provide the average time it takes to provide content to a bot. This is provided as an average time per page or per resource.

This chart presents the same type of information in the chart "Average response time" on the "Exploration sanity" page, but for SEA and vertical search optimization bots instead of SEO bots.

Best practices for SEA & vertical bots

  • You must already have the log monitoring option in order to track hits by SEA and vertical search optimization bots. (This information is extracted from analyses your log files.)

  • As this option increases the number of lines OnCrawl analyzes, it may have an impact on your pricing.

  • It is possible to follow only a selection of the additional bots by filtering your log files before sending them to OnCrawl. Remove log file lines related to bots you do not want to follow.

  • If you filter your log files before sending them to OnCrawl, you will need to adjust your filters. Leave lines by the bots you are interested in. You can base your filter on information from the official list of Google bot user-agents.

  • You may need to switch to the "Resources" tab on many of the charts if only a selection of bots visit your site, or if you only monitor some bots. This is the case if your pages attract media bots, but not AdSense or AdsBot bots.

  • Use the "Log Monitoring: Events" dataset in the Data Explorer to create reports with this information.

  • If a large portion of your traffic comes from image or video searches, monitor media googlebots to understand and predict trends in user behavior.

Going further

If you still have questions about the SEA & vertical search optimization bots information provided in OnCrawl, drop us a line at @oncrawl_cs or click on the Intercom button at the bottom right of your screen to start a chat with us.

Happy crawling!

You can also find this article by searching for:
cuáles crawlers se siguen en los registros, búsqueda vertical, sea
bots suivis dans les logs de serveur, sea, recherche verticale

Did this answer your question?