Under the Logs Monitoring analysis report, SEA & Vertical bots presents information on Google bots not specifically dedicated to SEO. This includes SEA bots and vertical search optimization bots, as well as AI bots that might be found crawling your site. By analyzing lines in your server logs, you can also follow the hits of these additional bots on your site.
How to get access to the SEA & vertical search optimization bots page
Once the option is activated, Oncrawl will analyze the next logs you upload for hits by additional Google bots. (You can contact us via chat to request this to be added to your plan, if it is not already included.)
The information for SEA and vertical search optimization bots is not retroactive: logs you have previously uploaded will not be automatically re-analyzed. If you are interested in retroactive information for SEA and vertical search optimization bots, you will need to analyze old logs for the period you are interested in. If this is the case, since you already have data for SEO bots and SEO visits, you can filter your logs before you re-upload them in order to limit the number of lines Oncrawl will analyze.
Monitored SEA and Vertical Search Optimization Bots
AdSense
These bots crawl pages for AdSense.
Mediapartners-Google (mobile)
Mediapartners-Google (desktop)
Adsbot
These bots check the quality and pertinence of pages used in paid ads campaigns.
AdsBot-Google-Mobile
AdsBot-Google (desktop)
AdsBot-Google-Mobile-Apps
Media
These bots crawl your site looking for indexable media.
Googlebot-Image
Googlebot-Video
AI
These bots crawl your site looking for information for AI LLMs.
ChatGPT bot
Note: you can track the behavior of Google's SEO bots, Googlebot (mobile) and Googlebot (desktop), in the other reports under Logs Monitoring.
Our data
In the analysis results for a crawl with log monitoring and SEA data, you can find information on SEA and vertical search optimization bots under the Log Monitoring report, in the "SEA & vertical bots" dashboard tab. The charts on this page are each available for pages and for resources such as documents, images, JavaScript, css, and videos.
The information on SEA and vertical search optimization bots is also available in the Data Explorer. To create a report based on this information, begin by switching the dataset to "Log Monitoring: Events".
At this time, fields concerning these bots cannot be used for segmentation or cross-analysis.
Please note that, outside of the "SEA & vertical bots" dashboard, anywhere you see "All bots" indicated in the interface, this refers to all SEO bots, that is: only googlebot (desktop) and googlebot (mobile).
Chart: Crawl behavior
This chart presents the number of hits per bot per day, on pages or on resources such as images.
Chart: Status codes encountered by bots breakdown
Use this chart to evaluate the status of URLs hit by SEA and vertical search optimization bots. For each bot, the status codes returned by the server (200, 3xx, 4xx, 5xx…) is displayed. Pages and resources are analyzed separately, each with their own tab.
This chart presents the same type of information in the chart "Status codes encountered by bots breakdown" on the "Exploration sanity" page, but for SEA and vertical search optimization bots instead of SEO bots.
Chart: Sum of bytes by bots
Use this chart to evaluate the number of bytes sent to each bot when it requests URLs. As pages and resources are usually very different in size, they are analyzed separately.
This chart presents the same type of information in the chart "Sum of bytes breakdown" on the "Exploration sanity" page, but for SEA and vertical search optimization bots instead of SEO bots.
Chart: Average response time by bots
If your log files include response time logged by the server, this chart will provide the average time it takes to provide content to a bot. This is provided as an average time per page or per resource.
This chart presents the same type of information in the chart "Average response time" on the "Exploration sanity" page, but for SEA and vertical search optimization bots instead of SEO bots.
Best practices for SEA & vertical bots
You must already have the log monitoring option in order to track hits by SEA and vertical search optimization bots. (This information is extracted from analyses your log files.)
As this option increases the number of lines Oncrawl analyzes, it may have an impact on your pricing.
It is possible to follow only a selection of the additional bots by filtering your log files before sending them to Oncrawl. Remove log file lines related to bots you do not want to follow.
If you filter your log files before sending them to Oncrawl, you will need to adjust your filters. Leave lines by the bots you are interested in. You can base your filter on information from the official list of Google bot user-agents.
You may need to switch to the "Resources" tab on many of the charts if only a selection of bots visit your site, or if you only monitor some bots. This is the case if your pages attract media bots, but not AdSense or AdsBot bots.
Use the "Log Monitoring: Events" dataset in the Data Explorer to create reports with this information.
If a large portion of your traffic comes from image or video searches, monitor media googlebots to understand and predict trends in user behavior.