Oncrawl monitors a comprehensive range of bots to give you complete visibility into how automated traffic interacts with your site. Below, you will find the complete list of search bots and AI bots currently supported in the platform.
If you would like to activate monitoring for specific bots, please contact your dedicated Customer Success Manager, Account Manager, or reach out through the in-app chat.
Currently supported bots:
Brand | Label | Kind | Agent pattern |
Google desktop | seo | (?<bot>Googlebot/[0-9.]+) | |
Google mobile | seo | (?<bot>Googlebot/[0-9.]+) | |
AdsBot Google desktop | sea | ^(?<bot>AdsBot-Google)[^-] | |
AdsBot Google mobile | sea | (?<bot>AdsBot-Google-Mobile); | |
Google Adsense | sea | ^(?<bot>Mediapartners-Google)$ | |
Google Adsense mobile | sea | (?<bot>Mediapartners-Google/[0-9.]+) | |
Google images | vertical | ^(?<bot>Googlebot-Image/[0-9.]+)$ | |
Google Speakr | vertical | ^(?<bot>Google-speakr)$ | |
Google video | vertical | ^(?<bot>Googlebot-Video/[0-9.]+)$ | |
AdsBot Google mobile apps | vertical | ^(?<bot>AdsBot-Google-Mobile-Apps)$ | |
Google-HotelAdsVerifier | sea | ^(?<bot>Google-HotelAdsVerifier/[0-9.]+)$ | |
Google store desktop | vertical | (?<bot>Storebot-Google/[0-9.]+) | |
Google store mobile | vertical | (?<bot>Storebot-Google/[0-9.]+) | |
Google Extended | vertical | (?<bot>Google-Extended) | |
Google news | vertical | ^(?<bot>Googlebot-News)$ | |
Google CloudVertexBot | ai user | (?<bot>Google-CloudVertexBot) | |
Gemini Deep Research | ai user | (?<bot>Gemini-Deep-Research) | |
Gemini Agent Mariner | ai user | (?<bot>GoogleAgent-Mariner) | |
Google Agent Search | ai user | (?<bot>GoogleAgent-Search) | |
Google Notebook LM | ai user | (?<bot>Google-NotebookLM) | |
Bing | Bing desktop | seo | (?<bot>[Bb]ingbot/[0-9.]+) |
Bing | Bing mobile | seo | (?<bot>[Bb]ingbot/[0-9.]+) |
Bing | Bing ads | sea | (?<bot>adidxbot/[0-9.]+) |
Bing | Bing ads mobile | sea | (?<bot>adidxbot/[0-9.]+) |
Baidu | Baidu desktop | seo | (?<bot>Baiduspider([+]|(-render)?/[0-9.]+)) |
Baidu | Baidu mobile | seo | .*\(.*(Phone|Android).*\).*(?<bot>Baiduspider(-render)?/[0-9.]+) |
Baidu | Baidu ads | sea | (?<bot>Baiduspider-ads/[0-9.]+) |
Baidu | Baidu images | vertical | (?<bot>Baiduspider-image/[0-9.]+) |
Baidu | Baidu video | vertical | (?<bot>Baiduspider-video/[0-9.]+) |
Baidu | Baidu news | vertical | (?<bot>Baiduspider-news/[0-9.]+) |
Yandex | Yandex desktop | seo | (?<bot>(YandexBot|YandexAccessibilityBot)/[0-9.]+) |
Yandex | Yandex mobile | seo | (?<bot>(YandexBot|YandexAccessibilityBot|YandexMobileBot)/[0-9.]+) |
Yandex | Yandex ads | sea | (?<bot>YandexAdNet/[0-9.]+) |
Yandex | Yandex images | vertical | (?<bot>YandexImages/[0-9.]+) |
Yandex | Yandex images mobile | vertical | (?<bot>YandexImageResizer/[0-9.]+) |
Yandex | Yandex news | vertical | (?<bot>YandexNews/[0-9.]+) |
Yandex | Yandex video | vertical | (?<bot>YandexVideo/[0-9.]+) |
OpenAI | Openai SearchBot | ai search | (?<bot>OAI-SearchBot/[0-9.]+) |
OpenAI | OpenAI ChatGPT User | ai user | (?<bot>ChatGPT-User/[0-9.]+) |
OpenAI | OpenAI GPTBot | ai training | (?<bot>GPTBot/[0-9.]+) |
Claude | Claude SearchBot | ai search | (?<bot>Claude-SearchBot/[0-9.]+) |
Claude | Claude User | ai user | (?<bot>Claude-User/[0-9.]+) |
Claude | Claude Bot | ai training | (?<bot>ClaudeBot/[0-9.]+) |
Perplexity | Perplexity Bot | ai search | (?<bot>PerplexityBot/[0-9.]+) |
Perplexity | Perplexity User | ai user | (?<bot>Perplexity-User/[0-9.]+) |
Mistral | Mistral User | ai user | (?<bot>MistralAI-User/[0-9.]+) |
🆕 CommonCrawl | CCBot | ai training | (?<bot>CCBot/[0-9.]+) |
🆕 Ibou | IbouBot | ai search | (?<bot>IbouBot/[0-9.]+) |
🆕 Ai2 | AI2Bot | ai training | (?<bot>AI2Bot) |
🆕 Amazon | AmazonBot | ai training | (?<bot>AmazonBot/[0-9.]+) |
🆕 Meta | Meta Web Indexer | ai search | (?<bot>Meta-WebIndexer/[0-9.]+) |
🆕 Meta | Meta External Fetcher | ai_user | (?<bot>Meta-ExternalFetcher/[0-9.]+) |
🆕 Meta | Meta ExternalAgent | ai training | (?<bot>Meta-ExternalAgent/[0-9.]+) |
🆕 ByteDance | Bytespider | ai training | (?<bot>Bytespider) |
🆕 ByteDance | Bytespider image | ai training | (?<bot>imageSpider) |
🆕 DuckDuckGo | DuckDuckBot | seo | (?<bot>DuckDuckBot/[0-9.]+) |
🆕 DuckDuckGo | DuckAssistBot | ai user | (?<bot>DuckAssistBot/[0-9.]+) |
🆕 Apple | Applebot | ai search | (?<bot>Applebot/[0-9.]+) |
🆕 Apple | Applebot Extended | ai training | (?<bot>Applebot-Extended/[0-9.]+) |
🆕 Brave | Brave Bot | seo | (?<bot>BraveBot/[0-9.]+) |
Supported sources of referral traffic:
Brand | Label | Referer pattern |
^(https?://(www|news)[.]google[.]|android-app://com\.google\.android\.(apps\.genie\.geniewidget|googlequicksearchbox)).* | ||
Gemini | ^(https?://gemini[.]google[.]com).* | |
Bing | Bing | ^https?://(www[.])?bing[.].*$ |
Baidu | Baidu | ^https?://((www|news)[.])?baidu[.].*$ |
Yandex | Yandex | ^https?://(www[.])?yandex[.].*$ |
OpenAI | Openai | ^https?://((chat[.])?openai|chatgpt)[.]com.*$ |
Claude | Claude | ^https?://([\w.-]+\.)?(claude\.ai|anthropic\.com)/.*$ |
Perplexity | Perplexity | ^https?://([\w.-]+\.)?perplexity\.ai/.*$ |
Mistral | Mistral | ^https?://([\w.-]+\.)?mistral\.ai/.*$ |
🆕 DuckDuckGo | DuckDuckGo | ^(https?://duckduckgo[.]com).* |
🆕 Brave | Brave | ^(https?://search[.]brave[.]com).* |
We’re continuously improving our bot detection capabilities. If you have any suggestions or would like to see new bots added to the supported list, don’t hesitate to contact us.