Oncrawl monitors a comprehensive range of automated traffic to give you complete visibility into how it interacts with your site. This page lists every crawler, fetcher and referral source currently supported in the platform.
If you would like to activate monitoring for a specific agent, please contact your dedicated Customer Success Manager, Account Manager, or reach out through the in-app chat.
Crawlers and fetchers
Not every automated request behaves the same way in your logs, and the difference matters when you read them:
A crawler requests many URLs on its own schedule, follows links, and comes back. This is the behaviour you plan crawl budget around, and the one where crawl frequency and depth are meaningful.
A fetcher requests a single URL because someone just asked for it: a prompt, a share, a link preview. It does not follow links and it does not come back on a schedule, so a spike in fetches reflects user demand rather than a crawl decision.
Most brands operate both. Each category below is labelled accordingly.
Brands covered
Ai2, Amazon, Apple, Baidu, Bing, Brave, ByteDance, Claude, CommonCrawl, DuckDuckGo, Google, Ibou, Meta, Mistral, OpenAI, Perplexity, Yandex
What the categories mean
Category | Name | Type | What it covers |
| Search engine crawlers | Crawler | Organic search crawlers. Their hits are aggregated at the search engine level, which is what powers newly crawled pages, active pages and fresh rank. |
| Paid search crawlers | Crawler | Check landing pages and product feeds for advertising. |
| Vertical crawlers | Crawler | Dedicated to a single surface: images, video, news, stores or apps. |
| AI search crawlers | Crawler | Improving search result quality and page indexation. They crawl asynchronously, with clear patterns (typically one visit per URL per day), and far fewer pages than Googlebot. |
| AI user fetchers | Fetcher | Real-time content fetching for user answers and citations. They fetch a single page on demand to answer a prompt, without following links. |
| AI training crawlers | Crawler | Content scraping for model training. They have no crawl budget concept and do not render JavaScript. |
The three AI categories are covered in depth in AI bots explained: what powers platforms like ChatGPT?
Supported crawlers and fetchers
Where a desktop and a mobile entry share the same user-agent pattern, they are not duplicates: the parser tells them apart with an additional test on the user agent.
Search engine crawlers seo
Brand | Label | User-agent pattern |
Baidu | Baidu mobile | .*\(.*(Phone|Android).*\).*(?<bot>Baiduspider(-render)?/[0-9.]+) |
Baidu | Baidu desktop | (?<bot>Baiduspider([+]|(-render)?/[0-9.]+)) |
Bing | Bing mobile | (?<bot>[Bb]ingbot/[0-9.]+) |
Bing | Bing desktop | (?<bot>[Bb]ingbot/[0-9.]+) |
Brave | Brave Bot | (?<bot>BraveBot/[0-9.]+) |
DuckDuckGo | DuckDuckBot | (?<bot>DuckDuckBot/[0-9.]+) |
Google mobile | (?<bot>Googlebot/[0-9.]+) | |
Google desktop | (?<bot>Googlebot/[0-9.]+) | |
Yandex | Yandex mobile | (?<bot>(YandexBot|YandexAccessibilityBot|YandexMobileBot)/[0-9.]+) |
Yandex | Yandex desktop | (?<bot>(YandexBot|YandexAccessibilityBot)/[0-9.]+) |
Paid search crawlers sea
Brand | Label | User-agent pattern |
Baidu | Baidu ads | (?<bot>Baiduspider-ads/[0-9.]+) |
Bing | Bing ads | (?<bot>adidxbot/[0-9.]+) |
Bing | Bing ads mobile | (?<bot>adidxbot/[0-9.]+) |
Google AdsBot desktop | ^(?<bot>AdsBot-Google)[^-] | |
Google AdsBot mobile | (?<bot>AdsBot-Google-Mobile); | |
Google Adsense | ^(?<bot>Mediapartners-Google)$ | |
Google Adsense mobile | (?<bot>Mediapartners-Google/[0-9.]+) | |
Google HotelAds Verifier | ^(?<bot>Google-HotelAdsVerifier/[0-9.]+)$ | |
Meta | Meta ExternalAds | (?<bot>[Mm]eta-[Ee]xternal[Aa]ds/[0-9.]+) |
OpenAI | OpenAI AdsBot | (?<bot>OAI-AdsBot/[0-9.]+) |
Yandex | Yandex ads | (?<bot>YandexAdNet/[0-9.]+) |
Vertical crawlers vertical
Brand | Label | User-agent pattern |
Baidu | Baidu images | (?<bot>Baiduspider-image/[0-9.]+) |
Baidu | Baidu news | (?<bot>Baiduspider-news/[0-9.]+) |
Baidu | Baidu video | (?<bot>Baiduspider-video/[0-9.]+) |
Google Extended | (?<bot>Google-Extended) | |
Google images | ^(?<bot>Googlebot-Image/[0-9.]+)$ | |
Google AdsBot mobile apps | ^(?<bot>AdsBot-Google-Mobile-Apps)$ | |
Google news | ^(?<bot>Googlebot-News)$ | |
GoogleOther | (?<bot>GoogleOther) | |
Google Speakr | ^(?<bot>Google-speakr)$ | |
Google store desktop | (?<bot>Storebot-Google/[0-9.]+) | |
Google store mobile | (?<bot>Storebot-Google/[0-9.]+) | |
Google video | ^(?<bot>Googlebot-Video/[0-9.]+)$ | |
Meta | Facebook External Hit | (?<bot>facebookexternalhit/[0-9.]+) |
Yandex | Yandex images | (?<bot>YandexImages/[0-9.]+) |
Yandex | Yandex images mobile | (?<bot>YandexImageResizer/[0-9.]+) |
Yandex | Yandex news | (?<bot>YandexNews/[0-9.]+) |
Yandex | Yandex video | (?<bot>YandexVideo/[0-9.]+) |
AI search crawlers ai search
Brand | Label | User-agent pattern |
Amazon | Amazon SearchBot | (?<bot>Amzn-SearchBot/[0-9.]+) |
Apple | Applebot | (?<bot>Applebot/[0-9.]+) |
Claude | Claude SearchBot | (?<bot>Claude-SearchBot/[0-9.]+) |
Ibou | IbouBot | (?<bot>IbouBot/[0-9.]+) |
Meta | Meta Web Indexer | (?<bot>[Mm]eta-[Ww]eb[Ii]ndexer/[0-9.]+) |
OpenAI | OpenAI SearchBot | (?<bot>OAI-SearchBot/[0-9.]+) |
Perplexity | Perplexity Bot | (?<bot>PerplexityBot/[0-9.]+) |
AI user fetchers ai user
Brand | Label | User-agent pattern |
Amazon | Amazon User | (?<bot>Amzn-User/[0-9.]+) |
Claude | Claude User | (?<bot>Claude-User/[0-9.]+) |
DuckDuckGo | DuckAssistBot | (?<bot>DuckAssistBot/[0-9.]+) |
Google Agent | (?<bot>Google-Agent) | |
Google Agent Mariner | (?<bot>GoogleAgent-Mariner) | |
Google Agent Search | (?<bot>GoogleAgent-Search) | |
Google CloudVertexBot | (?<bot>Google-CloudVertexBot) | |
Google Gemini Deep Research | (?<bot>Gemini-Deep-Research) | |
Google Gemini Bot (Google) * | ^(?<bot>Google)$ | |
AI Mode Bot (Google-Lens) * | (?<bot>Google-Lens) | |
Google Notebook LM | (?<bot>Google-NotebookLM) | |
Meta | Meta External Fetcher | (?<bot>[Mm]eta-[Ee]xternal[Ff]etcher/[0-9.]+) |
Mistral | Mistral User | (?<bot>MistralAI-User/[0-9.]+) |
OpenAI | OpenAI ChatGPT User | (?<bot>ChatGPT-User/[0-9.]+) |
Perplexity | Perplexity User | (?<bot>Perplexity-User/[0-9.]+) |
* Not documented by the vendor. Oncrawl tracks these from observed traffic, so their user agent and behaviour can change without notice.
AI training crawlers ai training
Brand | Label | User-agent pattern |
Ai2 | AI2Bot | (?<bot>AI2Bot) |
Amazon | AmazonBot | (?<bot>[Aa]mazon[Bb]ot/[0-9.]+) |
Apple | Applebot Extended | (?<bot>Applebot-Extended/[0-9.]+) |
ByteDance | Bytespider | (?<bot>Bytespider) |
ByteDance | Bytespider image | (?<bot>imageSpider) |
Claude | Claude Bot | (?<bot>ClaudeBot/[0-9.]+) |
CommonCrawl | CCBot | (?<bot>CCBot/[0-9.]+) |
Meta | Meta ExternalAgent | (?<bot>[Mm]eta-[Ee]xternal[Aa]gent/[0-9.]+) |
OpenAI | OpenAI GPTBot | (?<bot>GPTBot/[0-9.]+) |
Supported sources of referral traffic
Visits are attributed to these sources by matching the referer of the request.
Brand | Label | Referer pattern |
^(https?://(www|news)[.]google[.]|android-app://com\.google\.android\.(apps\.genie\.geniewidget|googlequicksearchbox)).* | ||
Gemini | ^(https?://gemini[.]google[.]com).* | |
Bing | Bing | ^https?://(www[.])?bing[.].*$ |
Baidu | Baidu | ^https?://((www|news)[.])?baidu[.].*$ |
Yandex | Yandex | ^https?://(www[.])?yandex[.].*$ |
OpenAI | Openai | ^https?://((chat[.])?openai|chatgpt)[.]com.*$ |
Claude | Claude | ^https?://([\w.-]+\.)?(claude\.ai|anthropic\.com)/.*$ |
Perplexity | Perplexity | ^https?://([\w.-]+\.)?perplexity\.ai/.*$ |
Mistral | Mistral | ^https?://([\w.-]+\.)?mistral\.ai/.*$ |
DuckDuckGo | DuckDuckGo | ^(https?://duckduckgo[.]com).* |
Brave | Brave | ^(https?://search[.]brave[.]com).* |
Public IP ranges used for verification
For these brands, Oncrawl checks the requesting IP against the ranges the vendor publishes, so a spoofed user agent is not counted. Brands absent from this list are matched on the user agent alone.
We’re continuously improving our bot detection capabilities. If you have any suggestions or would like to see new bots added to the supported list, don’t hesitate to contact us.