Oncrawl monitors a comprehensive range of automated traffic to give you complete visibility into how it interacts with your site. This page lists every crawler, fetcher, agent and referral source currently supported in the platform.
If you would like to activate monitoring for a specific agent, please contact your dedicated Customer Success Manager, Account Manager, or reach out through the in-app chat.
Crawlers, fetchers and agents
Not every automated request behaves the same way in your logs, and the difference matters when you read them:
A crawler requests many URLs on its own schedule, follows links, and comes back. This is the behaviour you plan crawl budget around, and the one where crawl frequency and depth are meaningful.
A fetcher requests a single URL because someone just asked for it: a prompt, a share, a link preview. It does not follow links and it does not come back on a schedule, so a spike in fetches reflects user demand rather than a crawl decision.
An agent browses your site on behalf of a user to complete a task: it navigates from page to page like a person would, often with an ordinary browser user agent and from IP addresses that are not the vendor's. Agents that sign their requests can still be identified, see Signed AI agents.
Most brands operate several of these. Each category below is labelled accordingly.
Brands covered
Ai2, Amazon, Apple, Baidu, Bing, Brave, ByteDance, Claude, CommonCrawl, DuckDuckGo, Google, Ibou, Meta, Mistral, OpenAI, Perplexity, Yandex
What the categories mean
Category | Name | Type | What it covers |
| Search engine crawlers | Crawler | Organic search crawlers. Their hits are aggregated at the search engine level, which is what powers newly crawled pages, active pages and fresh rank. |
| Paid search crawlers | Crawler | Check landing pages and product feeds for advertising. |
| Vertical crawlers | Crawler | Dedicated to a single surface: images, video, news, stores or apps. |
| AI search crawlers | Crawler | Improving search result quality and page indexation. They crawl asynchronously, with clear patterns (typically one visit per URL per day), and far fewer pages than Googlebot. |
| AI user fetchers | Fetcher | Real-time content fetching for user answers and citations. They fetch a single page on demand to answer a prompt, without following links. |
| AI training crawlers | Crawler | Content scraping for model training. They have no crawl budget concept and do not render JavaScript. |
| AI agents | Agent | Browse your site on behalf of a user to complete a task, page after page, with a regular browser user agent. They are identified by the signature they send, not by their user agent. |
The three AI categories are covered in depth in AI bots explained: what powers platforms like ChatGPT?
Supported crawlers, fetchers and agents
Where a desktop and a mobile entry share the same user-agent pattern, they are not duplicates: the parser tells them apart with an additional test on the user agent.
Search engine crawlers
Brand | Label | User-agent pattern |
Baidu | Baidu mobile | .*\(.*(Phone|Android).*\).*(?<bot>Baiduspider(-render)?/[0-9.]+) |
Baidu | Baidu desktop | (?<bot>Baiduspider([+]|(-render)?/[0-9.]+)) |
Bing | Bing mobile | (?<bot>[Bb]ingbot/[0-9.]+) |
Bing | Bing desktop | (?<bot>[Bb]ingbot/[0-9.]+) |
Brave | Brave Bot | (?<bot>BraveBot/[0-9.]+) |
DuckDuckGo | DuckDuckBot | (?<bot>DuckDuckBot/[0-9.]+) |
Google mobile | (?<bot>Googlebot/[0-9.]+) | |
Google desktop | (?<bot>Googlebot/[0-9.]+) | |
Yandex | Yandex mobile | (?<bot>(YandexBot|YandexAccessibilityBot|YandexMobileBot)/[0-9.]+) |
Yandex | Yandex desktop | (?<bot>(YandexBot|YandexAccessibilityBot)/[0-9.]+) |
Paid search crawlers
Brand | Label | User-agent pattern |
Baidu | Baidu ads | (?<bot>Baiduspider-ads/[0-9.]+) |
Bing | Bing ads | (?<bot>adidxbot/[0-9.]+) |
Bing | Bing ads mobile | (?<bot>adidxbot/[0-9.]+) |
Google AdsBot desktop | ^(?<bot>AdsBot-Google)[^-] | |
Google AdsBot mobile | (?<bot>AdsBot-Google-Mobile); | |
Google Adsense | ^(?<bot>Mediapartners-Google)$ | |
Google Adsense mobile | (?<bot>Mediapartners-Google/[0-9.]+) | |
Google HotelAds Verifier | ^(?<bot>Google-HotelAdsVerifier/[0-9.]+)$ | |
Meta | Meta External Ads | (?<bot>[Mm]eta-[Ee]xternal[Aa]ds/[0-9.]+) |
OpenAI | OpenAI AdsBot | (?<bot>OAI-AdsBot/[0-9.]+) |
Yandex | Yandex ads | (?<bot>YandexAdNet/[0-9.]+) |
Vertical crawlers
Brand | Label | User-agent pattern |
Baidu | Baidu images | (?<bot>Baiduspider-image/[0-9.]+) |
Baidu | Baidu news | (?<bot>Baiduspider-news/[0-9.]+) |
Baidu | Baidu video | (?<bot>Baiduspider-video/[0-9.]+) |
Google Extended | (?<bot>Google-Extended) | |
Google images | ^(?<bot>Googlebot-Image/[0-9.]+)$ | |
Google AdsBot mobile apps | ^(?<bot>AdsBot-Google-Mobile-Apps)$ | |
Google news | ^(?<bot>Googlebot-News)$ | |
GoogleOther | (?<bot>GoogleOther) | |
Google Speakr | ^(?<bot>Google-speakr)$ | |
Google store desktop | (?<bot>Storebot-Google/[0-9.]+) | |
Google store mobile | (?<bot>Storebot-Google/[0-9.]+) | |
Google video | ^(?<bot>Googlebot-Video/[0-9.]+)$ | |
Meta | Facebook External Hit | (?<bot>facebookexternalhit/[0-9.]+) |
Yandex | Yandex images | (?<bot>YandexImages/[0-9.]+) |
Yandex | Yandex images mobile | (?<bot>YandexImageResizer/[0-9.]+) |
Yandex | Yandex news | (?<bot>YandexNews/[0-9.]+) |
Yandex | Yandex video | (?<bot>YandexVideo/[0-9.]+) |
AI search crawlers
Brand | Label | User-agent pattern |
Amazon | Amazon SearchBot | (?<bot>Amzn-SearchBot/[0-9.]+) |
Apple | Applebot | (?<bot>Applebot/[0-9.]+) |
Claude | Claude SearchBot | (?<bot>Claude-SearchBot/[0-9.]+) |
Ibou | IbouBot | (?<bot>IbouBot/[0-9.]+) |
Meta | Meta Web Indexer | (?<bot>[Mm]eta-[Ww]eb[Ii]ndexer/[0-9.]+) |
Mistral | Mistral Index | (?<bot>MistralAI-Index/[0-9.]+) |
OpenAI | OpenAI SearchBot | (?<bot>OAI-SearchBot/[0-9.]+) |
Perplexity | Perplexity Bot | (?<bot>PerplexityBot/[0-9.]+) |
AI user fetchers
Brand | Label | User-agent pattern |
Amazon | Amazon User | (?<bot>Amzn-User/[0-9.]+) |
Claude | Claude User | (?<bot>Claude-User/[0-9.]+) |
DuckDuckGo | DuckAssistBot | (?<bot>DuckAssistBot/[0-9.]+) |
Google Agent | (?<bot>Google-Agent) | |
Google Agent Mariner | (?<bot>GoogleAgent-Mariner) | |
Google Agent Search | (?<bot>GoogleAgent-Search) | |
Google CloudVertexBot | (?<bot>Google-CloudVertexBot) | |
Google Gemini Deep Research | (?<bot>Gemini-Deep-Research) | |
Google Gemini Bot (Google) * | ^(?<bot>Google)$ | |
AI Mode Bot (Google-Lens) * | (?<bot>Google-Lens) | |
Google Notebook LM | (?<bot>Google-NotebookLM) | |
Meta | Meta External Fetcher | (?<bot>[Mm]eta-[Ee]xternal[Ff]etcher/[0-9.]+) |
Mistral | Mistral User | (?<bot>MistralAI-User/[0-9.]+) |
OpenAI | OpenAI ChatGPT User | (?<bot>ChatGPT-User/[0-9.]+) |
Perplexity | Perplexity User | (?<bot>Perplexity-User/[0-9.]+) |
* Not documented by the vendor. Oncrawl tracks these from observed traffic, so their user agent and behaviour can change without notice.
AI training crawlers
Brand | Label | User-agent pattern |
Ai2 | AI2Bot | (?<bot>AI2Bot) |
Amazon | AmazonBot | (?<bot>[Aa]mazon[Bb]ot/[0-9.]+) |
Apple | Applebot Extended | (?<bot>Applebot-Extended/[0-9.]+) |
ByteDance | Bytespider | (?<bot>Bytespider) |
ByteDance | Bytespider image | (?<bot>imageSpider) |
Claude | Claude Bot | (?<bot>ClaudeBot/[0-9.]+) |
CommonCrawl | CCBot | (?<bot>CCBot/[0-9.]+) |
Meta | Meta ExternalAgent | (?<bot>[Mm]eta-[Ee]xternal[Aa]gent/[0-9.]+) |
Mistral | Mistral Training | (?<bot>MistralAI-Training/[0-9.]+) |
OpenAI | OpenAI GPTBot | (?<bot>GPTBot/[0-9.]+) |
AI agents
Brand | Label | Signature-Agent header |
Meta | Meta Muse * |
|
OpenAI | ChatGPT Agent |
|
* Not documented by the vendor. Oncrawl tracks these from observed traffic, so their user agent and behaviour can change without notice.
Signed AI agents
AI agents that browse for a user look like a regular browser: a Chrome user agent and residential or cloud IP addresses. The user agent and the IP check cannot identify them. Some vendors sign these requests with Web Bot Auth, an emerging IETF standard: the request carries a Signature-Agent header naming the vendor, plus a cryptographic signature.
When a request matches no user-agent pattern, Oncrawl reads its Signature-Agent header and attributes the hit to the matching agent listed above. This requires the header to be present in your log lines. As an option, Oncrawl can also verify the signature against the keys the vendor publishes: this requires the Signature and Signature-Input headers in your logs too, and a hit whose signature is present but invalid is then discarded. Contact us to enable it.
Vendors only sign page navigations. Images, scripts and background requests made during the same session are not signed and are not attributed to the agent.
Supported sources of referral traffic
Visits are attributed to these sources by matching the referer of the request. When a request has no referer, Oncrawl falls back to the utm_source parameter of the requested URL: its host name is matched against the same patterns, so ?utm_source=chatgpt.com is attributed to OpenAI.
Brand | Label | Referer pattern |
^(https?://(www|news)[.]google[.]|android-app://com\.google\.android\.(apps\.genie\.geniewidget|googlequicksearchbox)).* | ||
Gemini | ^(https?://gemini[.]google[.]com).* | |
Bing | Bing | ^https?://(www[.])?bing[.].*$ |
Baidu | Baidu | ^https?://((www|news)[.])?baidu[.].*$ |
Yandex | Yandex | ^https?://(www[.])?yandex[.].*$ |
OpenAI | Openai | ^https?://((chat[.])?openai|chatgpt)[.]com.*$ |
Claude | Claude | ^https?://([\w.-]+\.)?(claude\.ai|anthropic\.com)/.*$ |
Perplexity | Perplexity | ^https?://([\w.-]+\.)?perplexity\.ai/.*$ |
Mistral | Mistral | ^https?://([\w.-]+\.)?mistral\.ai/.*$ |
DuckDuckGo | DuckDuckGo | ^https?://([\w.-]+\.)?(duckduckgo\.com|duck\.ai)(/.*)?$ |
Brave | Brave | ^(https?://search[.]brave[.]com).* |
Public IP ranges used for verification
For these brands, Oncrawl checks the requesting IP against the ranges the vendor publishes, so a spoofed user agent is not counted. Brands absent from this list are matched on the user agent alone.
We’re continuously improving our bot detection capabilities. If you have any suggestions or would like to see new bots added to the supported list, don’t hesitate to contact us.