Customize your user-agent in Oncrawl

Learn how to change the bot identity used to crawl your website.

Updated over a week ago

Oncrawl lets you choose the bot identify that it uses when crawling.

Oncrawl provides default user-agents for a desktop and a mobile bot belonging to Oncrawl.

You can modify this user-agent by providing a bot name and a full user-agent string. For verified domains, you also have the option of customizing the full user-agent.

Default Oncrawl user-agent for both desktop and mobile

When choosing a default user-agent, Oncrawl will allow you to provide a custom name for the bot that will be used to crawl your site.

Because the Oncrawl bot does not ignore rules directed at bots, changing the bot name will make it follow rules for bots with the name you give it.

You may want to do this if you're trying to test rules on your site for different bots, such as rules targeting Google's crawler.

You might also find this interesting to test if your site restricts access to other bots based on their name.

You'll need to modify this in the crawl settings before running your crawl. Here's how.

Modifying the bot name in the crawl settings

From the project page, click Set up new crawl to open the crawl settings page.

Click on Crawl bot to expand the section.

The field Bot name is required. By default, we name our bot OnCrawl, but you can replace this with anything you'd like.

The full user agent that will be used is displayed on the right.

Modifying full user-agent to crawl your domains

Oncrawl offers the ability to customize the full user-agent string on the domain that you own if you have verified it. (Read more about verified domains here.)

Modifying the full user-agent can help meet security requirements, or better approximate interactions between your site and a bot (like Googlebot) to which you provide specific rendered content or custom instructions.

  • Select crawl with custom user-agent

  • Provide a bot name (such as "Googlebot")

  • Paste or type the full user-agent string of the bot identity you want to use

For example, you may want to use one of the official Googlebots according to the information provided by Google itself: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

Here is an example:

  • Bot name: Googlebot

  • Full user-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Did this answer your question?