We let you specify the name of the bot that will be used to crawl your site.
Because the Oncrawl bot does not ignore rules directed at bots, changing the bot name will make it follow rules for bots with the name you give it.
You may want to do this if you're trying to test rules on your site for different bots, such as rules targeting Google's crawler.
You might also find this interesting to test if your site restricts access to other bots based on their name.
You'll need to modify this in the crawl settings before running your crawl. Here's how.
Modify the bot name in the crawl settings
From the project page, click "Set up new crawl" to open the crawl settings page.
Click on "Crawl bot" to expand the section.
The field "Bot name" is required. By default, we name our bot "OnCrawl", but you can replace this with anything you'd like.
The full user agent that will be used is displayed on the right.
That's it!
Note: To prevent abusive crawling, the full user-agent cannot be modified. If your site blocks the Oncrawl user-agent directly, you should create an exception for the Oncrawl bot.