Why you should verify domain ownership in your workspace
Oncrawl aims to remain an ethical, good citizen of the web by respecting crawl limitations imposed by the owners of websites.
More importantly, Oncrawl helps you to protect your domain against server load mismanagement, robot.txt instruction overrides and other malicious bot activity.
This means that we require a domain to be verified before you can use restricted features such as:
Crawl speed: crawl speed is limited to 1URL/s for unverified domains (domains you do not have ownership or equivalent priviledges on).
Virtual robot.txt overrides: the robot.txt instructions can't be overriden and replaced with a custom robots.txt file on unverified domains.
Customize user-agent: You can modify this user-agent by providing a bot name and a full user-agent string.
Verifying domain ownership at workspace level with a third-party connected app
Owners and Managers can add verified domains to a workspace using a third-party connected application such as Google Analytics or Google Search Console.
In the workspace menu, click on Workspace and then on Admin.
To add a verified domain, first select a validation method.
Then, create an authentification for your Google Analytics or Google Search Console account.
This will then show you a list of domains you have access to through Google Analytics or Google Search Console. Select the domain or domains you want to verify. You can click on them individually, or use the selector elements to add several or all domains at once.
Click on “Add domain” to complete the verification process for the domains you selected.
What happens when a domain is verified
Domaines are verified within a workspace.
You can consult the list of verified domains in your workspace, in the admin settings.
When a domain is verified, users in your domain can unlock additional features in crawls.
This domain is verified for all crawls in all projects in the workspace where it is verified.
What to do if your domains can't be verified with Google Analytics or Google Search Console
You may have legitimate access to domains that you can't verify through Google Analytics or Google Search Console.
This might be the case in the following scenarios:
You want to crawl pre-prod or staging environment that is hosted on a subdomain owned by your host or by your CRM.
You want to crawl a website that doesn't have a Google Analytics or a Google Search Console tag on it.
You're working with an agency to audit a prospect's website that they've provided partial access to.
In these situations, please contact your Customer Success Manager or your Account Manager. We'll be able to help you validate this type of domain through other channels.
What to do if you want to include alternate Start URLs, or URLs in list mode that aren't on a verified domain in your workspace
Oncrawl allows you to provide multiple Start URLs for a crawl in spider mode, or to provide a list of URLs to crawl in list mode. There are many reasons you might want to include URLs that aren't in a domain you've verified.
If any of the URLs in your URLs lists, or in the alternate Start URL field is not from a verified domain, you cannot run a crawl using crawl features that are restricted for unverified domains.
To launch this crawl, you will need to:
In your crawl profile settings, remove the unverified alternate Start URLs or the unverified URLs in your URL list
OR disable the restricted features:
Remove or disable any virtual robots.txt overrides
Reduce the crawl speed to 1 URL per second or slower
Under Crawl bot, choose the Oncrawl desktop or mobile crawler, instead of the full custom user agent.
OR verify the additional domains in your workspace level using Google Analytics or Google Search Console