During staging or pre-production periods, it’s best to prevent Google from indexing a site that’s still being modified or tested. This usually involves preventing access to the website through a variety of means, such as using a proxy server.
Note: The robots.txt file and the meta robots tags on your pre-prod site should be the same as the ones you intend to use when you go live. This best practice helps avoid costly errors. In almost all cases, an htpasswd is sufficient protection for a pre-prod website.
Here’s how to allow the Oncrawl bot access to your staging or pre-production website.
Your pre-prod site is password protected
To crawl a site that is protected by a password, you will need to set up a crawl that provides the correct password to the site.
Scroll down and click Authentication to expand the section.
Tick the Enable HTTP authentication box.
Provide the username and the password used to access your site.
Optionally, you can select the type of authentication scheme (Basic, Digest, or NTLM).
To make it easier to recall what you're logging in, you can also provide an indication of the Realm that this login applies to (e.g. blog, CMS, admin…).
Your pre-prod site uses meta instructions aimed at bots (noindex, nofollow)
By default, the Oncrawl bot respects instructions to bots such as meta robots noindex and nofollow tags.
If your pre-production site uses these tags to keep bots away, you can tell Oncrawl to ignore them:
When setting up your crawl, go to the Crawl settings section and find Crawler behavior.
Tick Ignore noindex tags and/or Ignore nofollow tags, depending on the instructions used on your pre-prod site.
The Oncrawl bot will then crawl these pages and follow their links as if the tags were not there.
Note: these options only change how Oncrawl crawls your site. Remember to remove noindex and nofollow tags from pages that should be indexed before your site goes live.
Your pre-prod site uses IP restrictions
To crawl a site that limits access to a few whitelisted IP addresses, you will need to whitelist the IP addresses used by the Oncrawl bot.
From the Workspace screen, click on the Oncrawl IPs addresses section. This provides a list of all of Oncrawl’s IP addresses.
You can copy all IPs to your clipboard by clicking on the Copy all IPs button.
Whitelist all of these addresses, or ask your web development team to whitelist them, before launching your crawl.
Learn more about the IP addresses Oncrawl uses to crawl a website.
Your pre-prod site is located behind a proxy server
If your pre-prod site is located behind a proxy server, you can configure Oncrawl to crawl your site by overriding the DNS. Note that this option cannot be used with a JS crawl.
When setting up your crawl, make sure to click Extra settings at the top of the crawl settings page. The toggle button should be green if the extra settings are visible.
Scroll down and click DNS Override to expand the section.
Enter the IP address of the server you want to use and the corresponding domain.
Your pre-prod site uses a robots.txt to prohibit bots
You can override the robots.txt for the Oncrawl bot only using a virtual robots.txt file.
Scroll down and click Virtual robots.txt to expand the section.
Tick the Enable virtual robots.txt box.
Enter the domain for the pre-prod site and click the +.
Modify the virtual robots.txt you have just created to allow the Oncrawl bot access to your site.
If you need help, you can take a look at our documentation on virtual robots.txt files.