Hello and welcome to OnCrawl. You're about to get the most comprehensive report of your SEO performances, so let's get started and launch your first crawl.
First, you have to create a project. For example, the name of your website, and the start URL, which is the first page that will get analyzed by our robot.
Once your project is created, you can connect your Google Account to be able to verify your ownership of the project and to access your advanced settings. This is important if you want to modify your crawl speed.
Now you're ready to set up a new crawl.
If you're ok with the default configuration, then click on "Launch a new crawl". Otherwise, select "Edit settings" to get access to a tailored configuration.
Start by verifying your Google Account and your start URL. Notice that you can add several start URLs. This is useful if you have different versions of your website and you want to analyze how they are linked between each other.
Define your crawl limits. Insert a maximum number of URLs you want to analyze to avoid spending all of your quota in one single analysis. And define the maximum depth you want our robots to reach. This will help you stay away from spider traps.
If you are crawling a mobile version of your website, you can change the user-agent to mobile.
Speed up our robots by increasing the crawl speed. Just make sure that your website can handle it!
Don't forget to crawl your subdomains and your URLs with parameters to get a complete report.
Overwrite your robots.txt with a virtual one to allow or disallow some parts of your website.
If you need to crawl a website that is hosted on a different server, you can assign a different IP to any given host. For each line, separate the host from the IP.
Authorize additional HTTP headers by inserting them in this box.
You can also analyze a website in pre-production by using our HTTP authentification.
If you have a plan higher or equal to the Business plan, you'll be able to generate a combined traffic analysis with your Google Analytics data.
You have the possibility to compare the crawl you're about to launch with the last one. This way you can see changes you've made.
Finally, these two options will help you schedule your crawls, either by selecting a specific date, or by choosing a recurrence.
Name your crawl configuration to make sure you can find it easily the next time you sign in.
Now you're ready to launch your crawl!