This article will take you through the steps to set up log monitoring with Oncrawl:

Depending on how your website is set up, log files can be kept and stored at different locations and by different tools including: CDNs, load balancers, caches, servers...

For SEO log monitoring, you will need the log files created at the point where a user first interacts with your site. This might mean you need the log files from your CDN or your load balancer, for example, rather than your server.

Depending on how your website is set up, you might need multiple files to cover all of your site.

Make sure you have the files for the parts of your site you want to monitor, such as mobile pages or subdomains.

You will need to make sure Oncrawl can read your log files. To do so, you'll need a log file and a basic text editor, such as Notepad (Windows) or Notes (Mac).

First, check the file extension.

Open the file in your text editor. This is what your log file looks like. Oncrawl's job is to make sense of this. Yours is to make sure it contains all the information Oncrawl needs to extract.

Oncrawl will be looking for organic visits and for Googlebot hits. Here's what they might look like:

Sample log line for a Googlebot hit

You can see "Googlebot" in the user agent:

Sample log line for an organic visit

You can see the "https://www.google.es/" as the referer:

You should be able to confirm the following points by looking at your log file:

Your log lines might look different from the examples above if your website uses a different type of server. Oncrawl regularly analyzes logs from IIS, Ngix, and Apache servers, and can support any server type that will provide a log file in JSON format.

Learn how to configure the right log format when using Apache or Ngnix here

The following information is required:

The following information is required if your site uses HTTPS:

The following information is required if your log files contain information for multiple subdomains:

The following information is optional but highly recommended:

Your company or client might not want to provide full log files to Oncrawl. If required by your in-house policies, you can filter your log files to remove lines that are not bot hits or SEO (organic) visits.

Find more on how to filter log lines here.
​

In order to prepare an automatic analysis of your log files, we'll ask you a bunch of questions during the setup. Make sure you know how to answer them. You might need to ask your IT department for help.

At this point, you're ready to activate log monitoring in the application if you haven't done so already. You will need to have the log monitoring option included in your Oncrawl subscription.

From your project page, click on "ADD DATA SOURCES".

In the first tab of the data sources page, click on "ACTIVATE LOGS MONITORING".

Use the question checklist from the previous step ("3. Prepare your parsing questions") to answer all questions in steps 1 ("Configure your needs") and 2 ("Logs completeness").

Now you are ready to upload your first log files to Oncrawl.

You can leave the page in the application and come back when you're ready.

Before going any further, make sure you know where your log files are stored, and that you can access them.

You may need to negotiate this with your IT department, or ask them to set up a solution that places a copy of log files in a location you have access to every day.

Keep in mind that the location of your log files depends on how your site and server are set up. The logs you need may be stored in multiple locations.

If your logs are stored on a server, the most common location is /var/log .

If your server is used for multiple websites, make sure you are able to tell which logs are for which site. If the server logs requests for all sites in the same file you will need to:

Next, set up your upload method.

There are two methods available:

Using this method, you don't have to provide your files. Oncrawl will come get them using one of our available connectors.

Please contact us to obtain and set up a connector. We support connectors for:

You should also contact us if you prefer to set up a data stream for:

To manually upload files to Oncrawl, you will need to connect to your private, protected FTP space and place your files in the folder for your project.

First, you'll need to compress your log files, or "zip" them. You can use any program that produces one of the following common formats:

Make sure that your network firewall is open for FTP connections.

Use a FTP client solution such as FileZilla to connect. You may need the following information:

Make sure that your connection is secured (FTPS). If your FTP client does not use TLS by default, the way FileZilla does, you will need to to turn the option "Explicit FTP over TLS".

Oncrawl does not use an authentification key for the FTPS connection.

Once connected, you will see folders for each project in your account:

Open the folder for your project and drop the zipped file(s) there.

You're (almost) done!

Learn how to secure your FTP connection here (this makes sure you're using FTPS)

Learn how to automate the daily log files uploading here

Your log files are now available.

Return to the log setup page in the "Add Data Sources" section of the Oncrawl application. (If you've closed this page, you can return by clicking on "ADD DATA SOURCES" from the project page, and making sure you're looking at the "Log Files" tab.)

You should see a message that Oncrawl has found the files you uploaded. The number of files found by Oncrawl should be the same as the number of files you uploaded.

Click "I'VE UPLOADED ALL FILES".

This will take you to step 4 ("Check logs format"). This screen reports on how Oncrawl has broken down each line in your file into separate informational elements, and what information about your log files it has determined.

It's normal to see "Parse failed" in the first line ("Result") at first. You will not be able to continue until this says "Everything seems OK".

To change a "Parse failed" into "Everything seems OK", you will need to correct any errors in the "Issues", the "HTTP / HTTPS analysis" and the "Subdomain analysis" sections. (You can go ahead with a parse schema even if there are still warnings, as long as the interpretation by Oncrawl seems ok to you.)

The "Issues" section lists the high-level problems found.

You can have warnings even if the result is "Everything seems OK":

The "HTTPS / HTTPS analysis" section tells you how Oncrawl will handle the difference between HTTP and HTTPS in your log files. For example, if you see: "Error: HTTP scheme is required but could not be extracted", it means:

The "Subdomain analysis" section tells you how Oncrawl will handle possible different subdomains in your log files. For example, if you see: "Warning: HTTP host could not be extracted. Full URLs rebuilt from default URL: https://www.yoursite.com", it means:

The sections "Parse sample", "OK lines", "Filtered lines" and "Error lines" are to give you an idea of what Oncrawl was able to find in your log files. If you're very familiar with your site, this information can help you confirm that files have been correctly parsed.

To correct errors and warnings, move down to the "Configure logs parser" section and switch to "Manual".

For each element found by Oncrawl, use the drop-down menu at the top to choose the type of information the element represents. You probably will not need to use all of the items in the drop-down menu.

You can also click on the "Advanced" tab. This is where you can turn off IP verification for Googlebots by unticking the "Check Google IP" checkbox, or set a timezone if your server doesn't provide one.

Click "CHECK LOGS FORMAT" when you're done.

If you still see "Parse error" or warnings, click "CONFIGURATION IS NOT OK" at the bottom of the page. This will allow you to edit the parsing again.

Repeat this cycle until you have the results "Everything seems OK" and you no longer have warnings.

If everything is okay and you no longer have any warnings, click "CONFIGURATION IS OK"

In case of difficulties, you can contact us by chat. Tell us you're dealing with a "parse error".

Oncrawl's Log Manager tool allows you to track the processing of your log files.

You can find the "LOG MANAGER TOOL" button on the project page next to the "ADD DATA SOURCES" button.

"File Processing" displays information about the different steps in processing your files. It shows the time the last file was received, the timestamp of the last useful line (either a Googlebot hit or a hit coming from a search engine page), as well as the status of files in this process queue.

Some raw information is used for live log monitoring ("Live events"), but the full information from a file is aggregated by day ("Aggregated data") to allow it to be used in cross analysis with other information on the platform.

Information from both processing queues is available.

In the second section on this page, the Log Manager Tool displays graphs to track trends in the quality of your log files and in the regularity with which they are uploaded and processed.

Finally, the tool shows an explorable data table with information including File Name, Deposit Date, File Size, SEO visits and Bot lines, Filtered Lines, and Errors.

You can click on a file name to see examples of data from that file. This gives you a better idea of how Oncrawl interprets your log data.

Log monitoring is now set up correctly for your project.

Click on the "SHOW LOGS MONITORING" from your project page, and begin your logs analysis.

This will take you to the "Log Analysis" tab in the analysis sidebar. You can also access these reports from any analysis in your project.

You can also modify your crawl profiles to cross-analyze crawl data and log data in future analyses:

You can also find this article by searching for:
paso a paso cómo comenzar con la configuración de monitoreo
mise en place du suivi des logs, étape par étape