Two kinds of problems with log monitoring data can trigger an alert in Oncrawl:
We haven't received any new log files within a given period of time
We've received files, but couldn't detect any recent organic visits or search engine bot hits.
When you set up alerts, you set the period of time, or the threshold period, that we use to determine whether the last file or the last useful line in your files is recent enough.
To receive these alerts, set them up following the instructions in this article.
No new files within the threshold period
If we haven't received any files from you within the period you defined, you'll receive an alert.
This can occur:
If you forgot to manually upload a file.
If something in your automated process has stopped working.
Resolving alerts for no new files
This alert will be resolved when we once again receive a new log file for your project. At that time, we'll send you an email confirming that this issue has been resolved.
If you use a connector or an automated process, this can occur when changes are made to your server, your tools, or your accounts that are used as part of this process.
You may need to review each step of the set-up or the process to discover what went wrong.
If you need to check the last file we received from you:
Click on Log Manager Tools from the project home page.
Look at the dates for Last file received in the "File processing" block.
Additional details about this last file received can be found further down the page.
Please note: Until we receive a new file from you, you will not be alerted again.
No useful log lines for the threshold period in the latest files
If we can't find any lines that represent hits with a monitored search engine as a referrer (organic or SEO visits) or any hits from a monitored search engine bot (bot hits) that occur within the period you defined, you'll receive an alert.
These types of lines -- SEO visits and bot hits -- are what we call useful lines.
This can occur if:
The log files you send us are empty: that is, they don't contain any data on this type of hit.
The log files you send us contain data, but can't be parsed. This means that the format of your log files has changed: the parsing you set up with Oncrawl no longer matches the content of your file.
Resolving alerts in the case of no recent useful lines
When Oncrawl once again detects useful lines within the threshold period, we'll send you an email confirming that this issue has been resolved.
But this might not be as straightforward as when no new files have been received. To uncover more information about the problem that will help you find a solution, you can use the Log Manager Tools:
Click on Log Manager Tools from the project home page.
Make sure that your recent file records recent hits
Oncrawl might have received a file from you two hours ago, but the hits in it are 8-12 hours old. If your threshold is less than the delay to the most recent hit (in this case: 8 hours), this situation will have triggered an alert. There are no hits known to Oncrawl within the threshold you set.
You can verify this by making sure that Oncrawl has recently received a file and that the hits in that file are also recent:
Check the date of the most recent file
Check dates of hits recorded by recent file
Here, for example, there is a 4-hour gap between the most recent event in the file, and the time when Oncrawl received the file from you.
In some cases, you might always encounter the situation described above, where despite frequent uploads, there is a significant delay between when the lines were recorded, and when the file becomes available to you.
Check whether this situation is normal:
Scroll down the page to the Files processed section
Check the dates of the most recent files
Check dates of hits recorded by recent files
In many cases, a long delay is not normal. However, if it is and if you can't reduce the delay, then you should increase the alerting threshold. This will ensure that you receive alerts only when they indicate that something is not working normally.
If you don't normally have a significant difference between when the file was uploaded and the dates and times of the hits it contains, then you'll need to figure out what caused the delay between when the log lines were recorded, and when you obtained and uploaded the file.
Make sure that your recent files contain any lines
Oncrawl might have received a file from you that contains 0 lines, or a file in which 100% of the lines have been filtered. The size of these files will likely be unusually small.
Check the size and contents of the file in the Files processed section of the Log Monitoring Tools
Click on the name of the file to view more details
Check the total number lines found in the file
This can happen if you pre-filter your log files before sharing them with Oncrawl, using a filter that is too aggressive. In other words, you might have removed search engine bot hits and organic visitors. In this case, you can adjust your filtering.
This can also occur if your server setup has changed. There are often multiple components that record logs. The log files you are providing might not record user interactions (from bots and visitors), but instead only indicate the interactions that a CDN or other upstream component passes to your server. You may need to change how you collect your log files.
Make sure that your recent files contain useful lines
Oncrawl might have received log files that contain lines pertaining to website traffic, but no "useful" lines from search engine bots or users.
To diagnose this:
Check the number of SEO visits and bot hits in the file
Check the number of filtered lines
Verify whether or not you filter your log files before sending them to Oncrawl
Often, this can occur if you pre-filter the log lines you share with Oncrawl. A filter that removes bot hits and organic visits is too aggressive, and needs to be modified.
If this isn't a filtering issue, it is most likely a parsing issue. Parsing issues can occur if there were recent changes to your server or your server environment. If you need to modify how your files are parsed by Oncrawl, please reach out to us using the chat button at the bottom right of your screen and we'll be happy to help.
Please note: Until we begin to detect useful lines within the threshold period, you will not be alerted again about missing useful lines.
However, if we stop receiving log files from you, you will receive a new alert for the missing files.