Sometimes something goes wrong with a log file upload, bot hits are abnormally low, or regular trends in cross analysis look unusual. In these cases, you can check for yourself what is going on using OnCrawl's Log Manager Tools.
These tools are available from the project home page, at the top right of the screen, under Tasks. Click on "LOG MANAGER TOOLS".
Here you can follow up on the processing of uploaded log files.
What kind of information is available?
In this tool, we provide a lot of important information.
"File Processing" shows information about the steps in processing your files. First, OnCrawl reads the files and then extracts the data that is used in log monitoring. Some raw data appears in live log monitoring, but the full usable contents of the file are aggregated by day to allow for cross analysis with other data on the platform. You will have access to all data when the aggregation process is completed.
The second part of the tool displays graphs about the quality of the files OnCrawl has received. "Quality" is evaluated on based on three characteristics:
File compression: Are files received compressed?
Share of useful content: The number of lines showing a bot hit or a hit from a user coming from a search engine, compared to the lines in the file representing other types of hits
File upload frequency: The frequency at which new log files are made available to OnCrawl.
Each of these characteristics has its own status (no data, all good, optimizable and not optimized) and shows an historical analysis of the files OnCrawl has received and processed.
OnCrawl also provides an indication of the hits from fake Googlebots that it found in your log files.
The last part of the tool returns an explorable data table with information about each file. It contains the file name, date of upload, and information about the number of analyzed lines, filtered lines, and lines with errors. It also lists the file status.
You can click on the name of the file to see samples of each type of line for the file. This helps you quickly understand the data available in OnCrawl from that file.
How to verify that nothing is wrong with uploaded files
It's important to keep an eye on the trends in the log monitoring metrics. File sizes and bot hits/SEO visits should follow a regular line or curve, and there should be few erroneous lines.
You should not have warnings in the processing status column.
You should also pay attention to any spikes in the graphics for "Lines unprocessed: errors" and "Files unprocessed: errors".