Log monitoring & AI bots
Set up, automate, and analyze your log data.
23 articles
Log Monitoring Setup Step-by-stepPrerequisite: An Oncrawl Log Monitoring Subscription
How many days of logs should I upload?Find out how many days of logs to upload for different uses in OnCrawl's analyses.
Log monitoring date rangesWhat dates are covered in the log monitoring report? How can you change the period or the level of detail?
[Tutorial] How to upload a log file to OnCrawlNeed help getting started with log file analysis? Here's one place to start!
How often should I upload new log files to OnCrawl?Log monitoring only works if it is constantly and continuously updated. How often should you upload new logs?
How to secure a FTP connection when uploading log files to Oncrawl?If you manually upload log files to Oncrawl via our FTP server, you must use a secure (FTPS) connection.
How to automate the daily upload of log files on the FTP?This is a sample script for Apache server on Linux
Server side log format configurationLearn how to setup your server log configuration regarding the type of server you are using.
Splunk integrationIf you're a Splunk user, the OnCrawl Splunk integration automates the uploading of indexed logs to your private, secure FTP space.
How to set up log monitoring if you use Amazon S3Do you use an Amazon S3 bucket to store your log data? Here's how to connect your logs to Oncrawl.
How to set up log monitoring if you use GCPDo you use a Google Cloud Storage bucket to store your log data? Here's how to connect your logs to OnCrawl.
How to set up Oncrawl log monitoring with AkamaiUsing Akamai and interested in using Oncrawl for your SEO log analysis? Here's what you need to know.
SEA & vertical botsThe SEA & Vertical bots dashboard provides information about hits by additional Google bots that is extracted from an analysis of your logs.
How to filter log linesDon't want to share everything? Want faster processing? Remove unnecessary lines from your logs before uploading to OnCrawl.
How to get the most out of live log monitoringHere's how to use the live log monitoring function in OnCrawl to see in near real time how Google explores your site
How to view bot hits and SEO traffic from different search enginesOncrawl supports log monitoring for Google, Bing, Baidu and Yandex
How to detect and analyze AI and LLMs bots hits using log monitoringLearn how to enable and analyze the detection of AI and LLM bots in Oncrawl using log monitoring.
Oncrawl log analyzer supported bot listThe following list includes all bots currently supported in the Oncrawl platform. The list is continuously updated to ensure accurate log monitoring.
How to reduce the size of log filesThe size of access log files can be easily reduced by 95% with pre-filtering and compression
How to check the quality and status of uploaded log filesThe OnCrawl Log Manager Tool is an interface that can help you understand and monitor your logs uploading and parsing processes.
How to calculate the number of resources needed for log analysisOnCrawl log analysis levels are based on the number of lines in your logs. Here's how to estimate how many lines you need.
Receive alerts when no new log data is receivedOncrawl will alert you by email if we don't have recent, usable data from your logs. Here's how to set up the alerts.
How to resolve log monitoring alertsOncrawl will alert you by email if we don't have recent, usable data from your logs. Here's how to resolve an issue if you receive an alert.