If you store your log data in an Amazon S3 bucket, we can help you automate uploading log data to OnCrawl.

Please contact us (use the blue Intercom button at the bottom right of your screen) to put automated daily log file uploading in place.

You will need the following information:

  • Bucket name
  • Prefix (This is the name of the directory in the bucket in which your log files are found.)
    Note: your website might use more than one prefix. Please provide all of the prefixes for log monitoring data on the domains and subdomains you want analyzed.
  • An access key
  • A secret key*

*We recommend encrypting your AWS secret key before sending it to us.

This can be done very simply using Keybase.io. Once encrypted, your secret key can only be decrypted by a single recipient defined at the time of encryption.

For this, you must:

1. Paste your secret key in the "Message to encrypt" text box on this page: https://keybase.io/encrypt#oncrawl

2. Click "Encrypt".

3. Send us the entire result found in the "The secret message" text box.

This is a PGP message that can only be decrypted by the owner of the https://keybase.io/oncrawl account.

Please make sure that the logs stored in your bucket under the provided prefixes:

  • Contain both SEO (organic) visits and bot hits,
  • Contain logs only for the site you would like to analyze. Logs for non-production sites, partner sites, or domains not included in your project must not be included in the provided prefixes.

This information will allow us to configure the connection for you.


To make daily uploads of log files easy for you, our platform retrieves new log files each day in order to update your log monitoring regularly, and ensures that there are no gaps in your log data in OnCrawl.

How long does it take to process log files?

When setting up log monitoring:

  • Processing time will depend on the amount of data that needs to be handled. This will take longer the first time you connect your site's log data: we will collect log data for the period of up to 90 days before you activated log monitoring, if it is available.
  • Provide correct and complete information. It will take longer to receive your initial log data if our first attempts with incorrect information don't succeed.

When log monitoring is already in place:

  • New log monitoring data is automatically processed at regular intervals and is typically available within 24 hours of the logged activity.

Are you interested in log analysis and using Amazon S3 to store your log data? Let us know--we're happy to help!

You can also find this article by searching for:
automaci├│n, automation, Amazon S3, AWS S3, AS3

Did this answer your question?