Sample script for Linux

Here is a simple script that reads the last complete access log file, filter lines containing google, compress the file and send it to OnCrawl FTP.

The script can be adapted to any kind of servers.

You need to replace the FTP_* variables with real values.

#!/bin/bash

FTP_SERVER='ftp.oncrawl.com'
FTP_USER='<YOUR ONCRAWL USERNAME>'
FTP_PASS='<YOUR ONCRAWL PASSWORD>'
FTP_FOLDER='<PROJECT NAME>'

# Find the previous *complete* access file.
LOG_FILENAME='/var/log/apache2/access.log.1'
#LOG_FILENAME='/var/log/nginx/access.log.1'

GREP_WORD='google'
LOG_TMP_FILE="/tmp/access.log-$(date +%Y-%m-%d).${GREP_WORD}_only.gz"

grep $GREP_WORD $LOG_FILENAME | gzip > $LOG_TMP_FILE

curl -s -T $LOG_TMP_FILE "ftp://${FTP_SERVER}/${FTP_FOLDER}/" --user $FTP_USER:$FTP_PASS

rm $LOG_TMP_FILE

Let's call this script oncrawl_upload_logs.sh   and make it executable:

chmod +x oncrawl_upload_logs.sh

Now, let's execute it once to make sure it is working properly:

./oncrawl_upload_logs.sh

Using your favorite FTP client, connect to your OnCrawl FTP account, browse to your project directory, and check that your new file has been properly uploaded.
If that did not work, you can remove the -s  option of curl to report any error.

Automate the execution with cron

Let's assume you have put the script at /home/philippe/local/bin/oncrawl_upload_logs.sh  
With crontab I can automate the execution every day at 7AM. Let's start editing our crontab with:

$ crontab -e

Now I must add a new line to my crontab file to execute oncrawl_upload_logs.sh  

0 7 * * * /home/philippe/local/bin/oncrawl_upload_logs.sh

Save the file, and you're set

Automate the execution in logrotate

If your webserver log files are rotated with logrotate, it is also possible to add a directive to logrotate to call our upload script right after the log file has been rotated.
Let's assume your are using nginx webserver. Now we can add the following lines to the file /etc/logrotate.d/nginx  :


        postrotate
                /home/philippe/local/bin/oncrawl_upload_logs.sh
        endscript

Other web server, other OS

It's possible to automate the upload with virtually any web server or OS. The sample script here is for Apache/Nginx on Linux because it's a very common use case. More examples will be added here in the future. If you need further assistance to automate you daily file upload to OnCrawl, please contact our support.

You can also find this article by searching for:
automatizar transferencia envío registros
automatiser le téléchargement des logs, envoi automatique des logs

Did this answer your question?