All Collections
Log Monitoring
Log monitoring
How to automate the daily upload of log files on the FTP?
How to automate the daily upload of log files on the FTP?

This is a sample script for Apache server on Linux

Updated over a week ago

Sample script for Linux

Here is a simple script that reads the last complete access log file, filter lines containing google, compress the file and send it to OnCrawl FTP.

The script can be adapted to any kind of servers.

You need to replace the FTP_*  variables with real values.

This script also requires the main username and password associated with your account.

#!/bin/bash

FTP_SERVER='ftp.oncrawl.com'
FTP_USER='<YOUR ONCRAWL USERNAME>'
FTP_PASS='<YOUR ONCRAWL FTP PASSWORD>'
FTP_FOLDER='<PROJECT NAME>'

# Find the previous *complete* access file.
LOG_FILENAME='/var/log/apache2/access.log.1'
#LOG_FILENAME='/var/log/nginx/access.log.1'

GREP_WORD='google'
LOG_TMP_FILE="/tmp/access.log-$(date +%Y-%m-%d).${GREP_WORD}_only.gz"

grep -i $GREP_WORD $LOG_FILENAME | gzip > $LOG_TMP_FILE

curl -s -T $LOG_TMP_FILE "ftp://${FTP_SERVER}/${FTP_FOLDER}/" --user $FTP_USER:$FTP_PASS --ftp-ssl

rm $LOG_TMP_FILE

If your dedicated FTP space was created specifically for you by Sales or Customer Success Manager, you will need to make a slight change to this script. Replace the line that begins with "curl -s -T" with the following line (remove ${FTP_FOLDER}/):

curl -s -T $LOG_TMP_FILE "ftp://${FTP_SERVER}/" --user $FTP_USER:$FTP_PASS --ftp-ssl

Let's call this script oncrawl_upload_logs.sh   and make it executable:

chmod +x oncrawl_upload_logs.sh

Now, let's execute it once to make sure it is working properly:

./oncrawl_upload_logs.sh

Using your favorite FTP client, connect to your OnCrawl FTP account, browse to your project directory, and check that your new file has been properly uploaded.
If that did not work, you can remove the -s  option of curl to report any error.

Automate the execution with cron

Let's assume you have put the script at /home/philippe/local/bin/oncrawl_upload_logs.sh  
With crontab I can automate the execution every day at 7AM. Let's start editing our crontab with:

$ crontab -e

Now I must add a new line to my crontab file to execute oncrawl_upload_logs.sh  

0 7 * * * /home/philippe/local/bin/oncrawl_upload_logs.sh

Save the file, and you're set

Automate the execution in logrotate

If your webserver log files are rotated with logrotate, it is also possible to add a directive to logrotate to call our upload script right after the log file has been rotated.
Let's assume your are using nginx webserver. Now we can add the following lines to the file /etc/logrotate.d/nginx  :


        postrotate
                /home/philippe/local/bin/oncrawl_upload_logs.sh
        endscript

Other web server, other OS

It's possible to automate the upload with virtually any web server or OS, including services like CloudFlare. The sample script here is for Apache/Nginx on Linux because it's a very common use case. More examples will be added here in the future. If you need further assistance to automate you daily file upload to OnCrawl, please contact our support.

You can also find this article by searching for:
automatizar transferencia envío registros
automatiser le téléchargement des logs, envoi automatique des logs

Did this answer your question?