Sample script for Linux
Here is a simple script that reads the last complete access log file, filter lines containing google, compress the file and send it to OnCrawl FTP.
The script can be adapted to any kind of servers.
You need to replace the FTP_* variables with real values.
This script also requires the main username and password associated with your account.
#!/bin/bash
FTP_SERVER='ftp.oncrawl.com'
FTP_USER='<YOUR ONCRAWL USERNAME>'
FTP_PASS='<YOUR ONCRAWL FTP PASSWORD>'
FTP_FOLDER='<PROJECT NAME>'
# Find the previous *complete* access file.
LOG_FILENAME='/var/log/apache2/access.log.1'
#LOG_FILENAME='/var/log/nginx/access.log.1'
GREP_WORD='google'
LOG_TMP_FILE="/tmp/access.log-$(date +%Y-%m-%d).${GREP_WORD}_only.gz"
grep -i $GREP_WORD $LOG_FILENAME | gzip > $LOG_TMP_FILE
curl -s -T $LOG_TMP_FILE "ftp://${FTP_SERVER}/${FTP_FOLDER}/" --user $FTP_USER:$FTP_PASS --ftp-ssl
rm $LOG_TMP_FILE
If your dedicated FTP space was created specifically for you by Sales or Customer Success Manager, you will need to make a slight change to this script. Replace the line that begins with "curl -s -T" with the following line (remove ${FTP_FOLDER}/):
curl -s -T $LOG_TMP_FILE "ftp://${FTP_SERVER}/" --user $FTP_USER:$FTP_PASS --ftp-sslLet's call this script oncrawl_upload_logs.sh and make it executable:
chmod +x oncrawl_upload_logs.sh
Now, let's execute it once to make sure it is working properly:
./oncrawl_upload_logs.sh
Using your favorite FTP client, connect to your OnCrawl FTP account, browse to your project directory, and check that your new file has been properly uploaded.
If that did not work, you can remove the -s option of curl to report any error.
Automate the execution with cron
Let's assume you have put the script at /home/philippe/local/bin/oncrawl_upload_logs.sh
With crontab I can automate the execution every day at 7AM. Let's start editing our crontab with:
$ crontab -e
Now I must add a new line to my crontab file to execute oncrawl_upload_logs.sh
0 7 * * * /home/philippe/local/bin/oncrawl_upload_logs.sh
Save the file, and you're set
Automate the execution in logrotate
If your webserver log files are rotated with logrotate, it is also possible to add a directive to logrotate to call our upload script right after the log file has been rotated.
Let's assume your are using nginx webserver. Now we can add the following lines to the file /etc/logrotate.d/nginx :
postrotate
/home/philippe/local/bin/oncrawl_upload_logs.sh
endscript
Other web server, other OS
It's possible to automate the upload with virtually any web server or OS, including services like CloudFlare. The sample script here is for Apache/Nginx on Linux because it's a very common use case. More examples will be added here in the future. If you need further assistance to automate you daily file upload to OnCrawl, please contact our support.
You can also find this article by searching for:
automatizar transferencia envío registros
automatiser le téléchargement des logs, envoi automatique des logs