Akamai provides global edge technology including CDNs amongst many other services.
Services hosted on Akamai generate logs.
There are multiple ways to use these logs for SEO analysis with Oncrawl. We'll look at two of them below.
We are interested in analysis logs from HTTP(S) traffic. No matter which method you use, we recommend using the W3C standard log format for analysis within Oncrawl. This will make setup easier.
Using LDS (Log Delivery Service)
What you'll do: Configure Akamai edge access logs to be pushed to an Akamai cloud storage service (or FTP server) and pulled into any log-processing/visualization tool.
In this scenario, Oncrawl will mirror logs from Akamai hosted FTP service.
To fine tune log delivery for Oncrawl, we suggest filtering the log events of interest. This allows you to share only events related to organic traffic and search engine bots with Oncrawl. You can use a configuration similar to the one above.
You will need to include the following lines:
User-agents:
APIs-Google*
Mediapartners-Google*
AdsBot-Google*
FeedFetcher-Google*
*DuplexWeb-Google*
*Google Favicon*
*Storebot-Google*
*googleweblight*
*Googlebot*
Referrers:
Once this is up and running, Oncrawl can mirror through FTP or SFTP depending on your needs. You'll need to generate credentials for Oncrawl and contact our CSM team to set up the mirroring.
Using DataStream (Live Logs)
What you'll do: Configure a DataSource using DataStream and ensure that events from your edge infrastructure are streamed to an Amazon S3 bucket.
Oncrawl will be able to process aggregated events streams as soon as they are deposited in the S3 bucket using an SNS topic queue.
Once the DataStream is up and running, follow the instructions to set up the SNS topic queue. Oncrawl will automatically receive and process log files when made available to S3. With this method, Oncrawl can provide monitoring at a very low latency using our Live Log monitoring solution.
If you need help along the way, our CSM team is available to provide support.