Apache

Recommended format:

LogFormat "%{HOST}i:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined_host_ssl
  • %{HOST}i : hostname of the query
  • %p : server port
  • %h : client IP
  • %l : remote logname. Not used by OnCrawl by standard in log formats
  • %u : remote user. Not used by OnCrawl by standard in log formats
  • %t : request date time
  • %r : first line of request
  • %>s : response status code
  • %b : size of the response in bytes
  • %{Referer}i : referer
  • %{User-Agent}i : User agent

Full documentation available here.

Nginx

Recommended format:

log_format combined_vhosts '$host:$server_port $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';

Reference documentation:

Nginx log_format
Nginx variables

IIS

Recommended formats: W3C Extended Log File Format or NCSA Common Log File Format.

The ISS Log File Format is not accepted.

Additional information (Microsoft)

W3C Extended Log File Format

Example format containing time, client IP address, method, URI stem, protocol status and protocol version:

#Software: Internet Information Services 6.0 
#Version: 1.0
#Date: 2001-05-02 17:42:15
#Fields: time c-ip cs-method cs-uri-stem sc-status cs-version
17:42:15 172.16.255.255 GET /default.htm 200 HTTP/1.0 

NSCA Common Log File Format

The following example lists these properties:

  • IP address
  • Remote log name of the user (blank in this example and represented by a "-" )
  • Domaine/user
  • Date and time
  • Command
  • Status code returned
  • Bytes of data sent
172.21.13.45 - Microsoft\fred [08/Apr/2001:17:39:04 -0800] "GET /scripts/iisadmin/ism.dll?http/serv HTTP/1.0" 200 3401

CloudFlare

For those using CloudFlare, you need to have an Enterprise plan in order to access your (raw) log files. Then, you need to activate Enterprise Log Share.
Also, you can set up a daily upload of your log files to our FTP (with the credentials you received by email).

Best practices

If you host multiple subdomains on the same server:

  • Make sure the subdomains are clearly identified in each log entry. One way to do this is to provide the full URL instead of only the request query string.

Regardless of your log format, the following information is required:

  • URL: Since the full URL isn't usually found in server logs, we reconstitute it based on some or all of the following fields:
     - scheme (http/https): can be determined using the port or the ssl protocol
     
    - hostname
     
    - path: can be found in the http request
     
    - query string: usually included in the path
  • date
  • referer
  • user agent
  • http status 

Regardless of your log format, the following information is optional:

  • response size (in bytes)
  • load time
  • client IP: This is used to verify that lines with a Googlebot user agent are genuine bot visits. This can be left empty if the user agent does not contain "Googlebot".


You can also find this article by searching for:
configuración del servidor, formato de logs, registros
paramétrer le serveur, format de logs

Did this answer your question?