Skip to main content
All CollectionsLog MonitoringLog monitoring
Server side log format configuration
Server side log format configuration

Learn how to setup your server log configuration regarding the type of server you are using.

Updated over a week ago


Recommended format:

LogFormat "%{HOST}i:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined_host_ssl
  • %{HOST}i : hostname of the query

  • %p : server port

  • %h : client IP

  • %l : remote logname. Not used by OnCrawl by standard in log formats

  • %u : remote user. Not used by OnCrawl by standard in log formats

  • %t : request date time

  • %r : first line of request

  • %>s : response status code

  • %b : size of the response in bytes

  • %{Referer}i : referer

  • %{User-Agent}i : User agent


Recommended format:

log_format combined_vhosts '$host:$server_port $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';

Reference documentation:


Recommended formats: W3C Extended Log File Format or NCSA Common Log File Format.

W3C Extended Log File Format

Example format containing time, client IP address, method, URI stem, protocol status and protocol version:

#Software: Internet Information Services 6.0 
#Version: 1.0
#Date: 2001-05-02 17:42:15
#Fields: time c-ip cs-method cs-uri-stem sc-status cs-version
17:42:15 GET /default.htm 200 HTTP/1.0 

NSCA Common Log File Format

The following example lists these properties:

  • IP address

  • Remote log name of the user (blank in this example and represented by a "-" )

  • Domaine/user

  • Date and time

  • Command

  • Status code returned

  • Bytes of data sent - Microsoft\fred [08/Apr/2001:17:39:04 -0800] "GET /scripts/iisadmin/ism.dll?http/serv HTTP/1.0" 200 3401


For those using CloudFlare, you need to have an Enterprise plan in order to access your (raw) log files. Then, you need to activate Enterprise Log Share.
Also, you can set up a daily upload of your log files to our FTP (with the credentials you received by email).

Best practices

If you host multiple subdomains on the same server:

  • Make sure the subdomains are clearly identified in each log entry. One way to do this is to provide the full URL instead of only the request query string.

Regardless of your log format, the following information is required:

  • URL: Since the full URL isn't usually found in server logs, we reconstitute it based on some or all of the following fields:
     - scheme (http/https): can be determined using the port or the ssl protocol
    - hostname
    - path: can be found in the http request
    - query string: usually included in the path

  • date

  • referer

  • user agent

  • http status 

Regardless of your log format, the following information is optional:

  • response size (in bytes)

  • load time

  • client IP: This is used to verify that lines with a Googlebot user agent are genuine bot visits. This can be left empty if the user agent does not contain "Googlebot".

Did this answer your question?