URLs appear in many formats.

Oncrawl crawler resolves certain types of URLs encountered during a crawl in order to process them and to avoid errors in identifying duplicate content.

Oncrawl will regularize slashes in the URL where they are meant to indicate a relative path, according to the following rules:

Oncrawl resolves all relative URLs. A relative and a complete URL are counted as a single URL, rather than two different URLs on your site.

For example:

will be recorded as:

Oncrawl's built-in system removes session IDs if they are present in your URLs.

This cannot be disabled.

By default, parameters are not filtered or removed, except for session IDs mentioned above.

However, in the crawl profile settings, you can configure the crawler to filter on or to remove certain parameters.

Additionally, Oncrawl may change the order of a the parameters in a query string in a URL for the purpose of comparison and analysis.

The order of the parameters' values will not be changed. This can be disabled if necessary.

By default, this means that you might find URLs with parameters in the crawl results that don't exist on your site in exactly the same form, character for character.

However, the URLs used in the crawl results will always be functionally equivalent to the actual URLs found on your site.

For example:

may be treated as:

This allows us to treat all of the URLs with the same parameters in a different order (q=search&utm=email&parameter1=value1  or parameter1=value1&q=search&utm=email  or utm=email&q=search&parameter1=value1  or ...)
as the same page.
​
​Note: We always keep multiple values for the same parameter in their original order. For example, if your query string contains parameter1=value1&parameter1=value2  and we change the order of the parameters, the re-ordered URL will still contain the exact string parameter1=value1&parameter1=value2.
​

Oncrawl truncates (removes and ignores) the content in a URL following a hash (#).

For example:

will be recorded as:

Note: We keep hashbangs (#!) and the content that follows a hashbang. This is used as a render indicator in certain types of JavaScript.
​

We encode special and non-ASCII characters as HTML entities starting with a % symbol in the URL.

For example:

will be treated as the same as:

Oncrawl adds a missing trailing slash.

For example:

will be treated as the same as:

Oncrawl removes default ports from the URL.

Default ports are 80  for HTTP and 443  for HTTPS.

All other ports are left in the URL.

For example:

will be treated as the same as:

Oncrawl's analysis is not case sensitive where host names are concerned.

For example:

is the same as:

However, uppercase and lowercase characters do make a difference in the URL path after the domain name.

For example:

is different from: