You can use data scraping to create custom fields and obtain analyses based on the content or structure of your pages. You might also know this function as "custom extraction."
​
To create a custom field, you will need to set up a scraping configuration in the crawl setting interface. When you define new crawl settings, you can add specific extraction rules to retrieve data when the page is fetched.
​

A Custom Field is some attribute of a web page that you want to collect for each page in your website. A Custom Field becomes a new column that you can add to any report in the Data Explorer.

You will be able to use the different values of your new attribute to analyze the pages of your site.

Custom Fields require the code source to be examined. The examination is carried out during the process of crawling your website. Because of this, they must be set before running a crawl.
​

Customs fields use extraction rules to find attributes in the source code of your pages during a crawl. These rules can be written as REGEX, as XPATH queries, or as a combination of both.

The choice of REGEX or XPATH depends on what you are looking for in the source code. You can use the following rules of thumb:

Once you've found what you're looking for, you can use Transformations to modify the final results. This can be useful, for example, if you want to count the number of times something occurs, or measure other characteristics of the content you were looking for.

The rules and transformations are set up when you set up a crawl. You can skip ahead to "Creating a Custom Field" for more information.
​

REGEX is used to find a specific string of text in the source code. A REGEX expression describes the text or type of text you are looking for using generic characters that can be interpreted by a search program.

Because the expression uses certain special characters to represent characteristics of what you're looking for, you have to escape special characters by placing a \ in front of them. Characters that must be escaped include: ? . / ( ) [ ] { }

A single REGEX may search for several strings. By default, only the first search string, or group, will be taken into account. To use a different group, or multiple groups, you will need to provide an output pattern.

Use the expression {n} to indicate a group, and replace "n" with the number of the group in the order it appeared in the search pattern. (The first group is number 0.)

For example, to search for a date written as 2018/07/19 and return 19-07-2018:

For help with REGEX, please see our quick guide here.
​

XPATH is a query language that can be used to find elements in a structured document, such as a webpage in HTML or XML. It describes the structural element you are looking for in your page source.

If you're used to using CSS selector, you'll want to use XPATH. To convert any existing CSS selector expressions to XPath, you can use this site: https://css2xpath.github.io/

For example, it can be used to find the text of all H2 headers on a page:

For help with XPATH queries, you can find more information in the XPath tutorial at w3schools.com.
​

Data scraping is an optional feature. If you can't enable scraping in the "Scraping" section in your crawl settings, you don't have access to this option.

Click on the Intercom button at the bottom right of the screen to talk to our team about adding data scraping to your plan.
​

Give a name to your Custom Field. This name will be used in the Data Explorer to find and display the columns.

Try to give an explicit name based on the information you want to extract.

Pick a name made up of lowercase and uppercase letters, hyphens, underscores, and numbers. Special characters, including accents, cannot be used in names of custom fields.

Set up the parsing rules. You can use multiple steps. Each successive step will be applied to the result of the previous step. 

For each step:

(We give you a few examples of some of our frequently-used rules at the end of this article. Go ahead and jump to the end if want.)
​

Drop empty values: if any of the values found do not contain any characters, ignore them. This might be the case if, for example, you searched for the name of a WordPress category, but have pages that do not have a category.

Normalize URLs: if the results are URLs, format them all as "http://yourdomain.com/path". This removes relative URLs ('/path").

Apply a pattern: add characters before or after the values you find. The value found is represented by the {0} in the field below. You can place text before or after the {0}. Make sure not to delete or modify the {0} if you want to see the value itself in your results.

Get number of values: instead of listing the values found, count the number of results.

Replace HTML entities with display characters: if the value found contains characters such as accents or other characters with special HTML, such as the & for the character "&", display the character instead of the HTML.

Condense white spaces: if your result contains a series of adjacent spaces, line returns, and other "white" spaces, replace the series with a single space.

Get value's length: count the number of characters in the found value and replace the value with the number of characters.
​

Decide in what format you want to export the results.

You can keep all values, or just the first value found.

A value can be saved as a:

Before saving, it's a good idea to make sure that your scraping rule works the way you expect it to. Check your rule by running it on a sample URL, or by providing text content for a sample analysis.

The result will display on the right.

Don't forget to click "Save Custom Field" in the verification box when you're satisfied!
​

If it were to happen, the result would be truncated by either:

Because of this, fixed-size fields (boolean, integer, ...) are parsed in priority versus others (strings,..) to ensures that we scrape as much data as possible.

You can use the Data Explorer with the field processing_warning to have more details about truncated or dropped scraped fields.

Custom field name: amphtml

Parsing step 1:

  Kind of rule:              XPATH

  Rule:                       //link[@rel="amphtml"]

Parsing step 2:

  Kind of rule:              XPATH

  Rule:                       string(//@href)

Transformations:      Drop empty values, Normalize URLs

Export:

  Keep...                   All values

  As…                      String (e.g. "lorem ipsum")

Custom field name: breadcrumb

Parsing step 1:

  Kind of rule:              XPATH

  Rule:                         //*[@itemtype="http://schema.org/BreadcrumbList"]//*[@itemprop="name"]/text()

Transformations:      none

Export:

  Keep...                   All values

  As…                      String (e.g. "lorem ipsum")

This is for a schema.org breadcrumb integration. XPATH should be adapted to your integration.

Custom field name: breadcrumb_depth

Parsing step 1:

  Kind of rule:              XPATH

  Rule:                       //*[@itemtype="http://schema.org/BreadcrumbList"]//*[@itemprop="name"]/text()

Transformations:      Get number of values

Export:

  Keep...                   All values

  As…                      Number (e.g. 123)

This is for a schema.org breadcrumb integration. XPATH should be adapted to your integration.

Custom field name: GA_is_present

Parsing step 1:

  Kind of rule:              REGEX

  Rule:                       _gaq.push\(\['_setAccount', 'UA-XXXXXXXX-X'\]\);

  Output pattern:      none

Transformations:     none

Export:

  Keep...                   First value

  As…                      Boolean value (e.g. true)

NB: replace 'UA-XXXXXXXX-X' with your account ID

If you still have questions about data scraping and custom fields, feel free to drop us a line at @oncrawl_cs or click on the Intercom button at the bottom right of your screen to start a chat with us.

Happy crawling!

You can also find this article by searching for:
​extracción de datos campos personalizados métricos personalizados
extraire données du code source, champs personnalisées, métriques customisés