When a bot visits a website, it views the HTML of a page and visits the links that it finds. If your website is built using JavaScript (whether all of the website, or only parts of the content and navigation), this may mean that the main content of the page is not visible in the HTML, but constructed dynamically after the HTML is fetched. In this case, the bot will not see content or links on the page, and will not be able to discover the pages in your site.
You can solve this by pre-rendering the JavaScript. It’s easy to do in Oncrawl!
Top three reasons to render JavaScript for a crawl
For some sites, it's impossible to crawl the site without JavaScript enabled. But that doesn't mean you don't need more information on how the website performs, and you can only get that information from a crawl.
JavaScript can alter content and links by adding information. Rendering JavaScript ensure that you're analyzing the "real" version of your website, and that it behaves as intended.
JavaScript can be used to redirect one URL to another. Crawling with JavaScript helps you make sure you've audited all of your redirected pages.
Enable the Crawl JS option
Click Set up a new crawl and choose the crawl profile you want to enable JavaScript rendering for.
Click on the Crawl JS title to expand the section.
Check the Crawl the website as a JavaScript website option.
Now you can launch the crawl as usual.
Crawling and rendering the JavaScript for a URL will consume 3 tokens from your quota, compared to just 1 token for only crawling the URL.
Flatten Shadow DOM (additional JS option)
In the Crawl JS crawl settings, you can enable Flatten Shadow DOM to expose this content to the Oncrawl crawler.
What does this mean?
Many websites make use of the Shadow DOM to create pages that are more lightweight and reactive. The page contains short, custom HTML, often for parts of the page that are used more than once on the page or around the site.
When the page is loaded, it uses JavaScript to "flatten" this custom element into the real content, which may contain text, images, and links, just like any other content on the page.
Google has explained that flattening the shadow DOM is part of their process for rendering JS on a web page. Oncrawl behaves in the same way.
On-site requirement in order to be able to flatten the Shadow DOM
If your website is built using content that is condensed, then expanded in the shadow DOM, it can be set up in one of two ways:
Open, in which the website shares the "flattened" code and content with a service or a person that looks at the final rendered page source.
Closed, in which only the user looking at a browser can see the content; it doesn't display to someone looking at the page source.
Bots like Oncrawl (and Googlebot!) can only take into account open shadow DOM content, even when the shadow DOM option is enabled.
If your content isn't rendering because it is closed, this is due to a decision by your webmaster or dev team. Neither Oncrawl nor Google can see this content:
(Source)
Default treatment of the Shadow DOM in Oncrawl
Like Google, Oncrawl flattens the Shadow DOM by default, except in crawl profiles with JS enabled that were created before Oncrawl introduced this option.
If you don't want Oncrawl to flatten your Shadow DOM when rendering JS, you can disable this option.
If you previous crawled without flattening the Shadow DOM but would like to enable this option, this might affect content and linking across your site, making it difficult to compare your new crawl to previous crawls, or making unexpected changes to your website data.
Variation in content metrics
If the Shadow DOM contains additional text, it will impact duplicate content, near-duplicate content analysis, word count, and other similar metrics.
Variation in internal linking structure and numbers of links
If the Shadow DOM contains additional links, it will impact the inrank and link flow across your site, the Inrank of pages, the number of internal and external links on a page, the number of links pointing to a page, the pages identified as orphans, and other similar metrics.
Variation in payload and page speed
If the Shadow DOM contains content that takes time to load, it will impact the payload metrics for those pages.
Possible broken scraping rules
If you have scraping rules that do not directly target the element they're looking for (for example, ones that list the entire path to the element: /html/body/div/div[2]/div[3]…), enabling this option might break your scraping rule. This is because the injected JS content from the shadow DOM will likely change the absolute path to the element you want to capture.
These scraping rules will need to be rewritten to work when taking the new content into account.
Best practices for crawling JS
If your site is built using JavaScript, it may still be worth it to see what your site looks like with JavaScript disabled. Run your crawl with and without JavaScript enabled to compare the results.
The JavaScript crawl is not compatible with authentication by username and password. If you're working with a pre-prod website built with JavaScript, you may need to plan accordingly.
Check your quotas before running a JavaScript crawl. JavaScript crawls use more resources than normal crawls, so crawling a page in JavaScript will consume 3 URLs in your quota.