AI Search Glossary
Name | Description | Technical definition |
AI Bots | Web crawlers used by AI/LLMs. AI bots fall into three main families of crawlers (training, search, user).Their behavior and goals differ from traditional SEO bots. See the details below. | Web crawlers with a specific user agent. See our list of supported bots. |
AI Training Bots | AI Training bots crawl your website to scrape content that may be used in LLM training. Their activity reflects how AI systems gather data from your site. | Web crawlers with a specific user agent. See our list of supported bots. |
AI Search Bots | AI Search bots crawl your website asynchronously for indexing or improving the search results. Their activity is closer to classic SEO signals like Googlebot hits. | Web crawlers with a specific user agent. See our list of supported bots. |
AI User Bots | AI User bots crawl your website in real-time to find the answer to a user prompt. Their activity is key signal of visibility in AI interfaces that can be used as a proxy for impressions. | Web crawlers with a specific user agent. See our list of supported bots. |
Clicks | Visits to your site that come from an AI chatbot/LLM platform. | Number of log lines where the referrer contains a chatbot/LLM domain, for the selected time period. |
Citations | Estimated number of times your pages could be cited in AI answers, using AI User Bot visits as a proxy. | Number of log lines where the user agent is an AI User Bot, the IP matches official ranges, and the status code is 200 or 304. |
Pages clicked | Unique pages that received at least one visit from an AI chatbot/LLM. | Number of unique URLs with at least one referral visit from a chatbot/LLM. |
Crawled for citations | Unique pages crawled by AI User bots for citation purposes. | Number of unique URLs (with status code 200 or 304)that have at least one AI User Bot hit . |
Crawled for indexation | Unique pages crawled by AI Search Bots for indexing purposes. | Number of unique URLs with at least one AI Search Bot hit. |
CTR | Estimated click-through rate from an AI platform. The calculations is based on clicks and potential citations. | CTR = click / citations * 100. |
Pages CTR | Estimated page-level click-through rate. The calculation is based on pages clicked and pages crawled for citations. | CTR = pages with clicks / pages crawled for citations * 100. |
Understanding Oncrawl's AI Search Lens: Dashboards and Charts
Oncrawl's AI Search Lens is a powerful tool designed to help you measure, audit, and improve your website's visibility in AI search by combining crawl metrics with your server log files.
This article will guide you through the key dashboards and charts available in the AI Search Lens.
1. AI Search Visibility Dashboard
This dashboard provides a comprehensive overview of your AI visibility and clicks. Leveraging your server log files, it offers a reliable way to monitor AI bot interaction over time and quantify performance.
Key features:
Time frame selection: You can select a specific time frame for analysis. You can choose to aggregate data over days, weeks or months.
AI LLM platform focus: Filter data by the specific AI LLM platform you want to track. Choose from ChatGPT, Perplexity, Claude, Mistral, and Gemini (for deep research prompts only).
Charts within AI Search Visibility Dashboard:
AI bot activity over time:
Tracks the volume of citations (bot visits from AI user bots like ChatGPT, Perplexity, Claude, or Mistral AI, and Gemini's deep research bot).
Shows unique pages crawled for citations.
Displays clicks and the pages getting clicks from AI search platforms.
Data filtering: This chart automatically applies filters to status codes (200 or 304) to ensure accuracy. Server errors (404, 500) or redirects do not convert into citations. You can also exclude specific page types (e.g., pagination, filters, low-content pages) to refine the analysis to only include pages with citation potential.
Total volume for selected time frame & CTR:
Shows the total volume of citations for the selected time frame.
Introduces the ChatGPT CTR (or Perplexity CTR, etc.), calculated as clicks divided by citations, multiplied by 100.
Splits data by page groups to identify content types achieving the best results.
Citations and clicks by page level:
Provides metrics for pages crawled for citations, pages clicked, and page CTR (pages clicked divided by pages crawled for citation).
2. AI Search Insights Dashboard
This dashboard performs cross-analysis using data from your crawl and server logs to provide deeper insights into your AI search performance.
Reports within AI Search Insights Dashboard:
a. Coverage report
This report helps you understand which sections of your website are visible in AI search.
AI LLM platform filter: Filter by specific AI LLM platforms like ChatGPT, Perplexity, Claude, Mistral, etc.
Three-step funnel:
Indexation: Identifies the volume of pages from your website that are crawled for indexation or citation by AI search bots. This helps you pinpoint content completely ignored by AI search.
Citation: Excludes pages crawled for indexation only, focusing solely on pages crawled for citation.
Traffic: Looks at pages that are being clicked versus those that are not, helping to identify pages that fail to convert into traffic.
Dedicated ratios:
Indexation ratio: Number of pages crawled for indexation by search bots across your website.
Citation ratio: Number of pages crawled for citation by AI user bots in your website structure.
Clicked pages ratio: Number of pages receiving referral traffic from AI platforms (using log data).
Funnel breakdown by page group (reversed funnel):
Starts with pages clicked.
Then pages crawled for citation (excluding those already clicked).
Followed by pages crawled for indexation only (not crawled for citation and not clicked).
Ends with what is completely ignored by LLMs/AI chatbots.
Allows switching segmentation by page type, content type, indexation, clicks, etc., to understand website coverage and uncover insights.
b. Visibility impact report
This report focuses on AI user bots and cross-references their activity with your crawl metrics.
Depth impact on ChatGPT visibility: For each depth level, it shows how many pages are crawled by ChatGPT user bots during the log period.
Search visibility impact report: Takes the four steps of the funnel and analyzes different metrics by average. This allows you to compare and identify correlations with metrics like load time, word count, title size, and number of schema.
Filters for accuracy: Apply filters on page type (e.g., articles only, product pages only) to dig deeper into specific content segments.
c. Opportunities report
This report aims to identify quick wins based on the collected data.
Citation missed opportunities:
Highlights pages crawled for citation by AI user bots that have no chance of converting due to status code errors (400, 500), missing content, or thin content (less than 150 or 300 words).
These pages represent fixable issues that could improve citation rates.
ChatGPT final opportunities (signals to investigate):
Clicked-only pages: Pages that received clicks in ChatGPT but were not crawled by AI user bots or AI search bots. This "unusual pattern" warrants investigation.
Pages crawled for citation but not clicked: These pages never convert. Investigation can reveal if this is due to user intent, content summarizability, incorrect status codes, or content readability issues (e.g., behind JavaScript).
Pages crawled for indexation only: ChatGPT attempts to index them but never uses them for citation, another "unusual pattern" to analyze.
Orphan pages:
Traffic orphans: Pages clicked but not part of the website structure.
Citation crawl orphans: Pages with citations but not in the structure.
Indexation crawl orphans: Pages with indexation crawls but not in the structure.
ChatGPT data vs. Google Search Console: Compares ChatGPT dialog data with Google Search Console to find unusual patterns, such as pages with impressions but never crawled for citations or indexation, or pages with clicks but never crawled.
ChatGPT logs vs. Google logs: Compares logs from both sources to identify interesting patterns and actionable insights.
Custom charting and cross-data analysis
Beyond these pre-configured reports, the AI Search Lens allows you to create your own custom charts. You can cross-reference data from ChatGPT, Perplexity, Claude, etc., with other sources such as scraping data, Google Search Console data, and backlinks. You can also combine this with Oncrawl's AI content analysis (quality metrics on URLs) to see if content quality correlates with actual visibility.
This comprehensive set of dashboards and charts empowers you to gain a deep understanding of your website's performance in the evolving AI search landscape.