AI Crawler Tracking and What Bot Requests Tell You

What Should AI Crawler Tracking Count?

Count recorded requests, with the page, time and requester label where your logging method supplies them. Start with a small question: are important product pages being requested, and what response did the logging layer observe? That is useful operational evidence even before you know whether any customer arrived.

OpenAI’s crawler reference distinguishes OAI-SearchBot for search, GPTBot for potential model training and ChatGPT-User for certain user-triggered actions. Those purposes differ. Grouping all three as ChatGPT visitors would mix automated collection with software fetching a page for someone.

1. Choose One Place to Collect Requests

Choose the logging layer that can see the traffic you care about. A WordPress plugin is convenient when WordPress runs for the request. An edge logger can see requests before they reach your application, but only when the traffic passes through its configured route.

Cloudflare’s AI Crawl Control documentation describes request visibility and per-crawler access policies. It is a useful alternative when your immediate task is managing access. Attriboo’s AI crawl tracking feature is for bringing supported request totals into the same workspace as website and revenue reports.

Use one Attriboo logging method per website. Running the plugin and Worker together can report the same request twice. Keep a note of the method, activation date and covered hostnames. Those details make a later increase easier to interpret than an unexplained total.

2. Connect the Logger to the Correct Website

Open Setup in Attriboo and expand Track AI crawlers. Select WordPress or Cloudflare for your chosen method. Confirm the selected website before copying anything; a correct installation attached to the wrong website still produces a misleading report.

1. For WordPress, download the plugin from Setup. Upload and activate it through Plugins → Add New Plugin → Upload Plugin, then open Settings → Attriboo.
2. Enter the Website ID and current workspace API key. Check the plugin’s tracking-script option if you already installed the browser script elsewhere; keep a single browser-tracking installation.
3. For Cloudflare, use the generated Worker code from Setup. Add the workspace key as the ATTRIBOO_KEY secret and configure the intended domain route.
4. Follow the current crawler setup guide for the remaining platform steps. Ask your website administrator to check proxying, caching and route coverage if those controls are unfamiliar.

Keep the workspace key out of public pages, screenshots and shared documents. It grants workspace API access, so it is not just a label for a crawler. If the key is rotated, update the chosen logger too. The ordinary browser tracking script measures a different kind of activity and does not replace server-side crawl logging.

3. Check Coverage and Requester Identity

Treat the crawler name as a classification to check, not an identity guarantee. Attriboo’s supplied WordPress and Worker loggers recognize documented names in the User-Agent header. That header can be copied. These loggers do not independently verify the requester against the operator’s network ranges.

For a consequential access decision, ask the person managing your server or CDN to compare the request’s network evidence with the operator’s current verification guidance. OpenAI publishes separate IP resources for its named agents. A failed or incomplete check belongs in an unresolved group until someone investigates it.

• Check the hostname and route actually covered by the logger. Include any separate www or content subdomain that matters to the review.
• Compare a small request sample at the collection layer with what arrived in the report. Allow for batching; a blank count immediately after installation is not enough to diagnose failure.
• Review caching and blocks before concluding that no crawler visited. Requests answered elsewhere may never reach an application plugin.
• Keep testing traffic and overlapping loggers out of production comparisons. A logging change can explain a jump without any change in AI interest.

4. Read Crawler Totals Without Mixing Them With Revenue

Open Dashboard, find the AI reporting widget and select View all. In AI traffic & revenue, keep the website and reporting dates visible, then choose Bot crawls in Sort. Inspect that column separately from Visitors, Order credit and Net revenue.

The engine view groups request activity by service. For the exact bot token, path, status and network verification, return to your request logs. Do not describe the engine table as a complete request-by-request security log. The export is useful for the report it actually contains.

Compare like periods after collection is stable. If requests to a product page stop, first check routing, server responses and site changes. If requests grow but visits stay flat, the immediate conclusion is only that the two counts moved differently. There is no supported reason to divide them into a conversion rate.

5. Choose an Action That Matches the Evidence

Use the request evidence to investigate access and content availability. Use browser visits to inspect what people did. Use connected payments to discuss recorded revenue credit. Each question deserves its own check and owner.

If useful pages repeatedly return errors in your logs, ask the site owner to inspect those URLs and response rules. If a crawler reads only old URLs, review the current internal links and sitemap before changing content. Neither repair guarantees an AI recommendation, but both address a problem your evidence actually shows.

When the question becomes whether AI referrals bring buyers, move to the ChatGPT traffic tracking workflow. Check source labels, recorded paths and payment matches there. For an agency update, report the crawl observation alongside the customer outcome without turning one into proof of the other.

A useful client note might read: “Our logger recorded requests to the pricing page this week. We checked collection coverage. Separately, the website recorded AI referrals and connected payments under the stated attribution model. Next, we will investigate the pages visitors reached before buying.” Fill in verified values from the appropriate records.

Common questions