Record AI crawler requests

Report server-observed AI crawler requests as a batch of daily bot/page/status counts.

POST/api/v1/crawls

Access: Workspace API key · server only · Permission details

JSON request body

JSON request fields
Field	Type	Description
`siteId`Required	`string`	Website ID.
`sourceIntegration`	`string`	Reporting integration. Values: cloudflare, wordpress, middleware. Default: `cloudflare`.
`hits`Required	`array`	At most 500 entries. Each has botName, urlPath (default /), statusCode (default 200) and count (default 1, capped at 1,000,000).

`hits[]` fields

hits nested fields
Field	Type	Description
`botName`	`string`	Recognized crawler name.
`urlPath`	`string`	Path beginning with /, up to 500 characters.
`statusCode`	`integer`	HTTP status, 100–599.
`count`	`integer`	Positive whole count.

Request examples

Examples run from your server or terminal. Configure ATTRIBOO_TOKEN privately for authenticated requests. Replace example website IDs and values with your own. The examples make real requests when you run them.

cURL request

curl -X POST "https://www.attriboo.com/api/v1/crawls" \
  -H "Authorization: Bearer $ATTRIBOO_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{
  "siteId": "site_example",
  "sourceIntegration": "middleware",
  "hits": [
    {
      "botName": "OAI-SearchBot",
      "urlPath": "/pricing",
      "statusCode": 200,
      "count": 3
    }
  ]
}'

Response

Illustrative response. This endpoint returns raw JSON. Optional coverage, notes and continuation fields vary by report.

Response fields
Field	Type	Description
`ok`	`boolean`	Whether the request succeeded.
`kept`	`integer`	Kept crawl count.
`used`	`integer`	Workspace usage this month; may be absent if all bots were ignored.
`limit`	`integer`	Workspace monthly crawl cap.
`capped`	`boolean`	Whether the monthly cap was reached.

JSON response

{
  "ok": true,
  "kept": 3,
  "used": 3,
  "limit": 100000,
  "capped": false
}

Behavior and limits

Uses the workspace key. OAuth tokens are not accepted. This route does not use the report rate-limit headers.

Unknown bot names are ignored. Supported names: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, Perplexity-User, PerplexityBot, MistralAI-User, DuckAssistBot, meta-externalagent, Amazonbot, Bytespider and CCBot.

Legacy single-hit bodies can send siteId, botName, urlPath, statusCode and sourceIntegration without hits. Prefer batches for new integrations.

Crawls beyond the monthly cap are dropped. Free workspaces keep up to 100,000 a month; paid workspaces up to 5,000,000. This is separate from pageview/goal allowances.

Counts are additive. Retrying a successful batch can count it twice. Crawls are not visits or citations. Error responses contain error and may also contain ok: false.

HTTP status	Meaning
400	Invalid body or crawl recording failed.
401	Invalid or missing workspace key.
404	Unknown website.
413	More than 500 entries in a batch.

Errors and retries OpenAPI JSON