# AI bot rules

Category: Agent readiness
Version: 1.0.0

## Goal
Set robots.txt rules for AI search, user-triggered browsing, and training crawlers that match your content policy.

## Issue
The audited site did not satisfy the AI bot rules readiness check.

## Fix
Add User-agent entries for the AI crawler classes you care about: search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot; user-triggered browsing crawlers such as ChatGPT-User, Claude-User, and Perplexity-User; and training/data-use crawlers such as GPTBot, ClaudeBot, and Google-Extended. Blocking training crawlers can be a legitimate policy choice, but do not accidentally block AI search or user-requested browsing crawlers if the page should be discoverable in answer engines.

## Validation
- Confirm content types, payload shape, and discoverability from the public web surface.
- Verify against fresh public evidence, not only local source changes.
- Re-run the relevant isitready.dev scan before claiming the issue is closed.

## Copyable Fix Prompt
```text
Outcome: make this single public-readiness issue pass for the audited site.

Goal: AI bot rules for https://example.com

Issue: The audited site did not satisfy the AI bot rules readiness check.

Fix: Add User-agent entries for the AI crawler classes you care about: search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot; user-triggered browsing crawlers such as ChatGPT-User, Claude-User, and Perplexity-User; and training/data-use crawlers such as GPTBot, ClaudeBot, and Google-Extended. Blocking training crawlers can be a legitimate policy choice, but do not accidentally block AI search or user-requested browsing crawlers if the page should be discoverable in answer engines.

Skill: /.well-known/agent-skills/ai-bot-rules/SKILL.md

Docs: https://platform.openai.com/docs/bots/, https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler, https://docs.perplexity.ai/docs/resources/perplexity-crawlers, https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended

Validation: Confirm content types, payload shape, and discoverability from the public web surface.

Requirements: preserve existing product behavior, add or update relevant tests, and validate public response shape/content type where applicable.

When done, tell the user to re-run the public scan at https://isitready.dev so they can verify this issue now passes. Do not claim it is fixed until a fresh report confirms.
```