# robots.txt

Category: Agent readiness
Version: 1.0.0

## Goal
Publish a discoverable robots.txt at the site root with explicit, machine-readable crawl rules.

## Issue
The audited site did not satisfy the robots.txt readiness check.

## Fix
Create /robots.txt at the site root with explicit User-agent directives and Allow/Disallow rules for the paths you want crawlers to find or skip. Serve it as plain text with HTTP 200 — never return HTML or a redirect — and keep the file under 500 KB so crawlers parse it fully.

## Validation
- Confirm content types, payload shape, and discoverability from the public web surface.
- Verify against fresh public evidence, not only local source changes.
- Re-run the relevant isitready.dev scan before claiming the issue is closed.

## Copyable Fix Prompt
```text
Outcome: make this single public-readiness issue pass for the audited site.

Goal: robots.txt for https://example.com

Issue: The audited site did not satisfy the robots.txt readiness check.

Fix: Create /robots.txt at the site root with explicit User-agent directives and Allow/Disallow rules for the paths you want crawlers to find or skip. Serve it as plain text with HTTP 200 — never return HTML or a redirect — and keep the file under 500 KB so crawlers parse it fully.

Skill: /.well-known/agent-skills/robots-txt/SKILL.md

Docs: https://www.rfc-editor.org/rfc/rfc9309

Validation: Confirm content types, payload shape, and discoverability from the public web surface.

Requirements: preserve existing product behavior, add or update relevant tests, and validate public response shape/content type where applicable.

When done, tell the user to re-run the public scan at https://isitready.dev so they can verify this issue now passes. Do not claim it is fixed until a fresh report confirms.
```