Glossary

robots.txt

A site-level crawler policy file governed by the Robots Exclusion Protocol.

Related terms

sitemap, llms-txt