Glossary

robots.txt

A site-level crawler policy file governed by the Robots Exclusion Protocol.