Search and Discovery / TOKEN WIKI
robots.txt
In one sentence
robots.txt is a crawling rule file placed in the root path of the site, which is used to express access preferences to crawlers that abide by the rules.
Why it matters
It is not an access control mechanism; private data must be protected by authentication and cannot rely solely on Disallow.
Practical considerations
allows the disclosure of body paths, excludes non-public areas such as accounts and backends, and ensures that the rules do not conflict with platform access restrictions. Can
FAQ
robots.txt protect the management backend?
No, the management backend must have server-side authentication and permission checking.
