Universite/doc-html
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

robots: Exclusion IA bots

Browse Source
This commit is contained in:
Emmanuel Medina
2025-11-05 10:17:13 +01:00
parent 11dfbc7dfc
commit a0a2c0061b
1 changed files with 10 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
robots.txt
View File

@@ -1,5 +1,13 @@
# https://www.robotstxt.org/ # https://www.robotstxt.org/
# Deny crawling of all content User-Agent: GPTBot
User-agent: * User-Agent: ClaudeBot
User-agent: Claude-Web
User-Agent: PerplexityBot
User-Agent: CCBot
User-Agent: Google-Extended
Disallow: /
User-agent: Yandex
User-agent: Baiduspider
Disallow: / Disallow: /