# Crawl rules. The content-signal policy — search=yes, ai-input=yes, # ai-train=no — with its definitions and the Article 4 reservation is # declared in llms.txt; robots.txt carries only the enforceable half. # Training crawlers visit only to collect for model training, which this # site does not grant (ai-train=no), so they are blocked outright. The # answer-engine products those companies run are unaffected: they read # through their fetch and index bots, listed in the next group. User-agent: GPTBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: ClaudeBot Disallow: / # AI answer engines are listed explicitly rather than left to fall through # to the wildcard group — an ambiguous robots.txt is how a citation # disappears. The functional pages are excluded for every agent: nothing # there is an answer. User-agent: OAISearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: meta-externalagent User-agent: Bingbot Allow: / Disallow: /preview Disallow: /cart Disallow: /checkouts Disallow: /account Disallow: /password User-agent: * Allow: / Disallow: /preview Disallow: /cart Disallow: /checkouts Disallow: /account Disallow: /password # /search stays crawlable on purpose: it carries noindex, and a disallow would # hide that signal from the crawler instead of being obeyed alongside it. # # /llms.txt is the plain-text site index for answer engines — not a sitemap, so # it is not listed in a Sitemap: directive. Sitemap: https://curatedpret.com/sitemap.xml