# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content # TheSchicht — robots.txt # https://theschicht.com · Northfast Limited # # Canonical source: this file (public/robots.txt in the Astro repo). # # --------------------------------------------------------------------------- # ⚠ DO NOT ENABLE Cloudflare's "Managed robots.txt" / Content Signals policy. # It PREPENDS its own block at the edge, creating a second, contradictory # "User-agent: *" group and Disallowing the AI crawlers this file # deliberately allows. Verified 2026-07-10: the origin served 590 bytes, # the edge delivered 2426. Cloudflare dash → AI Crawl Control → turn off. # # After editing this file you MUST purge the Cloudflare cache — it is # served with max-age=604800 (7 days), so changes are otherwise invisible # to Google for up to a week. # --------------------------------------------------------------------------- # --------------------------------------------------------------------------- # Default — the whole public site is indexable. # The Disallow lines are Ghost internals (admin, members, email/preview URLs). # None of them are content. # # NOTE — deliberately NOT blocking the legacy WordPress query permalinks # (/?p=N, /?cat=N, /?page_id=N, /?m=YYYYMM) here, even though they are junk. # # Blocking a URL in robots.txt stops Google from CRAWLING it — which means # Google never sees the 410 Gone we now return, and the URL can linger in the # index indefinitely as "Indexed, though blocked by robots.txt". To de-index a # URL you must LET Google fetch it and see the 410. Once they have dropped out # of Search Console (weeks), a Disallow can be added to save crawl budget. # See wordpress-purge.md. # --------------------------------------------------------------------------- User-agent: * Allow: / Disallow: /ghost/ Disallow: /members/ Disallow: /p/ Disallow: /email/ Disallow: /r/ Disallow: /unsubscribe/ # --------------------------------------------------------------------------- # Google — search AND the ad crawlers. # # Mediapartners-Google is the AdSense crawler; AdsBot-Google rates ad landing # pages. If either is blocked, ads cannot serve. Both are named explicitly so # that no future edit to the "*" group can accidentally starve them. # --------------------------------------------------------------------------- User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-News Allow: / User-agent: Google-InspectionTool Allow: / User-agent: Mediapartners-Google Allow: / User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / # --------------------------------------------------------------------------- # Other search engines # --------------------------------------------------------------------------- User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / # --------------------------------------------------------------------------- # AI / agent crawlers — ALLOWED by choice. # # TheSchicht publishes an llms.txt site index and wants to be citable by AI # assistants. To opt out of model training later, flip the relevant group to # "Disallow: /" HERE — do not rely on Cloudflare's managed block to do it, # because that also rewrites everything else. # --------------------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / User-agent: meta-externalagent Allow: / # --------------------------------------------------------------------------- # Discovery # # Two sitemaps by necessity: Ghost generates /sitemap.xml (homepage, posts, # authors, tags); Astro generates /sitemap-astro.xml (the 24 app pages — # /about/, /markets/, /docs/, /glossary/ and its terms, /legal/ …). # Ghost's sitemap index cannot reference the Astro one, so BOTH are declared # here and BOTH must be submitted in Search Console. # --------------------------------------------------------------------------- Sitemap: https://theschicht.com/sitemap.xml Sitemap: https://theschicht.com/sitemap-astro.xml # LLM-friendly site index: https://theschicht.com/llms.txt # Security contact: https://theschicht.com/.well-known/security.txt # Humans behind the site: https://theschicht.com/humans.txt