# Playway Books — crawler policy
#
# The 2026 crawler landscape is tripartite: a vendor runs a TRAINING crawler, a
# SEARCH indexer that grounds its answer engine, and a LIVE-FETCH agent that
# retrieves a page because a user asked for it right now. They are separate user
# agents with separate purposes, and a single blanket rule cannot express a
# policy about them. Each is named below so the policy is explicit rather than
# inherited from `User-agent: *`.
#
# Playway's position: everything is allowed. A parent asking an assistant "how
# do I help my child with chemical formulas" is exactly the moment this book
# should be named, and a method nobody has heard of has nothing to gain from
# withholding itself. Revisit only if the business ever sells the content it
# currently gives away.

# ── Traditional search ────────────────────────────────────────────────────────
User-agent: *
Allow: /
# /go/ is the Buy button's counter, not a page. It redirects to a retailer and
# has nothing to index; crawling it would also inflate the click count with
# presses no person made. EVERY named group below restates this line, because a
# named group replaces this one rather than adding to it — see the note further
# down, and scripts/structure-scan.mjs, which fails the build if one forgets.
Disallow: /go/
#
# THE .md TWINS ARE NOT EXCLUDED HERE, AND THAT IS THE POINT OF THIS GROUP.
#
# They are the token-efficient twin of each HTML page, written for AI ingestion
# (see /llms.txt). The reason to withhold them is narrow: a human landing on raw
# Markdown from a SERP is a poor result and a duplicate-content signal. That
# rationale applies to engines which show results to PEOPLE, so the exclusion
# lives on the four named indexers below and nowhere else.
#
# This group catches every crawler that has NOT been named — and in 2026 that
# population is overwhelmingly a new AI agent rather than a new search engine.
# Excluding .md here would therefore withhold the corpus from precisely the
# readers it was written for, until somebody remembered to add a group for them.
# The default is permissive on purpose: an agent nobody has heard of yet gets
# the whole site on arrival, which is Playway's stated position rather than a
# concession. Adding a name below is then an act of RESTRICTION, never of
# permission, and no new agent has to wait for it.

# THE `Disallow: /go/` BELOW IS NOT A DUPLICATE OF THE ONE ABOVE, AND DELETING
# IT REOPENS A REAL HOLE. Under RFC 9309 a crawler obeys exactly ONE group: the
# most specific one matching its name, which REPLACES the `*` group rather than
# adding to it. So the moment Googlebot got its own group here for the .md rule,
# it stopped inheriting `Disallow: /go/` — and the four indexers below were free
# to crawl the Buy counter, each press landing in the D1 table as a click no
# person made. Every named group that restricts anything has to restate the
# whole policy for itself. scripts/structure-scan.mjs enforces this.

User-agent: Googlebot
Allow: /
Disallow: /go/
Disallow: /*.md$

User-agent: Bingbot
Allow: /
Disallow: /go/
Disallow: /*.md$

User-agent: DuckDuckBot
Allow: /
Disallow: /go/
Disallow: /*.md$

User-agent: Applebot
Allow: /
Disallow: /go/
Disallow: /*.md$

# ── AI training crawlers ──────────────────────────────────────────────────────
# Corpus builders. Allowing these trades content for entity recognition in the
# next model generation.
#
# EVERY GROUP FROM HERE DOWN CARRIES `Disallow: /go/`, AND NONE OF THEM
# RESTRICTS ANY CONTENT. The site is still open to all of them: /go/ is the Buy
# button's redirect endpoint, not a page, and there is nothing there to read.
# The exclusion is here because an AI agent that follows a Buy link lands in the
# D1 click counter exactly as a reader pressing the button does, and that table
# records no user agent by design — so a crawler's visit and a person's decision
# become indistinguishable the moment it is written. It has to be prevented
# rather than detected.
#
# This was fixed for the four traditional indexers first and missed here, which
# left seventeen agents free to inflate the count. The rule is the same for all
# of them and the reason has nothing to do with what tier an agent belongs to.
User-agent: GPTBot
Allow: /
Disallow: /go/

User-agent: ClaudeBot
Allow: /
Disallow: /go/

User-agent: Google-Extended
Allow: /
Disallow: /go/

User-agent: Applebot-Extended
Allow: /
Disallow: /go/

User-agent: meta-externalagent
Allow: /
Disallow: /go/

User-agent: Amazonbot
Allow: /
Disallow: /go/

User-agent: Bytespider
Allow: /
Disallow: /go/

User-agent: CCBot
Allow: /
Disallow: /go/

# ── AI search indexers ────────────────────────────────────────────────────────
# These build the retrieval index an answer engine cites from. Blocking one
# removes this book from that engine's answers entirely.
User-agent: OAI-SearchBot
Allow: /
Disallow: /go/

User-agent: Claude-SearchBot
Allow: /
Disallow: /go/

User-agent: PerplexityBot
Allow: /
Disallow: /go/

# ── AI live-fetch agents ──────────────────────────────────────────────────────
# Retrieve a page in real time because a user asked. Blocking one produces a
# visible failure inside someone's chat window, not a silent absence.
User-agent: ChatGPT-User
Allow: /
Disallow: /go/

User-agent: Claude-User
Allow: /
Disallow: /go/

User-agent: GoogleAgent-Mariner
Allow: /
Disallow: /go/

User-agent: Perplexity-User
Allow: /
Disallow: /go/

User-agent: DeepSeek-User
Allow: /
Disallow: /go/

User-agent: MistralAI-User
Allow: /
Disallow: /go/

Sitemap: https://playwaybooks.com/sitemap-index.xml