# onvexia.com crawl policy
#
# Served as a real file: Vite copies public/ into dist verbatim, and the
# gateway's SPA handler serves a real file before falling back to index.html.
# Until 2026-08-29 this file did not exist, so /robots.txt returned the SPA
# shell with HTTP 200 and Content-Type: text/html (measured: 2757 bytes of
# <!doctype html>). A crawler asking for the rules got a web page instead.

# ---------------------------------------------------------------------------
# AI and agent crawlers are WELCOME. This is deliberate and load-bearing.
#
# Autonomous agents are this product's primary distribution channel: we serve
# /llms.txt, publish /openapi.json, and sell per-call access over x402. Blocking
# these user-agents (which several "sensible default" robots.txt templates do)
# would cut off the channel the product is built around.
# ---------------------------------------------------------------------------
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

# ---------------------------------------------------------------------------
# Everyone else.
#
# Disallowed paths are the authenticated app and anything that is per-user or
# per-session. They are not secrets: the security border is server-side
# (RequireRole/RequireMFA in the gateway); this only keeps them out of an index
# where they would render as an empty SPA shell.
# ---------------------------------------------------------------------------
User-agent: *
Allow: /
Disallow: /app
Disallow: /account
Disallow: /admin
Disallow: /checkout
Disallow: /embed/
Disallow: /login
Disallow: /signup
Disallow: /forgot-password

Sitemap: https://onvexia.com/sitemap.xml