User-agent: * Allow: / # Block crawlable parameterized duplicates of list endpoints (audit v2.2.5 M-3). # Clean URLs stay fully crawlable; only query-string variants are disallowed. # R2 H-1 (2026-09-08): allow the page assets past Disallow: /*?. # r7 C1 (2026-09-27): the site renamed style.css to style.min.css and added # versioned /site.js; the allow rules must name the files the HTML actually cites # (RFC 9309 longest-match, so these beat Disallow: /*? for their ?v= URLs). Allow: /style.min.css* Allow: /site.js* # r8 M17 (2026-09-28): the /og/charts/ images carry cache-buster queries; # without an Allow, Disallow: /*? hides the site's own cited charts. Allow: /og/charts* Allow: /og/* Disallow: /*? # A-2 (2026-09-25, decided by the site owner): AI retrieval and grounding are # welcome (with attribution); model training on this content is not permitted. # Full policy: https://martechsignal.com/ai-policy/ Content-Signal: search=yes, ai-input=yes, ai-train=no # ── Content Signals Policy (Cloudflare, CC0) ──────────────────────────────── # As a condition of accessing this website, you agree to abide by the following content signals: # # (a) If a content-signal = yes, you may collect content for the corresponding use. # (b) If a content-signal = no, you may not collect content for the corresponding use. # (c) If the website operator does not include a content signal for a corresponding use, the website operator neither grants nor restricts permission via content signal with respect to the corresponding use. # # The content signals and their meanings are: # # search: building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website's contents). Search does not include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers). # ai-train: training or fine-tuning AI models. # # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # ── AI training crawlers: denied ──────────────────────────────────────────── # RFC 9309 note: once a token has a named group, the `*` group no longer applies # to it at all, so each group below is complete on its own. Search crawlers and # user-triggered fetch agents are intentionally unnamed and keep the `*` rules. User-agent: GPTBot Content-Signal: ai-train=no Disallow: / # Anthropic runs three crawlers (support.claude.com, verified 2026-09-27): # ClaudeBot collects training data, Claude-User fetches pages when a Claude # user asks a question, Claude-SearchBot indexes for search. The training # crawler stays blocked; the retrieval pair stays open, so the ai-input=yes # declaration on /ai-policy/ is enforced exactly as declared. User-agent: ClaudeBot Content-Signal: ai-train=no Disallow: / User-agent: Claude-User Allow: / Content-Signal: search=yes, ai-input=yes, ai-train=no User-agent: Claude-SearchBot Allow: / Content-Signal: search=yes, ai-input=yes, ai-train=no # r7 H14 (2026-09-28): this is a deliberate decision on the record, not an # inherited template (the audit's actual ask). Google-Extended is blocked as of # 2026-09-26: the token governs model training, and keeping content out of # training is worth losing Gemini API / Vertex AI grounding citations. The cost is # real and accepted: the site drops out of Gemini's grounding pipeline. Search and # AI Overviews are unaffected (they follow the allowed Googlebot index). Full # reasoning: https://martechsignal.com/ai-policy/ User-agent: Google-Extended Content-Signal: ai-train=no Disallow: / User-agent: CCBot Content-Signal: ai-train=no Disallow: / User-agent: Applebot-Extended Content-Signal: ai-train=no Disallow: / User-agent: meta-externalagent Content-Signal: ai-train=no Disallow: / User-agent: Bytespider Content-Signal: ai-train=no Disallow: / User-agent: Amazonbot Content-Signal: ai-train=no Disallow: / Sitemap: https://martechsignal.com/sitemap.xml # r32 L-37 (2026-10-06): every page serves a Markdown mirror at index.md # (content-type text/markdown) for retrievers - same content, clean money-token # parity. Discoverable by convention; this comment is the pointer. # M17 (r9, 2026-09-28): machine-readable catalog entry points for AI systems. # (Plain comments, not directives - crawlers ignore unknown lines per RFC 9309.) # A-3 (v2.4.0 audit): ARD entry source (spec §5.1) # Agentmap: https://martechsignal.com/.well-known/ard.json # (L5, r9: 'Agentmap:' is not a standard robots.txt directive and trips the # Lighthouse robots-txt validator. The ARD manifest is still discoverable via # in every page head, so this stays a comment.) # A3 M-22 (2026-09-27): training-adjacent data collectors named explicitly. # Google-Extended is deliberately BLOCKED (training refusal wins over Gemini # grounding citations, decided 2026-09-26 — see its group below and /ai-policy/); # Meta-WebIndexer and Meta-ExternalFetcher stay allowed. Earlier comment here # claimed grounding was open; that was wrong and contradicted the rule (r7 H14). User-agent: cohere-ai Content-Signal: ai-train=no Disallow: / User-agent: Diffbot Content-Signal: ai-train=no Disallow: / # r11 M-6 (2026-09-29): the finding was right — seven named training tokens # inherited the `*` group instead of an explicit deny. Each gets its own # complete group below (RFC 9309 note at the top of this section applies). User-agent: AI2Bot Content-Signal: ai-train=no Disallow: / User-agent: AI2Bot-Dolma Content-Signal: ai-train=no Disallow: / User-agent: ImagesiftBot Content-Signal: ai-train=no Disallow: / User-agent: PanguBot Content-Signal: ai-train=no Disallow: / User-agent: omgili Content-Signal: ai-train=no Disallow: / User-agent: omgilibot Content-Signal: ai-train=no Disallow: / User-agent: Timpibot Content-Signal: ai-train=no Disallow: / User-agent: KangarooBot Content-Signal: ai-train=no Disallow: / User-agent: Kangaroo Bot Content-Signal: ai-train=no Disallow: / User-agent: Cotoyogi Content-Signal: ai-train=no Disallow: /