Build a module that declares AI usage and coordinates Markdown alternates, on top of Markdownify¶
Context and problem statement¶
A Drupal site that wants to serve AI agents well needs two things: a Markdown version of each public page, fetched cheaply and cached safely, and a machine-readable statement of how its content may be used (AIPREF Content-Usage, Cloudflare Content Signals, RSL 1.0, llms.txt). In September 2026 the first was largely solved by Markdownify; the second had no Drupal implementation. Which code should fill the gap, and where should it live?
Decision drivers¶
- Prefer configuring or contributing to what exists over writing new code.
- The declaration vocabularies are unstable (three incompatible AIPREF spellings across six draft revisions), so a vocabulary change has to be configuration and a new plugin, not a code change to stored data.
- A Markdown response chosen by
AcceptorUser-Agentmust never enter a cache keyed on the URL alone. - The project seeks security advisory coverage, which rules out depending on unstable releases in the main module.
Considered options¶
- Configure Markdownify, RobotsTxt and llms.txt only.
- Contribute the declarations to Markdownify.
- Contribute the declarations to RobotsTxt and llms.txt separately.
- A new module that depends on Markdownify for conversion and owns the declarations, the format selection, caching, conditional requests and logging.
Decision outcome¶
Chosen option: a new module, agent_surface, depending on Markdownify. Markdownify keeps HTML-to-Markdown conversion and .md routing; RobotsTxt, llms.txt and Response Filesystem Cache stay optional, reached through their hooks or a submodule. Gaps found in Markdownify are reported upstream rather than patched here.
Consequences¶
- Good: the declarations reach every channel (headers, front matter, robots.txt, llms.txt, RSL) from one setting per bundle, and a vocabulary revision is a new serializer plugin.
- Good: the main module depends only on Markdownify, which is covered by the security advisory policy; the alpha llms.txt dependency is isolated in a submodule.
- Bad: the module depends on Markdownify's internals for route matching and its alter hooks, so a Markdownify release can break it. CI tests against its latest release.
Acceptance criteria¶
The gap counts as filled when all of these hold. All were met at 1.0.0.
- Every public route with HTML output has a Markdown alternate whose access matches the HTML route's exactly (the fallback submodule covers routes Markdownify does not).
- Each category (training, inference, search, any automated processing) can be declared Allow, Disallow or Unstated for the site and per bundle, and Unstated emits nothing.
- The declarations appear as response headers, Markdown front matter and robots.txt lines, and, with the submodules, in llms.txt and an RSL licence.
- A Markdown response selected by
AcceptorUser-Agentisprivate, no-store; noVary: User-Agentis emitted. - Conditional requests are answered with a 304 that has no body, page cache hits included.
- Requests for Markdown are logged with an estimate of the tokens saved, and the log is purged by retention.
Pros and cons of the options¶
Configure Markdownify, RobotsTxt and llms.txt only¶
- Good: no new code.
- Bad: none of them can emit
Content-UsageorContent-Signalheaders or front matter, and none has a per-bundle declaration. The declaration half stays unsolved.
Contribute the declarations to Markdownify¶
- Good: one module for site builders to install.
- Bad: declarations apply to HTML pages, robots.txt and llms.txt, which are outside a Markdown converter's scope; the maintainers would carry a fast-moving standards surface unrelated to conversion.
Contribute to RobotsTxt and llms.txt separately¶
- Good: each channel lives in the module that owns it.
- Bad: no shared model of the signals, so the same choice is configured three times and the channels can disagree; headers and front matter still have no home.
A new module on top of Markdownify¶
- Good: one model of the signals feeding every channel; conversion stays upstream.
- Bad: a new project to maintain.