Skip to content

Sites RobotsTxt

Sites RobotsTxt serves each site its own /robots.txt through the RobotsTxt module: the site's text, or the global one while the site's is empty. A sub-site has none of its own; the /robots.txt of www.hydrion.example is Hydrion's and covers Hydrion UK and Hydrion US. The hub answers anonymous visitors, crawlers included, with 404 on /robots.txt: it serves them only its Anonymous hub routes (Sites UI).

Install

A project of its own, drupal.org/project/sites_robotstxt:

composer require drupal/sites_robotstxt
drush en sites_robotstxt

Requires Sites, robotstxt (robotstxt).

Configure

  1. Delete robots.txt from the web root and keep Composer from restoring it: "[web-root]/robots.txt": false under extra.drupal-scaffold.file-mapping in composer.json. Otherwise the web server serves that file on every host.
  2. Write the global text on RobotsTxt's page (/admin/config/search/robotstxt), under Global.
  3. A site writes its own in robots.txt for this site, in its robots.txt section (/admin/sites/{site}/robots-txt, Search and metadata group of the dashboard), with Manage site settings or Administer robots.txt (Site settings).

To refuse every crawler on staging only, tick Value per environment for robots.txt content on the site type and enter the staging text on the section's Environments tab (Environments).

Settings

The setting plugins it adds, with their keys on Site settings:

Admin pages

Path Title
/admin/sites/{site}/robots-txt robots.txt

Last updated 2026-10-16

Used in