Search Engine Bots (robots.txt)

Search Engine Bots (robots.txt)

Acquia Source CMS provides users the ability to directly customize the robots.txt file and includes built-in support for Answer Engine Optimization (AEO) standards. This functionality assists AI agents to discover, read, and interact with site content, ensuring full visibility to autonomous AI systems and agentic browser workflows. The configuration includes individual controls to manage how traditional search engines, AI crawlers, and automated agents interact with site content allowing the site to pass most readiness checks with minimal configuration.

Prerequisites

Ensure that users have the following roles and permissions:

• The Site Administrator role.
• The Administer discovery files permission.

Configuring robots.txt 

You can declare how automated systems process your content through standardized signals, agent-specific allowances, and custom path rules. Source CMS compiles your preferences and automatically appends them to your site's compiled robots.txt file.

To configure content signals:

1. Log in to Source CMS to open the Sites Dashboard.
2. Locate the site that you want to configure.
3. Click Edit Site.
4. On the left navigation menu, click Configuration.
5. Select Search and metadata.
6. Select Search Engine Bots.
7. Apply your desired configurations across the available fields. For more information, refer to Configuration Sections.
   Note

   You must save the form to view your updated output in the Preview block.

Configuration Sections

Use the following sections within the Search Engine Bots page to configure your robots.txt directives and manage automated agent traffic:

Content-Signal directives

Set specific signals to declare how automated systems process site data. Technical definitions for these external statuses are maintained at contentsignals.orgcontentsignals.org.

You can configure a state for each of the following signals:

For each signal, select one of three states:

• Allow: Permits the action.

• Deny: Prevents the action.

• Unset: Omits the directive entirely.

After you save the configuration, the robots.txt file includes your preferences. For example, a user concerned about AI models training on proprietary content can set the ai-train signal to Deny while keeping the search signal set to Allow.

AI Crawler Rules

Manage allowances for specific AI bots. Several known crawlers populate in this section by default. 

To block a specific model, change its directive to Disallow: /.

Custom Rules

Append site-specific path restrictions. 

For example, add Disallow: /london2026/registration to prevent crawlers from hitting a registration form while allowing them to index the main event page.

System Block (Read-Only)

A non-editable list of default directory and file rules generated automatically by Drupal core.

Preview

A read-only window displaying the combined output exactly as it will serve at /robots.txt. This window updates only after you save the configuration

Automated Discovery Protocols

While configuring your robots.txt file provides direct control over bot traffic, Source CMS also includes passive features designed to assist autonomous systems and agentic workflows. No manual configuration is required to enable these features.

Markdown content negotiation

Source CMS automatically serves Markdown versions of page content to AI agents, which eliminates manual configuration. 

The Markdown content negotiation performs the following actions:

• Returns a Markdown-rendered version of page content when an AI agent sends a request with the Accept: text/markdown header.
• Sets Content-Type: text/markdown on Markdown responses.
• Includes an x-markdown-tokens header in responses to help agents estimate token costs.
• Adds a link tag with rel="alternate" and type="text/markdown" in the HTML head of every page so that agents that parse HTML discover Markdown availability before they make a separate request.
• Serves standard HTML by default for all non-negotiated requests to preserve browser behavior.

Linking Response Headers and API Catalog

Source CMS passively advertises site-level resources through HTTP Link headers per RFC 8288 and publishes a machine-readable API catalog. This setup ensures AI agents discover available endpoints without reading documentation. These features exist in the system by default and require no user configuration

Link Headers: Source CMS injects Link response headers on all HTTP responses to advertise site-level resources per RFC 8288. When JSON:API public access is enabled, the system automatically adds a Link header with rel="api-catalog" pointing to /.well-known/api-catalog.

API Catalog: Source CMS serves a machine-readable API catalog at /.well-known/api-catalog in the application/linkset+json format per RFC 9727. The catalog automatically includes entries for JSON:API with anchor URLs and service descriptions that point to the OpenAPI specification. AI agents use this catalog to discover available endpoints and capabilities.

To configure content signals:

1. Log in to Source CMS to open the Sites Dashboard.
2. Locate the site that you want to configure.
3. Click Edit Site.
4. On the left navigation menu, click Configuration.
5. Select Search and metadata.
6. Select Search Engine Bots.
7. Apply your desired configurations across the available fields. For more information, refer to Configuration Sections.
   Note

   You must save the form to view your updated output in the Preview block.

Configuration Sections

Use the following sections within the Search Engine Bots page to configure your robots.txt directives and manage automated agent traffic:

Content-Signal directives

Set specific signals to declare how automated systems process site data. Technical definitions for these external statuses are maintained at contentsignals.orgcontentsignals.org.

You can configure a state for each of the following signals:

For each signal, select one of three states:

• Allow: Permits the action.

• Deny: Prevents the action.

• Unset: Omits the directive entirely.

After you save the configuration, the robots.txt file includes your preferences. For example, a user concerned about AI models training on proprietary content can set the ai-train signal to Deny while keeping the search signal set to Allow.

AI Crawler Rules

Manage allowances for specific AI bots. Several known crawlers populate in this section by default. 

To block a specific model, change its directive to Disallow: /.

Custom Rules

Append site-specific path restrictions. 

For example, add Disallow: /london2026/registration to prevent crawlers from hitting a registration form while allowing them to index the main event page.

System Block (Read-Only)

A non-editable list of default directory and file rules generated automatically by Drupal core.

Preview

A read-only window displaying the combined output exactly as it will serve at /robots.txt. This window updates only after you save the configuration

Automated Discovery Protocols

While configuring your robots.txt file provides direct control over bot traffic, Source CMS also includes passive features designed to assist autonomous systems and agentic workflows. No manual configuration is required to enable these features.

Markdown content negotiation

Source CMS automatically serves Markdown versions of page content to AI agents, which eliminates manual configuration. 

The Markdown content negotiation performs the following actions:

• Returns a Markdown-rendered version of page content when an AI agent sends a request with the Accept: text/markdown header.
• Sets Content-Type: text/markdown on Markdown responses.
• Includes an x-markdown-tokens header in responses to help agents estimate token costs.
• Adds a link tag with rel="alternate" and type="text/markdown" in the HTML head of every page so that agents that parse HTML discover Markdown availability before they make a separate request.
• Serves standard HTML by default for all non-negotiated requests to preserve browser behavior.

Linking Response Headers and API Catalog

Source CMS passively advertises site-level resources through HTTP Link headers per RFC 8288 and publishes a machine-readable API catalog. This setup ensures AI agents discover available endpoints without reading documentation. These features exist in the system by default and require no user configuration

Link Headers: Source CMS injects Link response headers on all HTTP responses to advertise site-level resources per RFC 8288. When JSON:API public access is enabled, the system automatically adds a Link header with rel="api-catalog" pointing to /.well-known/api-catalog.

API Catalog: Source CMS serves a machine-readable API catalog at /.well-known/api-catalog in the application/linkset+json format per RFC 9727. The catalog automatically includes entries for JSON:API with anchor URLs and service descriptions that point to the OpenAPI specification. AI agents use this catalog to discover available endpoints and capabilities.