Acquia Source CMS provides users the ability to directly customize the robots.txt file and includes built-in support for Answer Engine Optimization (AEO) standards. This functionality assists AI agents to discover, read, and interact with site content, ensuring full visibility to autonomous AI systems and agentic browser workflows. The configuration includes individual controls to manage how traditional search engines, AI crawlers, and automated agents interact with site content allowing the site to pass most readiness checks with minimal configuration.
Prerequisites
Ensure that users have the following roles and permissions:
The Site Administrator role.
The Administer discovery files permission.
Configuring robots.txt
You can declare how automated systems process your content through standardized signals, agent-specific allowances, and custom path rules. Source CMS compiles your preferences and automatically appends them to your site's compiled robots.txt file.
Use the following sections within the Search Engine Bots page to configure your robots.txt directives and manage automated agent traffic:
Content-Signal directives
Set specific signals to declare how automated systems process site data. Technical definitions for these external statuses are maintained at contentsignals.orgcontentsignals.org.
You can configure a state for each of the following signals:
Signal
Description
search
Specifies whether AI systems use the content in search results and answer generation.
ai-input
Specifies whether AI systems use the content as input for other tasks.
ai-train
Specifies whether AI systems use the content to train models.
For each signal, select one of three states:
Allow: Permits the action.
Deny: Prevents the action.
Unset: Omits the directive entirely.
After you save the configuration, the robots.txt file includes your preferences. For example, a user concerned about AI models training on proprietary content can set the ai-train signal to Deny while keeping the search signal set to Allow.
AI Crawler Rules
Manage allowances for specific AI bots. Several known crawlers populate in this section by default.
To block a specific model, change its directive to Disallow: /.
Custom Rules
Append site-specific path restrictions.
For example, add Disallow: /london2026/registration to prevent crawlers from hitting a registration form while allowing them to index the main event page.
System Block (Read-Only)
A non-editable list of default directory and file rules generated automatically by Drupal core.
Preview
A read-only window displaying the combined output exactly as it will serve at /robots.txt. This window updates only after you save the configuration
Automated Discovery Protocols
While configuring your robots.txt file provides direct control over bot traffic, Source CMS also includes passive features designed to assist autonomous systems and agentic workflows. No manual configuration is required to enable these features.
Markdown content negotiation
Source CMS automatically serves Markdown versions of page content to AI agents, which eliminates manual configuration.
The Markdown content negotiation performs the following actions:
Returns a Markdown-rendered version of page content when an AI agent sends a request with the Accept: text/markdown header.
Sets Content-Type: text/markdown on Markdown responses.
Includes an x-markdown-tokens header in responses to help agents estimate token costs.
Adds a link tag with rel="alternate" and type="text/markdown" in the HTML head of every page so that agents that parse HTML discover Markdown availability before they make a separate request.
Serves standard HTML by default for all non-negotiated requests to preserve browser behavior.
Linking Response Headers and API Catalog
Source CMS passively advertises site-level resources through HTTP Link headers per RFC 8288 and publishes a machine-readable API catalog. This setup ensures AI agents discover available endpoints without reading documentation. These features exist in the system by default and require no user configuration
Link Headers:Source CMS injects Link response headers on all HTTP responses to advertise site-level resources per RFC 8288. When JSON:API public access is enabled, the system automatically adds a Link header with rel="api-catalog" pointing to /.well-known/api-catalog.
API Catalog:Source CMS serves a machine-readable API catalog at /.well-known/api-catalog in the application/linkset+json format per RFC 9727. The catalog automatically includes entries for JSON:API with anchor URLs and service descriptions that point to the OpenAPI specification. AI agents use this catalog to discover available endpoints and capabilities.
Search Engine Bots (robots.txt)
Acquia Source CMS provides users the ability to directly customize the robots.txt file and includes built-in support for Answer Engine Optimization (AEO) standards. This functionality assists AI agents to discover, read, and interact with site content, ensuring full visibility to autonomous AI systems and agentic browser workflows. The configuration includes individual controls to manage how traditional search engines, AI crawlers, and automated agents interact with site content allowing the site to pass most readiness checks with minimal configuration.
Prerequisites
Ensure that users have the following roles and permissions:
The Site Administrator role.
The Administer discovery files permission.
Configuring robots.txt
You can declare how automated systems process your content through standardized signals, agent-specific allowances, and custom path rules. Source CMS compiles your preferences and automatically appends them to your site's compiled robots.txt file.
Use the following sections within the Search Engine Bots page to configure your robots.txt directives and manage automated agent traffic:
Content-Signal directives
Set specific signals to declare how automated systems process site data. Technical definitions for these external statuses are maintained at contentsignals.orgcontentsignals.org.
You can configure a state for each of the following signals:
Signal
Description
search
Specifies whether AI systems use the content in search results and answer generation.
ai-input
Specifies whether AI systems use the content as input for other tasks.
ai-train
Specifies whether AI systems use the content to train models.
For each signal, select one of three states:
Allow: Permits the action.
Deny: Prevents the action.
Unset: Omits the directive entirely.
After you save the configuration, the robots.txt file includes your preferences. For example, a user concerned about AI models training on proprietary content can set the ai-train signal to Deny while keeping the search signal set to Allow.
AI Crawler Rules
Manage allowances for specific AI bots. Several known crawlers populate in this section by default.
To block a specific model, change its directive to Disallow: /.
Custom Rules
Append site-specific path restrictions.
For example, add Disallow: /london2026/registration to prevent crawlers from hitting a registration form while allowing them to index the main event page.
System Block (Read-Only)
A non-editable list of default directory and file rules generated automatically by Drupal core.
Preview
A read-only window displaying the combined output exactly as it will serve at /robots.txt. This window updates only after you save the configuration
Automated Discovery Protocols
While configuring your robots.txt file provides direct control over bot traffic, Source CMS also includes passive features designed to assist autonomous systems and agentic workflows. No manual configuration is required to enable these features.
Markdown content negotiation
Source CMS automatically serves Markdown versions of page content to AI agents, which eliminates manual configuration.
The Markdown content negotiation performs the following actions:
Returns a Markdown-rendered version of page content when an AI agent sends a request with the Accept: text/markdown header.
Sets Content-Type: text/markdown on Markdown responses.
Includes an x-markdown-tokens header in responses to help agents estimate token costs.
Adds a link tag with rel="alternate" and type="text/markdown" in the HTML head of every page so that agents that parse HTML discover Markdown availability before they make a separate request.
Serves standard HTML by default for all non-negotiated requests to preserve browser behavior.
Linking Response Headers and API Catalog
Source CMS passively advertises site-level resources through HTTP Link headers per RFC 8288 and publishes a machine-readable API catalog. This setup ensures AI agents discover available endpoints without reading documentation. These features exist in the system by default and require no user configuration
Link Headers:Source CMS injects Link response headers on all HTTP responses to advertise site-level resources per RFC 8288. When JSON:API public access is enabled, the system automatically adds a Link header with rel="api-catalog" pointing to /.well-known/api-catalog.
API Catalog:Source CMS serves a machine-readable API catalog at /.well-known/api-catalog in the application/linkset+json format per RFC 9727. The catalog automatically includes entries for JSON:API with anchor URLs and service descriptions that point to the OpenAPI specification. AI agents use this catalog to discover available endpoints and capabilities.
Did not find what you were looking for?
If this content did not answer your questions, try searching or contacting our support team for further assistance.
Did not find what you were looking for?
If this content did not answer your questions, try searching or contacting our support team for further assistance.
Source CMS