By default, pages and assets within your Acquia DAM (Widen) site will not be indexed by search engines like Google. This includes assets embedded on other sites and portals that are set to public.

Enable crawling

Crawling can be enabled for any asset type that supports a valid CDN embed link, including images, PDFs, video file types, SVGs, and audio files. For a comprehensive list of supported file formats, visit What are the supported file types?. If you would like to enable search engine crawling on your assets, contact the Acquia DAM (Widen) Support team.

Once crawling is enabled, all assets with valid embed links are crawled. The setting is either on or off globally, you cannot customize it to specific file types.

We do not support custom modifications to the robots.txt file. Acquia DAM (Widen) manages the user agent list for robots.txt, and it cannot be customized by the customer.

Overview

Crawling enablement in Acquia DAM (Widen) is global. It applies to any asset that has a valid CDN embed link. You cannot enable or disable crawling per asset group, collection, or taxonomy. Once enabled, assets with valid embed links are eligible for discovery by compliant crawlers when the asset URLs are publicly accessible.

What this means

• Scope: The setting applies globally across your entire DAM tenant for all assets with valid embed links.
• No per-group control: There is no built-in switch to limit crawler access to specific asset groups only.
• Access still matters: Assets are available through embed links only when they are released and not expired. If an asset is not released, or if it has expired, authentication and authorization are required to view it, and it will not be available to search crawlers.

Controlling crawler access

The primary way to control whether or not an asset is crawlable is through its release and expiration status:

• Keep assets unreleased: Assets that are not released require authentication and are not accessible to crawlers through embed links.
• Use expiration dates: Expired assets also require authentication and will not be crawled.
• Tokenized or expiring links (API only): The Acquia DAM (Widen) API supports signed or expiring URLs. However, these are separate from embed links and are not the same mechanism that crawlers access. Use these for controlled sharing outside of the embed/crawling workflow.

• Custom modifications to the robots.txt file
• Custom user agent allowlists or blocklists
• Placing the Acquia CDN behind your own CDN or WAF (embed links remain reachable by crawlers regardless)
• Multiple delivery domains for embeds
• Per-asset or per-group firewall rules

Best practices

• Make a clear decision per asset class (public versus private) and manage release status and expiration dates accordingly.
• Regularly test representative public asset embed URLs from outside your corporate network to confirm they respond with HTTP 200 for permitted crawlers.
• For assets that should not be discoverable by search engines, ensure they remain unreleased or expired.

Can my assets be indexed by search engines?