Loading...


Related Products


How to allow-list the crawler in Akamai

Introduction

If you use Akamai for content delivery or security services, you may experience that Acquia Optimize scans are blocked or interrupted by the Akamai security tools. Akamai blocks most web crawlers and other bots. This article provides instructions on how to add the Acquia Optimize crawler to the list of allowed crawlers so that the scans are able to run smoothly.

Prerequisites

This section gives the prerequisites needed to set up Akamai so that the Acquia Optimize scan can work in Akamai.

The Bot Manager product must be enabled on your Akamai account in order for you to get access to the allow listing feature.

Instructions

This section provides instructions on how to allow list Acquia Optimize in Akamai.

Akamai offers two methods to allow list crawlers and bots.

Allow list the category

This section provides instructions on how to enable the SEO, Analytics or Marketing Bots category.

  1. Open your security configuration inside the Akamai platform UI. It is located on the menu on the left side of the page, under the domain name.
  2. Select the relevant security policy from the list in the Security Policies section.
  3. Select Bot Management from the menu on the left, in the category Match Target And Protections.
  4. Select Akamai Bot Categories on the list that opens on the right side of the page under Bot Management. The menu expands.

  5. Locate the SEO, Analytics or Marketing Bots category, under Akamai Bot Categories.
  6. Select Monitor from the Action menu.

​Allow-list only the Acquia Optimize crawler

This section provides instructions on how to allow list only the Acquia Optimize crawler as a custom-defined bot

It is possible to allow list a single crawler, for example Acquia Optimize, instead of subscribing to an entire category. To do this, follow these instructions provided by Akamai (an Akamai account is required to access the article).

Information about IP addresses and the Acquia Optimize user agent for the crawler may be needed when you set up a custom-defined bot.

IP addresses used by Optimize:

United States & Canada35.226.117.128
UK & Europe130.211.65.55
Australia,New Zealand, and Japan35.189.0.46

The Heartbeat tool requires a different IP address. If you use the Heartbeat module, allow list these IP addresses instead to ensure that the module functions properly.

34.38.129.137 EU1
34.34.150.42 EU2
34.69.190.50 US2
35.238.182.110 US3

If you do not know which one to use, contact your sales representative.

User Agent:
Mozilla/5.0 (compatible; Monsidobot/2.2; +https://monsido.com/bot.html; [email protected])

Alternative user agent:

Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36

For more information, visit:

Acquia Optimize Data Hosting and Security.

Did not find what you were looking for?

If this content did not answer your questions, try searching or contacting our support team for further assistance.

Back to Section navigation
Back to Site navigation