---
title: "How to allow-list the crawler in Akamai"
date: "2024-07-03T06:54:23+00:00"
summary: "Learn how to allow-list the Web Governance crawler in Akamai to ensure smooth scans. Step-by-step guide for marketers and developers."
image:
type: "article"
url: "/web-governance/help/59276-how-allow-list-crawler-akamai"
id: "c4612ff1-ebf3-409f-9aab-d1c025c268e2"
---

Introduction
------------

If you use Akamai for content delivery or security services, you may experience that Web Governance scans are blocked or interrupted by the Akamai security tools. Akamai blocks most web crawlers and other bots. This article provides instructions on how to add the Web Governance crawler to the list of allowed crawlers so that the scans are able to run smoothly.

Prerequisites
-------------

This section gives the prerequisites needed to set up Akamai so that the Web Governance scan can work in Akamai.

The _Bot Manager_ product must be enabled on your Akamai account in order for you to get access to the allow listing feature.

Instructions
------------

This section provides instructions on how to allow list Web Governance in Akamai.

Akamai offers two methods to allow list crawlers and bots.

*   [Allow-list the category](#h_94e5078e5f)
*   [Allow-list only the Web Governance crawler](#h_c4b51c7e00).

### Allow list the category

This section provides instructions on how to enable the SEO, _Analytics or Marketing Bots_ category.

Note

When you allow-list the entire category called _SEO, Analytics or Marketing Bots,_ you permit any bot or crawler that is in that category to run on your website. The list of allowed bots and crawlers in the category is maintained by Akamai. The Web Governance crawler is placed inside this category.

1.  Open your security configuration inside the Akamai platform UI. It is located on the menu on the left side of the page, under the domain name.
2.  Select the relevant security policy from the list in the _Security Policies_ section.
3.  _S_elect **Bot Management** from the menu on the left, in the category _Match Target And Protections._
4.  Select **Akamai Bot Categories** on the list that opens on the right side of the page under _Bot Management_. The menu expands.
    
    ![The location of the options in the steps above on the Akamai platform UI.](https://acquia.widen.net/content/af0aa445-90fa-4f91-8b16-8c450d5137d3/web/Mon_FAQ-AkamaiBotCategoriesButton.png)
    
5.  Locate the _SEO, Analytics or Marketing Bots_ category, under _Akamai Bot Categories_.
6.  Select **Monitor** from the _Action_ menu.
    
    ![The Akamai Bot Categories list.](https://acquia.widen.net/content/2695032c-2d5f-4c26-b962-17344508f70e/web/Mon_DomainSetup-AkamaiBotCategories.png)
    

### ​Allow-list only the Web Governance crawler

This section provides instructions on how to allow list only the Web Governance crawler as a custom-defined bot

It is possible to allow list a single crawler, for example Web Governance, instead of subscribing to an entire category. To do this, [follow these instructions provided by Akamai](https://techdocs.akamai.com/bot-manager/docs/categorize-define-own-bots) (an Akamai account is required to access the article).

Information about IP addresses and the Web Governance user agent for the crawler may be needed when you set up a custom-defined bot.

**IP addresses used by Web Governance**

The Heartbeat tool requires a different IP address. If you use the Heartbeat module, allow list these IP addresses instead to ensure that the module functions properly.

For the current list of IP addresses, visit [Data hosting and security](https://docs.acquia.com/acquia-optimize/data-hosting-and-security).

If you do not know which one to use, contact your sales representative.

**User Agent:**  
`Mozilla/5.0 (compatible; Monsidobot/2.2; +https://monsido.com/bot.html; info@monsido.com)`

Alternative user agent:

`Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36`

For more information, visit:

[Web Governance Data Hosting and Security](/en/articles/4872275-monsido-data-hosting-and-security#h_921aa5410a).