For information about the Acquia Optimize crawler, see:
https://docs.acquia.com/acquia-optimize/admin/configure-scans/what-monsidobot?cid=8fb13¶

Introduction¶

This article gives answers to some frequently asked questions about the Acquia Optimize crawler.

What is monsidobot?¶

Monsidobot is the name of the Acquia Optimize web crawler. It is used to do a scan on websites as requested by our clients as part of our offering.

I’m not a client. Why am I seeing requests from monsidobot?¶

This is happening as one or more of our clients has links to your website and we are making requests to determine if any links that they have on their site are still valid.

For non-client links, the crawler will try to make a HEAD request in order to be as resource-frugal as possible. Unfortunately, not all websites support HEAD, and so for a range of non 2XX response codes, the crawler tries to determine if the link is actually in a non-functional state or if the website does not support HEAD requests.

Why is monsidobot making a request even though the link is disallowed in robots.txt?¶

The request is made because we need to verify the status of the link. Unless you are a client, Acquia Optimize has no interest in the content of your website and will not save nor use it in any way outside of the response code for verification of the status of the link.