Introduction¶
This article gives answers to some frequently asked questions about the Acquia Optimize crawler.
What is Monsidobot?¶
Monsidobot is the name of the Acquia Optimize web crawler. It is the tool that is used to do a scan on websites as requested by our clients as part of our offering.
I’m not a client. Why do I have requests from Monsidobot?¶
This happens because one or more of our clients has links to your website on their website. We make requests to determine if the links that they have on their site are still valid.
For non-client links, the crawler attempts to make a HEAD request in order to be as resource-frugal as possible. Unfortunately, not all websites support HEAD, and so for a range of non 2XX response codes, the crawler tries to determine if the link is actually in a non-functional state or if the website does not support HEAD requests.
Why does Monsidobot make a request even though the link is disallowed in robots.txt?¶
The request is made because we need to verify the status of the link. Unless you are a client, Acquia has no interest in the content of your website and will not save any data nor use it in any way outside of the response code for verification of the status of the link.
Additional resources
- For information about the Acquia Optimize crawler, visit What is Monsidobot?.
- For information about how to set up and configure scans, visit Configure Domain Scans.
- For any other questions or concerns, contact support.