Monsidobot FAQ

What is Monsidobot?

Monsidobot is the name of the Acquia Web Governance web crawler. It is the tool that does the scan on websites, as requested by clients as part of the Acquia offering.This article provides the answers to several frequently asked questions about Monsidobot.

Why does monsidobot make a request even though the link is disallowed in robots.txt?

The purpose of the request is to verify the status of the link. Unless you are a client, Aquia has no interest in the content of your website and will not save nor use it in any way outside of the response code for verification of the status of the link.

I am not a customer. Why is Monsidobot making requests?

This happens when one or more Aquia Web Governance customers have posted links to your website on their own website. The crawler makes requests to determine the validity of the links.

For non-client links, the crawler attempts to make a HEAD request to save resources. Unfortunately, not all websites support HEAD, and so for a range of non-2XX response codes, the crawler attempts to determine if the link is actually in a non-functional state or if the website does not support HEAD requests.

Advanced information

This section gives advanced information about Monsidobot requests

• Monsidobot only respects crawl-delay on robots.txt if the connections per minute setting on the domain is NOT set.
  If the connections per minute setting IS set for the domain, the crawler ignores the crawl-delay on robots.txt and always uses that setting.
  If the connections per minute is NOT set, then the crawler looks for crawl-delay in the user agent monsidobot. If that does not exist, then crawler looks for googlebot.
• Crawl-delay is capped at max 10 seconds, and lowest at 2 seconds.
  To summarize, Monsidobot never goes below 2 seconds in order to avoid overwhelming the servers, even if users request less, and Monsidobot never waits more than 10 seconds to continue the crawl, even if a site is overly-conservative. Sites that request between 2 and 10 seconds are respected exactly, outside of that range the system overrides the request.

Additional resources

Monsidobot generally respects crawl-delay in robots.txt as long as the value is between 0 and 60. 60 is used for any higher value. For any other questions or concerns contact the Web Governance support team.

IP addresses, Ports, User agents

For more information, visit:

Data hosting and security.