How can I estimate the network bandwidth required by the Web Governance crawler?
Date Published: May 6, 2026
To estimate the network bandwidth required in order to allowlist and run the Web Governance crawler (monsidobot), consider the following:
- Monsidobot's standard scanning speed
- Your average webpage size
For every page included in the scan, monsidobot makes two HTTP requests:
- One standard
GET request - One request to Puppeteer for rendering and accessibility checks
Monsidobot makes two requests per second. The bandwith required is:
2 x (average page size in MB) per second