Introduction
This document provides tasks that you can do to troubleshoot scan issues before you contact support.
Allow-list the crawler throughout your hosting setup
Remember to include your CMS, WAF, security tools, CDNs, and any others that you use.
For information about the IP addresses including those for Uptime monitoring, the ports used, and user agents, read Data hosting and security.
Ensure that your hosting infrastructure is able to process the amount of requests generated by the crawler
It is possible to set a slower scan speed from within UI:
- Slow: 15 connections per minute.
- Normal (recommended): 60 connections per minute.
- Faster: 90 connections per minute.
- Very fast (be cautious): 120 connections per minute.
- Superfast (advanced users only) 180 connections per minute.
Notes
- Some CMS providers limit the scan rates of their users. To prevent errors, decrease the connection speed of your scan. For more information, read CMS rate limiting on bots and crawlers.
A Connections per minute setting of Faster or higher can increase the server response time. If the server is slow to respond, the scan speed does not change at these settings.
The crawler can perform a maximum of five concurrent scans on a single IP address. If multiple websites share an IP address, the scan applies the slowest Connections per minute setting in that group.
For full instructions for admin users on how to set up the scan and adjust the speed, read Domain scan setup.
Check if you have any password-protected or internal-only pages
The scan cannot proceed if you have not granted access to pages that require a password. For instructions on how to set this up, read Configure the scan for password-protected and internal pages.
Ensure that your CMS does not prevent access to crawlers or bots
Some CMS applications let you prevent search engines from indexing specific pages. These settings use labels like Hide from external search engines or Don't index this page. If you enable this type of setting on a page, the CMS can block the Web Governance crawler. As a result, the crawler does not scan the page.
Additional resources
For more information, read:
If you have tried all of the above and scans still do not run as expected, use the support features inside the application on the main menu toolbar.
Scan troubleshooting
Introduction
This document provides tasks that you can do to troubleshoot scan issues before you contact support.
Allow-list the crawler throughout your hosting setup
Remember to include your CMS, WAF, security tools, CDNs, and any others that you use.
For information about the IP addresses including those for Uptime monitoring, the ports used, and user agents, read Data hosting and security.
Ensure that your hosting infrastructure is able to process the amount of requests generated by the crawler
It is possible to set a slower scan speed from within UI:
- Slow: 15 connections per minute.
- Normal (recommended): 60 connections per minute.
- Faster: 90 connections per minute.
- Very fast (be cautious): 120 connections per minute.
- Superfast (advanced users only) 180 connections per minute.
Notes
- Some CMS providers limit the scan rates of their users. To prevent errors, decrease the connection speed of your scan. For more information, read CMS rate limiting on bots and crawlers.
A Connections per minute setting of Faster or higher can increase the server response time. If the server is slow to respond, the scan speed does not change at these settings.
The crawler can perform a maximum of five concurrent scans on a single IP address. If multiple websites share an IP address, the scan applies the slowest Connections per minute setting in that group.
For full instructions for admin users on how to set up the scan and adjust the speed, read Domain scan setup.
Check if you have any password-protected or internal-only pages
The scan cannot proceed if you have not granted access to pages that require a password. For instructions on how to set this up, read Configure the scan for password-protected and internal pages.
Ensure that your CMS does not prevent access to crawlers or bots
Some CMS applications let you prevent search engines from indexing specific pages. These settings use labels like Hide from external search engines or Don't index this page. If you enable this type of setting on a page, the CMS can block the Web Governance crawler. As a result, the crawler does not scan the page.
Additional resources
For more information, read:
If you have tried all of the above and scans still do not run as expected, use the support features inside the application on the main menu toolbar.
Web Governance