CMS rate limiting on bots and crawlers

Certain CMS providers can set a rate limit on all website bots and crawlers, which causes the scan to run slowly on those websites. This setting also affects other web governance or accessibility software that scans the websites. Currently, Shopify is one of the CMS providers that applies rate limits to web bots and crawlers.

To resolve this issue:

• Set the scan speed to Slow and change the scan frequency to bi-weekly or monthly.

  For instructions, read Domain scan setup.

• Apply path constraints and excludes to reduce the scope of scans. For product websites, you can often exclude variations of the same product and scan only the root product page.

  For instructions, read How to create path constraints and link exclusions.

• Split the domain into multiple scans to reduce the scope.

  Acquia Support can assist with the domain split and help you determine if this option is suitable for your requirements. For assistance, select Contact support in the Help menu to start a chat, or email Acquia Support.

How does this help:

• The server server response time can be adversely affected if the Connections per minute setting is Faster or higher. At these settings, the speed of the scan may not change if the server is slow to respond.
• Maximum five simultaneous scans can be performed per IP address. When several websites use the same IP, the scan uses the slowest Connections per minute setting from the IP address group.