CMS rate limiting on bots and crawlers

This article also answers the questions:

What is CMS Rate Limiting?

Is my CMS slowing down the scan on my website?

Can Shopify slow down my scan?

Introduction

Certain CMS providers may implement a rate limit on all website bots and crawlers. This causes the scan to run very slowly on those websites. Any other web governance/accessibility software that performs a scan on their websites is also affected by this.

Known CMS Providers that Limit Bots and Crawlers

Current CMS providers that apply rate limits include:

• Shopify.

Recommended Solutions

We recommend the following actions:

• Make sure the scan speed is set to slow.

  See the user guide article for instructions:

  • Configure Scans.

• Change the scan frequency to bi-weekly or monthly.

  See the user guide article for instructions:

  • Configure Scans.

• Apply path constraints and/or excludes to reduce the scope of the scans. For product websites, you can often exclude variations of the same product and only scan the root product page.

  See the user guide article for instructions:

  • Path Constraints and Link Excludes.

• Split up the domain into multiple scans. Our customer support team can help you determine if this is a good option, and can assist with the domain split.

  Contact the support team if you feel like this is the best option for you.

How will this help?

• The server answer time may be adversely affected if the connections per minute setting is Faster or higher. At these settings, the speed of the scan may not change if the server is slow to respond.
• We can do a maximum of five scans per IP address simultaneously. When several websites use the same IP, the scan uses the slowest connections per minute setting from the IP address group.

Additional Resources

This section gives some links to more resources that can be useful for this topic.

For more information, see the user guide articles:

• Configure Domain Scans
• Scan with Source Code Exclude.