Use path constraints to instruct the scan to only process parts of a domain. The scan compares URLs in the site with the pattern given. URLs that match the pattern are scanned and URLs that do not match the pattern are considered external links. Use link exclusions to instruct the scan to ignore parts of the site with a pattern. For example, /fr/.
The difference between the two is that in path constraints ALL pages under /en are scanned, and nothing else. In the link exclusions, all pages EXCEPT /fr are scanned.
A Regular Expression (regex) is an API that defines string patterns. Use a regex to search, manipulate, and edit a string in Java. Email validation and passwords are two examples of strings where Regex can define the constraints.
Note
Regular Expressions in source code exclusions are not 100% compatible with the
Policies
module. The languages are different (Java and Ruby).
This document provides instructions on how to set up path constraints and link exclusions.
Path constraints
A path constraint can consist of a word or a regular expression. In most cases, users set up path constraints to only crawl pages that contain the specfied parameter, and excludee any URL that does not contain it.
Note
Path constraints are required on path-based domains because without them, the crawler will crawl everything it can find on the domain.
Create a path constraint
Log in to your Web Governance account.
Locate the domain that you want to work with.
Click Action.
Select Edit Domain.
Scroll to the Advanced Domain Options section.
In the Path Constraints section:
Search: Enter a search parameter for matching strings within the Constraint Patterns list.
Constraint pattern: Enter a constraint pattern.
+ Add: Click + to add a new Constraint pattern. An empty row is added to the list.
Note
Some CMS selections have pre-filled parameters, as per the majority of users with those CMS operators. Click Default to remove them if needed. The Default button only appears for certain operators.
Make sure that the URL in the domain settings matches the path constraint. If this is not done, the crawl may go into retry, since the crawler starts on a page that conflicts with the constraint. For example, with a path constraint such as /en/booking, the crawler cannot begin on http://foo.com. The crawler will request http://foo.com, receive the page, and find that no links match http://foo.com/en/booking, and go into retry.
Delete a path constraint
Log in to your Web Governance account.
Locate the domain that you want to work with.
Click Action.
Select Edit Domain
Scroll to the Advanced Domain Options section.
Locate the entry.
Click Delete (a trashcan icon) to delete it from the list.
Link exclusions
Use link exclusions to instruct the crawler to completely ignore certain pages or entire sections of the website. Pages that match the pattern are not included in the scan. Exclude parameterized or duplicate URLs to help prevent inflated page counts and ensure that the scan results accurately reflect the unique content on your site.
Tip! If "Scan subdomains" is turned on for the domain, use the § sign in front of the exclude pattern to match URLs that use the full string instead of the relative one. For example, to exclude the scan for the "blog" subdomain, enter the pattern: §http://blog.foo.bar.
Create a link exclusion
Log in to your Web Governance account.
Locate the domain that you want to work with.
Click Action.
Select Edit Domain.
Scroll to the Advanced Domain Options section.
In the Link Excludes section:
Search: Enter a search parameter for matching strings within the Link excludes list.
Exclude pattern: Enter a pattern to exclude from the scan.
+ Add: Click + to add a new Exclude pattern. An empty row appears in the list.
Delete: Click the trashcan icon to delete an item from the list.
Internal URLs:
Operator: Click the drop-down arrow to select Contains, Starts with, or Regex.
URL: Type a URL in the field.
Delete: Click the trashcan icon to delete the row.
+ Add: Click to add a new Input Selector.
Important
It is possible to do a link exclusion for a link that is attached to an image. The link is then excluded from the scan. However, the image itself could still appear on the SEO and QA pages as an issue to be fixed if it does not meet other requirements (for example, missing ALT text).
Click Delete (a trashcan icon) to delete it from the list.
Examples
Filter out and exclude language-specific paths print pages with a pattern such as /fr/. This instructs the scan to ignore (and not scan) any URL with the pattern, for example http://foo.com/fr/.
Filter out redirected login pages with a pattern such as http://foo.com/bar/login. This instructs the scan to ignore all URLs with the pattern, for example: http://foo.com/bar/login. Relies on regex.
Exclude parameterized URLs to avoid inflated page counts. For example, add a pattern like: posts_per_page= to instruct the scan to exclude URLs such as http://foo.com/bar?posts_per_page=10. Relies on regex.
For advanced instructions on this topic, read CMS Integration.
Frequently asked questions
How to create path constraints and link exclusions
Introduction
Use path constraints to instruct the scan to only process parts of a domain. The scan compares URLs in the site with the pattern given. URLs that match the pattern are scanned and URLs that do not match the pattern are considered external links. Use link exclusions to instruct the scan to ignore parts of the site with a pattern. For example, /fr/.
The difference between the two is that in path constraints ALL pages under /en are scanned, and nothing else. In the link exclusions, all pages EXCEPT /fr are scanned.
A Regular Expression (regex) is an API that defines string patterns. Use a regex to search, manipulate, and edit a string in Java. Email validation and passwords are two examples of strings where Regex can define the constraints.
Note
Regular Expressions in source code exclusions are not 100% compatible with the
Policies
module. The languages are different (Java and Ruby).
This document provides instructions on how to set up path constraints and link exclusions.
Path constraints
A path constraint can consist of a word or a regular expression. In most cases, users set up path constraints to only crawl pages that contain the specfied parameter, and excludee any URL that does not contain it.
Note
Path constraints are required on path-based domains because without them, the crawler will crawl everything it can find on the domain.
Create a path constraint
Log in to your Web Governance account.
Locate the domain that you want to work with.
Click Action.
Select Edit Domain.
Scroll to the Advanced Domain Options section.
In the Path Constraints section:
Search: Enter a search parameter for matching strings within the Constraint Patterns list.
Constraint pattern: Enter a constraint pattern.
+ Add: Click + to add a new Constraint pattern. An empty row is added to the list.
Note
Some CMS selections have pre-filled parameters, as per the majority of users with those CMS operators. Click Default to remove them if needed. The Default button only appears for certain operators.
Make sure that the URL in the domain settings matches the path constraint. If this is not done, the crawl may go into retry, since the crawler starts on a page that conflicts with the constraint. For example, with a path constraint such as /en/booking, the crawler cannot begin on http://foo.com. The crawler will request http://foo.com, receive the page, and find that no links match http://foo.com/en/booking, and go into retry.
Delete a path constraint
Log in to your Web Governance account.
Locate the domain that you want to work with.
Click Action.
Select Edit Domain
Scroll to the Advanced Domain Options section.
Locate the entry.
Click Delete (a trashcan icon) to delete it from the list.
Link exclusions
Use link exclusions to instruct the crawler to completely ignore certain pages or entire sections of the website. Pages that match the pattern are not included in the scan. Exclude parameterized or duplicate URLs to help prevent inflated page counts and ensure that the scan results accurately reflect the unique content on your site.
Tip! If "Scan subdomains" is turned on for the domain, use the § sign in front of the exclude pattern to match URLs that use the full string instead of the relative one. For example, to exclude the scan for the "blog" subdomain, enter the pattern: §http://blog.foo.bar.
Create a link exclusion
Log in to your Web Governance account.
Locate the domain that you want to work with.
Click Action.
Select Edit Domain.
Scroll to the Advanced Domain Options section.
In the Link Excludes section:
Search: Enter a search parameter for matching strings within the Link excludes list.
Exclude pattern: Enter a pattern to exclude from the scan.
+ Add: Click + to add a new Exclude pattern. An empty row appears in the list.
Delete: Click the trashcan icon to delete an item from the list.
Internal URLs:
Operator: Click the drop-down arrow to select Contains, Starts with, or Regex.
URL: Type a URL in the field.
Delete: Click the trashcan icon to delete the row.
+ Add: Click to add a new Input Selector.
Important
It is possible to do a link exclusion for a link that is attached to an image. The link is then excluded from the scan. However, the image itself could still appear on the SEO and QA pages as an issue to be fixed if it does not meet other requirements (for example, missing ALT text).
Click Delete (a trashcan icon) to delete it from the list.
Examples
Filter out and exclude language-specific paths print pages with a pattern such as /fr/. This instructs the scan to ignore (and not scan) any URL with the pattern, for example http://foo.com/fr/.
Filter out redirected login pages with a pattern such as http://foo.com/bar/login. This instructs the scan to ignore all URLs with the pattern, for example: http://foo.com/bar/login. Relies on regex.
Exclude parameterized URLs to avoid inflated page counts. For example, add a pattern like: posts_per_page= to instruct the scan to exclude URLs such as http://foo.com/bar?posts_per_page=10. Relies on regex.
Web Governance