Bot traffic filtering

Many customers use Acquia Lift to analyze and track their traffic, and use that data to do basic instrumentations like track page views, clicks, and other activities. When bots and spiders crawl your website, that traffic isn’t interesting or relevant when defining segments or visitors.

Profile Manager attempts to filter out traffic from bots and spiders that may crawl your website, reducing false positives for tests and page views.

To filter these hits, Profile Manager uses the IAB/ABC International Spiders and Bots List to whitelist and blacklist user agents so their traffic is not counted for user traffic.

  • What does this list filter?

    The IAB/ABC International Spiders and Bots List aims to consistently filter well-behaving automated traffic from totals used for traffic measurement or billing purposes.

    This list primarily detects ‘good actors’ who self-declare as automated traffic. Some ‘good actors’ must pretend they’re not bots, such as when scanning for malware or brand safety violations.

  • What does this list contain?

    The IAB/ABC International Spiders and Bots List contains two lists, a whitelist and a blacklist, that are used in a three-step process for filtering user-agent strings.

    The first level of filtering (the whitelist) contains text patterns that generously match all known valid user-agent strings.

    The second level of filtering (the blacklist) contains text patterns that match detected spiders and bots that passed through the first step of filtration.

    Each text pattern on the blacklist may also include exceptions.

  • How many entries are on the list?

    The IAB/ABC International Spiders and Bots List uses pattern matching, so the number of entries does not correlate to its efficacy. In fact, a lower number of entries can indicate a better text pattern selection.

Contact supportStill need assistance? Contact Acquia Support

Acquia: Think Ahead

53 State Street, 10th Floor
Boston, MA 02109
United States
Phone: 888-922-7842

Map: Google Maps
View other locations