International IAB/ABC Spiders & Robots list: How to stop non-human activity that inflate ad impression and site traffic?
What is the International IAB/ABC Spiders & Robots list?
The IAB (Internet Advertising Bureau) US and ABC UK global list for ad and site traffic-related, non-human (robotic) activity, known as the International IAB/ABC Spiders & Robots list, helps filter known, non-human activity that can significantly inflate ad impression and site traffic counts. Effective use of this data can lead to a more transparent and accurate measurement for you and your clients. The implementation of this list is a requirement in the UK by ABC & JICWEBS ‘Web and AV Reporting Standards’ and in the US for the IAB‘s ‘Ad Impression Measurement Guidelines‘ & the MRC‘s ‘Invalid Traffic Detection and Filtration Guidelines‘.
What’s included in your annual subscription?
- The International IAB/ABC Spiders & Robots List
- The International IAB/ABC list of valid browsers
- The ABC IP Address Exclusion List
- Monthly updates
What is the format of the list?
There are actually two text files that make up the International IAB/ABC Spiders & Robots List – one that identifies known robots and one that identifies valid browsers. Both are lists of User-Agents. There are therefore two ways of using these lists and you should download the text files that correspond to how you wish to use them. Either using the single-pass method (excluding robots only) or the dual-pass method, recommended by the IAB US (including valid browsers THEN excluding remaining robots).
What type of IPs are provided? Which type of bot are blacklisted?
These IP addresses typically represent activity from external site monitoring tools and from certain robots which use generic User-Agents. The current IP address exclusion list is always available in 3 formats: CIDR, plain text and also in a perl-style regex format.
How big is it?
The Bots file downloads in a text file format and is 40kbs. The IP exclusion list downloads separately and is 3kbs.
How do I gain access?
Access can be direct through our website or we can set up access via FTP. We are looking at the implications of setting up an API.
What is the frequency of updates?
Updates are monthly on or before the 25th of each month
How would I be notified?
We email subscribers with any emergency changes that affect the list in addition to making newly revised lists available through the ABC website each month. Once subscribed you need to log in to the technical area of the ABC website each month to download the list.
“ABC/IAB Robots and Spiders list is a critical resource, providing a central repository where search engines and others committed to transparency have declared themselves.” Larry Goldstein, VP – Research Audit Services, comScore
FACT: ABC has adjusted claimed web traffic figures down by up to 60% due to non-human, robot activity
Traffic Fraud is a serious problem. Addressing this issue together as an industry is essential to preserving the integrity of the online ecosystem and for maintaining trust.
Note: IAB Europe members get a special discount on the subscription rate.