The IAB (Internet Advertising Bureau) US and ABC UK global list for ad and site traffic-related, non-human (robotic) activity, known as the International IAB/ABC Spiders & Robots list, helps filter known, non-human activity that can significantly inflate ad impression and site traffic counts. Effective use of this data can lead to a more transparent and accurate measurement for you and your clients. The implementation of this list is a requirement in the UK by ABC & JICWEBS 'Web and AV Reporting Standards' and in the US for the IAB's Ad Impression Measurement Guidelines' and the MRC's 'Invalid Traffic Detection and Filtration Guidelines'.
There are actually two text files that make up the International IAB/ABC Spiders & Robots List - one that identifies known robots and one that identifies valid browsers. Both are lists of User-Agents. There are therefore two ways of using these lists and you should download the text files that correspond to how you wish to use them. Either using the single-pass method (excluding robots only) or the dual-pass method, recommended by the IAB US (including valid browsers THEN excluding remaining robots).
These IP addresses typically represent activity from external site monitoring tools and from certain robots which use generic User-Agents. The current IP address exclusion list is always available in 3 formats: CIDR, plain text and also in a perl-style regex format.
The Bots file downloads in a text file format and is 40kbs. The IP exclusion list downloads separately and is 3kbs.
Access can be direct through our website or we can set up access via FTP. We are looking at the implications of setting up an API.
Updates are monthly on or before the 25th of each month
We email subscribers with any emergency changes that affect the list in addition to making newly revised lists available through the ABC website each month. Once subscribed you need to log in to the technical area of the ABC website each month to download the list.
"ABC/IAB Robots and Spiders list is a critical resource, providing a central repository where search engines and others committed to transparency have declared themselves.” Larry Goldstein, VP – Research Audit Services, comScore
Traffic Fraud is a serious problem. Addressing this issue together as an industry is essential to preserving the integrity of the online ecosystem and for maintaining trust.
To stop non-human activity that inflate ad impression and site traffic, download the registration form below and send it to the IAB Europe Team: communication@iabeurope.eu.
Note: IAB Europe members get a special discount on the subscription rate.
