Adoption of Supply Chain Transparency Standards in Europe

Digital advertising stakeholders require information about the supply chain to best serve their clients, select their partners, combat ad fraud, and avoid harmful content. From a buyer’s perspective, supply chain transparency is required to perform supply path optimisation: streamlining the paths between advertisers and buyers to maximise the efficiency of ad spend. The digital ad industry, through IAB Tech Lab working groups, has aligned on supply chain transparency standards such as ads.txt, sellers.json, and the SupplyChain object.

These standards aim to:

• Enable the verification of commercial relationships between publishers and ad tech intermediaries.
• Prevent the monetisation of counterfeit inventory, allowing buyers (i.e., ad tech intermediaries representing advertisers) to reject it before a bid is ever placed.
• Provide information on the depth of the publisher’s supply chain to buyers.
• Provide information that uniquely identifies ad tech companies within authorities such as the Trustworthy Accountability Group that provide certifications on issues such as fraud or brand safety.

Each publisher’s ads.txt file lists every exchange or reseller authorised to sell their impressions, letting buyers reject counterfeit inventory before a bid is ever placed. Sellers.json is the mirror image, published by each exchange or SSP to reveal information relating to certification under industry schemes and every seller it represents. The SupplyChain object, part of the OpenRTB specification for real-time bidding, embeds the full sequence of verified intermediaries inside each bid request so buyers can confirm which parties would be participating in the transaction.

The efficacy of Authorised Digital Sellers and Sellers.json depends on good adoption of the standards across the digital ad ecosystem. To evaluate the level of adoption of supply chain transparency standards, a sample of 2054 European online news publishers was assessed. The sample was compiled by combining third-party lists of online news publishers in European digital advertising markets and manually verifying that they serve ads. Web scraping tools were employed to collect ads.txt files while respecting established guidelines (respecting robots.txt, rate limiting, etc.). It should be noted that whether the publishers sell ad inventory programmatically was not evaluated, i.e., not all of the publishers in the sample need to have ads.txt. Furthermore, in some cases, the collection failed due to technical issues (e.g., robots.txt preventing the scrape) despite the fact ads.txt files are meant to be accessible through such collection methods. Overall, it was found that at least 72.64% of European web publishers in the sample hosted ads.txt files.

The collected pages were then analysed - first through an automated system that flagged files likely not to contain information according to the ads.txt specification and then through manual verification. 42 pages were flagged and manually verified not to contain ads.txt. The remaining 1492 ads.txt files were then parsed in accordance with the standard and using an automated system that flagged any lines with errors. Examples can be found in the table below. In total, 99.83% of lines were found to be valid.

Categories of ads.txt lines

The valid ads.txt lines were then analysed to obtain a list of unique ad system domains. Sellers.json collection was attempted using the same scraping tools as before and was successful about 61% of the time. It should be noted that the low rate of success is largely due to errors in ads.txt - certain lines were valid but contained incorrect ad system domains, and the list of unique ad system domains was not manually verified. It was observed that certain scraped pages contained the information normally found in sellers.json, but were not loadable due to errors such as missing brackets. An automated check was introduced, and 47 pages were estimated to contain unloadable sellers.json.

Errors in ad system domains

Finally, the information contained in ads.txt and sellers.json files were cross-referenced. Each valid ads.txt line was matched with a JSON block from the referenced ad system’s sellers.json file using the ID attribute. About 79% of valid ads.txt lines were successfully matched with sellers.json files.

The project yielded some additional information about the ads.txt files of the European web publishers in the sample, such as the split between DIRECT and RESELLER paths and the frequency of different variables, which can be found below.

Finally, the following recommendations were identified for publishers and ad systems:

1. Access
   1. Ensure ads.txt / sellers.json can be found in the root domain as per the standards or one redirect away:
      1. http://{publisher_domain}/ads.txt
      2. http://{advertising_system_domain}/sellers.json
   2. Exclude ads.txt / sellers.json from the scope of access control methods such as robots.txt.
      
2. Validation
   1. Ensure ads.txt files are valid either through manual verification or an automated tool. Specifically, check that:
      1. The format of each line (3 or 4 fields separated by commas) is correct
      2. Any comment is properly excluded using the ‘#’ symbol.
      3. The Type of account/relationship field is populated with one of two valid values (RESELLER or DIRECT).
      4. The ad system domain is valid and hosts a sellers.json file.
   2. Ensure that sellers.json files are loadable.

If you would like to know more about our work on the adoption of supply chain transparency standards and how you can get involved in our Programmatic Working Group, please contact our Data Analyst & Sustainability Lead, Dimitris Beis at beis [at] iabeurope.eu.