Thoughts on the proposed IAB Guidelines

UPDATE ON JANUARY 19, 2009: Peter Black from BPA Worldwide who was also on the IAB working group with Josh Chasin, wrote in and disagrees with Chasin’s characterization of who the “Unique User” language is targeting. I have email into the IAB and MRC’s George Ivie to clarify the situation. Watch this blog!

UPDATE ON JANUARY 18, 2009: Josh Chasin from comScore, who was a member of the IAB working group that defined the guidelines described in this post, wrote in to point out that I misinterpreted the IAB’s intent. While their web site clearly says …

“The IAB believes that all companies involved in audience measurement should be audited for their processes. These audits are intended to establish the source of any measurement discrepancies and to find potential solutions.

All measurement companies that report audience metrics have a material impact on interactive marketing and decision-making. Therefore, transparency into these methodologies is critical to maintaining advertisers’ confidence in interactive, particularly now, as marketers allocate more budget to the platform.”

… according to Chasin the IAB is excluding web analytics vendors from “all companies involved in audience measurement” and the type of companies that have a material impact on interactive marketing and decision-making. Since this doesn’t sound right to me at all I will warn the reader that some of the questions I raise in the following post may, in fact, be totally irrelevant (at least in the context of the IAB Proposed Measurement Guidelines.

If nothing else, with two days left in the open comment period, the IAB may want to use my confusion as an example and clarify the target for the recommendations made in the document.

Reader beware!

As long as we’re talking about web analytics standards I figured I would take the opportunity to offer up a few thoughts on the Interactive Advertising Bureau’s Audience Reach Measurement Guidelines that are open for public comment until January 20th. If you haven’t had a chance to read these proposed guidelines you should, especially if you have an interest in how we collectively communicate about data.

At 34 pages the document is certainly a slog to read–and I say this knowing full well that I have a tendency to write 50 page white papers! Since you’re all bright folks I’m just going to address some of the proposed language that stood out to me. And, as always, if you have any thoughts or positions on the proposed guidelines I’d love to hear from you!

Starting in Section 1.2 the IAB clarifies the relationship between “Unique Cookies”, “Unique Browsers”, “Unique Devices” and “Unique Users / Visitors”. The discussion about “Unique Devices” is interesting because this is a clear indication of the impact that mobile devices like the iPhone are having on audience measurement. Things start to get really interesting, however, in Section 1.2.4 where the IAB says (emphasis mine):

“However, in order to report a Unique User, the measurement organization must utilize in its identification and attribution processes underlying data that is, at least in a reasonable proportion,, attributed directly to a person. For instance, data collected from registrants is one possible source that can be utilized in creating a Unique Users measure by a census-based measurement organization, if registrants represent a reasonable proportion of the total user-base and when appropriate scientific projection methods are used for non-registrrants (sic). In no instance may a census measurement organization report Unique Users purely through algorithms or modeling that is not at least partially traceable to information obtained directly from people, as opposed to browsers, computers, or any other non-human element.“

Did you get that? Keep in mind that while at JupiterResearch I was among the first to publicize the decline in accuracy of visitor counting due to cookie deletion. In fact the report was subtitled “Addressing the Decline in Accuracy of Cookie-Based Measurement.” At the time people called me crazy and Seth Godin even accused me of living in an echo chamber (I have since forgiven Seth.)

Now, three years later, the IAB is expressly telling measurement vendors to stop reporting a metric called “Unique Visitors” or “Unique Users” unless they have a research-based strategy for determining the correct proportion of cookies to “real people” and have applied that calculation in a transparent way.

Whoa.

Think about this for a minute. Every one of the fine census measurement packages (nee web analytics) out there is reporting a Unique Visitor number, but I’ll go out on a limb here and propose that none of them are even vaguely adhering to the IAB proposed definition of a “Unique Visitor.” I’ll go a step further and postulate that, at least in the base offerings, these vendors don’t currently have the technical capability required to report an estimated/algorithmically derived “Unique Visitor” count based on scientific projection methods.

If I’m wrong about this I suspect I’ll hear about it, but I don’t think I’m wrong when it comes to the base offerings like SiteCatalyst, WebTrends Web Analyics, Coremetrics 2009, etc. And yes, I’m aware that end-users can use higher-end products like Discover on Premise and the data warehousing tools to apply a correction factor to UV counts, but that is not what the IAB is saying. This guideline is saying that correcting for cookie, browser, and device-related over-counting of unique visitors is the responsibility of the measurement vendor.

Again, whoa.

And as if that’s not a radical enough move, the document goes on to state in Section 2.2 that the vendors need to actually break out these correction factors across three components: first-cookie acceptance, deletion, and browser denial (again, emphasis mine):

“Cookie deletion rates, calibration methods and sources or estimation methods used to account for first-use, deletion, and non-accepting cookie groups should be disclosed by the audience measurement organization. The audience measurement organization should disclose census-based unique cookie counts and the estimated unique activity from first-use, deletion and non-accepting cookie groups separately and in aggregate. If the measurement organization relies on a unified model that makes reporting among these separate groups impossible, it may report these counts in aggregate only, but should be prepared to demonstrate in an audit the ability of its unified model to address each type of cookie completely.”

The IAB goes on in Section 2.4 to start to push web analytics into what is an uncomfortable position for some people, the use of algorithms and data models, to better report on unique visitors:

“As noted above, Publishers and Ad-servers will generally need to rely on algorithms (data models) to estimate the number of users attributable to the counts of Unique Cookies they develop. The underlying basis for this algorithm should be a study of actual users (i.e, people). Ideally, such a study would be based on direct contact and/or observation of people using the browser at the time of accessing web-site content or ads with the unique cookie, as well as observation of the number of browsers in use by these users. Additionally, inferences will need to be made about advertising activity of users with non-cookied browsers, so these types of users should also be contacted and observed. Also, the activity of users who access content from multiple locations (home, work, school) on different browsers should be factored into these algorithms.”

Finally, the IAB is telling the vendors they need to report the results of their research to their customers, essentially exposing flaws in their technology for all to see:

“The resulting study should be representative of, and projectable to, the users of the web-site or property, and periodically re-performed to reflect gradual changes in audience. Known weaknesses in the projection processes should be disclosed to users of Audience Reach Measurements.“

If you’re keeping track, the IAB is telling the vendors A) to completely change their definition of “Unique Visitors”, B) to start to actively research sources of inaccuracy on behalf of their customers, and C) pro-actively report known weaknesses in their system to their customers. Anyone want to place any bets on when the vendor community will adopt these recommendations? I’m going to be a little snarky here and put my money on “never in a million years.”

Seriously you have to love the IAB for putting this out there. Unlike the Web Analytics Association’s Standards which I believe are an excellent start but are a little soft in areas, the IAB is basically telling the measurement vendor community that they are doing the entire world a disservice by reporting unique visitor counts that are complete bollocks and they need to stop doing that post-haste! Okay, maybe I’m over-reading the document but the scope of changes required for any vendor to become IAB-compliant is dramatic, both technically and psychologically.

I’m not sure if Brandt Dainow had seen the IAB proposal when he besmirtched the fine work of the Web Analytics Association’s Standards Committee, but if you compare the two proposal documents (the WAA’s proposal can be found here in PDF form) you will detect a noticable difference. Personally I’m glad that my good friend Judah Phillips bridged the gap between the IAB and WAA and I find myself wondering, at least a little bit, whether the IAB+WAA relationship should be even deeper.

This all brings me to an excellent point that Bryan Robertson made on my last post on standards regarding how standards are defined and moved into common use. Bryan’s thesis is based on the W3C’s move from HTML 1.0 to XHTML and his point is that this transition to the XHTML standard came about because of A) a powerful standards body, B) a vocal community, and C) passionate thought leaders. Regarding a powerful standards body, Bryan specifically make a point that other folks have made, usually behind closed doors:

“Is the WAA powerful enough at this point in time, or do we need to continue to build momentum before the standards can be more bold? For example, is the WAA hand wringing too much over the polite “we’ll share with you if you share with us” arrangement with the IAB over standards definitions? Is the WAA in a tough position in trying to bring practitioners and vendors together at the same table?”

Bonus points to Bryan for willing to be direct on the conflict of interests arising from having two masters, vendor and practitioner. Again, I have nothing but profound respect for Angie and all of the other members of the WAA Standards Committee, but since I do know that vendors participated in the definition process I wonder a little bit how much impact they really had.

Anyway, I’m doing all the talking here and it’s a beautiful day so I will ask what you all think — either about the IAB proposal, Brandt Dainow’s assertion, Bryan’s thesis about the strength of the WAA, or anything else that strikes your fancy. Do you think the IAB standard for “Unique Users” has a snowball’s chance of being widely implemented? Do you think Brandt Dainow makes a good point (even if he does it in a lousy way)? Do you think the WAA may be better off working more closely with the IAB on Standards, given the IAB’s relative might?

My site host assures me that my comments table will not crash again so I look forward to hearing from you all.

Posted Wednesday, January 14th, 2009 | 23 responses | Add a Comment | Share, Save or Email

Bob Page

Sigh. While I applaud the IAB for the attempt, I’m concerned about all the ramifications to get closer to the “correct” number. This is statistics, not accounting. Some margin of error is going to happen.

If I were a vendor of commercial web analytics software, I’d add a user-settable “fudge” variable (e.g. 30%), and simply multiply by that to report the number. The user would have to disclose the fudge amount.

But defending the “algorithm” requires per-site research and calculations, which most sites won’t bother to do.

Michael Stone

As a specialist provider of analytics for mobile web sites, our view has always been that mobile provides a much greater opportunity to report true ‘unique visitor’ numbers rather than ‘unique cookies’ which is the reality for desktop web sites.

Mobile phones are far more personal devices and less likely to be ’shared’ in the same way a desktop PC is can be used by multiple people in the same household.

Furthermore, the underlying technology means that the handset’s unique identifier (MSISDN or IMSI) are available to the wireless carrier. In some cases, these identifiers are passed outside the carrier network in one form or another.

Our Mobile Analytics solution utilises packet capture at the egress of the mobile web servers and utilises these identifiers where available. For example, a major wireless carrier in Australia adds an HTTP header which is a unique customer identifer linked to their CRM system. This is the same identifier for the customer for their mobile, fixed and broadband services and is consistent for their life as a customer.

It cannot be deleted or blocked by the handset so is therefore a near perfect method to identify a unique visitor.

Most of the major US wireless carriers provide a similar method of identification.

We are also providing mobile web analytics to carriers from the ‘inside’ of the network, in which case we have these handset identifiers available for all browsing requests i.e. near perfect unique visitor counts!

Bring on the IAB definitions!

Michael Stone
CEO
Amethon Solutions
http://www.amethon.com

La IAB pone contra las cuerdas a los Usuarios Unicos. « La compreria

[...] 15, 2009 por clotet Según he podido leer hoy en el blog de E. Peterson en la última versión de la Guía de Estadares de la IAB las empresas de Medición deberían [...]

Brian Clifton

Hi Michael

Interesting to hear you have a “near perfect unique visitor counts” technique, but to be honest it scares me!

A handset unique identifier that “cannot be deleted or blocked by the handset” has major privacy implications, which as a user I would certainly not be happy with.

Then again, if mobile carriers were to reduce their ridiculously over priced tariffs as a consequence of having such user insight, maybe I would think again. My experience though is that mobile users have very little brand loyalty to carriers because of the feeling of paying through the nose.

Deleting/blocking cookies (or whatever the tracking method is) is a fundamental right of the end user. Marketers and web owners have to recognise and embrace that.

Jeopardising an entire industry (i.e. how people use the web) in the quest for an “accurate” unique visitor count is flawed. The return value of bragging rights and setting advertiser rate cards doesn’t justify the means. Surely metrics such as engagements per visit, recency and loyalty can be standardised in some way to provide an alternative with this unique visitor obsession…?

Brian Clifton

To add further to this debate…

The biggest flaw by far I come across when interpreting metrics for clients is their implementation of the tool(s).

A standardisation of what a best practise implementation should look like would be a great deal more value than pinning down a precise number of unique visitors IMO…

Posts about Internet Marketing Experts as of January 15, 2009 | The Lessnau Lounge

[...] Date: January 14, 2009 2:47:19 PM PST To: “johnmac’s living room” &lt Thoughts on the proposed IAB Guidelines – blog.webanalyticsdemystified.com 01/14/2009 As long as we’re talking about web analytics [...]

Steve Jackson

>>Do you think the IAB standard for “Unique Users” has a snowball’s chance of being widely implemented?<>Do you think Brandt Dainow makes a good point (even if he does it in a lousy way)?<>Do you think the WAA may be better off working more closely with the IAB on Standards, given the IAB’s relative might?<<

Hard to say. Since it’s so hard to report Unique Visitors with their standard some vendors (following IAB standards closely) have simply stopped doing it.

For instance TNS Gallup in Finland (and the Nordics) have a list media sites they publish for the benefit of advertisers. You can fine it here;
http://www.gallupweb.com/tnsmetrix/

It basically calls UV’s “Browsers” and only reports sessions and page views. It used to report Unique Visitors.

It kind of defeats the objective.

What I think the WAA need to achieve is a kind of compromise by defining what the vendors report in technical terms right now and defining a “perfect model” that vendors could work towards in their upgrades. Maybe a sliding scale could be introduced (level 0-5 where 0 is non-compliant with WAA standards and 5 is fully compliant).

I disagree with Bob Page this time (Sorry Bob). Web Analytics is about trend analysis and I never worry about number accuracy as such.

But as the industry matures folks are being judged and paid based on the numbers. We need to eventually be able to be accurate to around a 10% error and that is not the case at the moment. Some browser based tracking is reported 40% in-accurate due to cookie deletion and some 5%.

As a guy in HR how can you tie bonuses or set targets for visitors with that kind of fluctuation? You can’t really do it with any confidence and this is one of the things that needs to happen to get wider adoption inside companies.

There has to be some level of confidence in the numbers or you lose all credibility.

eric

Bob: But let me ask you … outside of the tremendous work you’re doing at Yahoo! and a few other places, is what we’re doing statistics? Or is it just bean counting where some of the beans are easily counted and others not so much so?

I guess I’m gonna call vendors and ask about the “statistical rigor” story and see what they say. Again, my gut feeling is that few models are actually being applied at the vendor-level (more so at the end-user level to be sure.)

So if “most sites won’t bother to do” the work necessary to correct “Unique Browser” counts (IAB term) to come up with more accurate “Unique User” numbers, why do you think the IAB bothered to include all that language? I agree that this is a lot of work but someone sure thinks it is important (someone other than me I mean.)

Michael: Agreed that mobile presents some fantastic opportunities to get closer to “real” numbers across the broader audience, but my understanding of most mobile analytics solutions is that the MSISDN and IMSI are not passed through universally. If that is the case then what you have is a “potential” sample to use in the calculation the IAB proposes, right? But you’d still need to ensure that the sample of MSISDN-providing handsets is representative of the whole (which it may or may not be, I don’t think you can know without testing the theory.)

Brian: I don’t disagree with your assessment that most web analytics is more flawed by poor implementations than the technology itself, but I think the IAB commentary that I published above transcends that concern. Let’s assume for a second that everyone gets their act together and implements properly. We still have (potentially significant) over-counting of audience size due to the cookie-related issues the IAB outlines, don’t we?

I know this is just another one of those “there is not accuracy, get over it” debates but perhaps the fact that it keeps coming up, over and over, means that people don’t want to get “over it” … they want a solution?

Steve: Agreed. I personally love that the industry we’re in continues to evolve. Keeps things fresh, don’t you agree!

I thank you all for your thoughts!

Wes Contreras

When the IAB expects analytics vendors to conduct offline surveys and statistical analysis in order to correct for cookie deletion and the like, how do they expect that vendor to account for the change in demographics from one site to another? How often are they expected to do it?

That is, the people who spend a lot of time at cookiesareevil.com are likely to exhibit different behavior than the viewers of the weloveadvertisers.com site. Less obviously, the cookie settings among males aged 19-21 is going to be very different than females aged 55 and up. Some sites appeal to one demographic, some to another. As a result, any survey is going to fail unless it’s specific to the audience viewing a particular site, and it becomes hard to push that responsibility to the vendor. Or anyone, really, given the money it would take to conduct that much detailed research.

Not to mention that cookie behavior changes with every new browser release and its updates to default behavior, as mobile devices change their capabilities and cookie functionality, and so on. In the rapid-moving world of technology, this type of survey data would have to be collected and reanalyzed at least annually – with all associated costs.

I think I’m on board with Eric when he predicts “never in a million years” as our adoption timeframe, though it’s hard to say whether our reasons are the same.

Adam Howitt

I read both this and your Brandt Dainow post with interest. As an independent Web Analytics specialist I don’t consider myself a statistician but as the title suggests – an analyst. I evolved into the role from a classic web development education and fear projections of uniques based on empirical evidence far more than the deletion of cookies.

To me it’s the old adage of not being able to see the wood for the trees. Analytics packages are designed to make it easy to spot undesirable trends (and desirable ones), design an experiment, test and learn. If you begin to base your experiments on projections instead of real numbers, or care about whether a session timed out in 30 or 35 minutes you risk wasting considerable amounts of your clients money on minutiae.

I think the IAB guidelines justifiably target the type of analytics classed as bean counting – how do you demonstrate the value of your site to advertisers. The WAA should be focussed on the education of analysts durable principles of analysis resulting in value for businesses through website evolution – because we all know cookies get deleted and if we can’t explain this to a CEO such that it doesn’t provide a permanent obstacle to our role then it ceases to be fun.

Personally, I prefer doubling conversion rates based on something I discovered or halving bounce rates because if the base number is 80 or 180, the trends will still be the same.

Josh Chasin

Hey Eric. As you know, I was a part of the IAB work group that developed these standards.

I think it is important to note that, although billed as an IAB document, in fact this is more accurately a collaboration between IAB and the MRC (Media Rating Council.) Indeed, I think it is fair to say that MRC’s George Ivie was the primary author, and George ran most of the work group meetings I attended.

My recollection was that there were, at times, representatives from some of the Web Analytic vendors at the table. Indeed both Webtrends and Omniture are listed as among the project participants (as well as my company, comScore; and Nielsen Online, Hitwise, and Compete.)

Here’s why I point out the prominent role of the MRC in the process: because right now the MRC does two kinds of audits in the online space. There is the research company audit, which comScore and Nielsen are both currently undertaking; then there is the site-centric audit, wherein publishers such as Yahoo, Univision, and the Weather Channel have submitted their site-centric data for accreditation.

In other words, in the MRC world view, and thus in the context of this document, I do not believe the “audience measurement organization” in question is the web analytics vendor. Rrather, I believe that with respect to site-centric data, the audience measurement organization is taken to be the publisher. These guidelines would apply to the website taking internal data public; not to an Omniture or Webtrends providing a publisher with data about their own website, for their own use.

Consider that audience measurement data from a company like comScore is syndicated– available to all subscribers, and ideally (for us) becoming “currency.” Whereas Web Analytics data is often called “internal” data. An Omniture doesn’t publish data on a given website and then make that data available for third parties to make business judgments about that site; rather, it is the publisher who may choose to take what we call “internal” data and share it externally (generally with prospective advertisers and their agencies.) It is the taking of the web analytic data public, or at least outside the organization, that triggers the need for these guidelines. While I certainly cannot speak on the IAB’s behalf, I can say that it was pretty clear to me that the intent of the IAB guidelines is to put the onus on the publisher who chooses to use web analytic data in the ad sales process– not on the analytic vendor who provides tools which that publisher may use to develop and track KPIs and monitor site performance.

This is why, for example, there has been no call for Google Analytics to submit to an MRC audit, whereas there was an immediate call for Google Ad Planner to do so.

Cheers,

–josh–

eric

Josh: Well that is certainly an interesting twist if you’re correct, no doubt you are, and I certainly appreciate your bringing this to my (and my readers) attention. I guess I missed the definition of “measurement organization” somewhere and became confused by references to things like “this document is intended principally to guide the definition and application of measures that are to be used for commercial, revenue-generation purposes” (page 1) and “an MRC Accredited Measurement Organization” (page 11)

Given this it’s good to know that the vendors themselves are off the hook, at least from an IAB and MRC perspective, at least for now. But one wonders if the IAB and MRC are about to bring increased pressure on publishers, how long it will be until the publishers start asking the vendors for help?

Certainly when that day arrives the questions I raised and the comments from readers above become relevant. Even in light of your point, I think it would be interesting to hear from the vendors and publishers regarding their ability to support the type of research required by the IAB.

Or, as is often the case, is the answer “use Excel”?

Thanks for correcting my misunderstanding Josh! I wish someone from MRC or IAB would have spoken up sooner but it has been a good conversation nonetheless.

Peter Black

I too was on the IAB working group and never thought the guidelines were just for publishers. Publishers use vendors to express their traffic data so it makes sense that vendors are encouraged to follow the guidelines as well.

As for Google Analytics, you can bet there are publishers out there using this traffic data for ad sales. Why not push for GA to be audited?

Josh Chasin

Peter, the publisher using GA in the sales process would be under the onus of these guidelines. Not GA.

Analytics and technology don’t move at the same speed. » Geary Fresh

[...] was reading a blog post by Eric Peterson (http://blog.webanalyticsdemystified.com/weblog/2009/01/thoughts-on-the-proposed-iab-guidelines.html), in which he mentions how the IAB is proposing to further define “Unique Visitors/Users,” and [...]

Peter Black

Josh, I understand but if GA is not compliant the publisher can’t hope to be compliant. If GA (and other analytics firms) were to become compliant then every publisher using it would then be also. How great would that be for building a critical mass of publishers adhering to the guidelines. The alternative is to push for adoption on a publisher by publisher basis and that is akin to building the pyramids- a long, slow process.

Joe Chasse

Great post Eric and certainly some interesting interpretations of the proposed IAB guidelines.

Before I pose a few questions, I want to be clear that I am in agreement that certain standards need to be in place for our industry. However, it is the manner in which guidelines are established, imposed, and enforced that I have some concerns about.

Based on the feedback of some very well-respected analytics-industry experts, it is clear that there is much, much, much ambiguity in the proposed guidelines and it behooves the IAB to continue efforts in improving the guidelines before rushing to impose them.

Additionally, the IAB should consider the costs to companies which will or may be compelled to meet these guidelines. With the current state of the economy, can these companies afford an IAB “tax” (I hesitate to use the word “invest” as it implies a ROI)?

Finally, how effective and flexible will some of the proposed IAB standards be when policies proposed by consumer advocacy groups & quasi-governmental agencies dealing with online privacy continually oppose and contradict such measures?

eric

Peter and Josh: It’s increasingly clear that there is some disagreement internal to the IAB process. I was hoping George would email or call me but I have not heard anything from him. Suffice to say these recommendations are confusing and so I have to wonder about their adoption in the broader market. Time will tell.

Joe: Thanks for your feedback! See my comment above to Peter and Josh … confusion indeed. Hopefully the IAB will address the language in the document as part of “round 2″ and we’ll see this issue cleared up.

Kara

Eric – any updates to this now that IAB has issued the final guidelines? From what I understand the WAA was working with the IAB, beginning last April, to address some of the issues you highlighted. But from what I can tell, the final version is identical to the draft released for review.

Unique Visitors ONLY Come in One Size … | Web Analytics Demystified

[...] got back to my repeated requests for information, despite two members of the IAB working group (Josh Chasin from comScore and Pete Black from BPA Worldwide) openly disagreeing in their interpretation of the [...]

Unique Visitors ONLY Come in One Size … | KlickIntelligence

[...] in January I published a note about the proposed IAB Audience Reach Measurement Guidelines that generated a fair amount of interest. At the time I applauded the IAB for providing guidance [...]

De ene unieke bezoeker is de andere niet | Webanalisten.nl

[...] Thoughts on the proposed IAB Guidelines http://blog.webanalyticsdemystified.com/weblog/2009/01/thoughts-on-the-proposed-iab-guidelines.html [...]

Unique Visitors ONLY Come in One Size « ericpeterson

[...] in January I published a note about the proposed IAB Audience Reach Measurement Guidelines that generated a fair amount of interest. At the time I applauded the IAB for providing guidance [...]

Add a Comment