comScore study sheds new light on risks to cookie-based measurement

Awhile back the folks at comScore called me and asked if I would be surprised to learn that cookies were being deleted at a pretty high rate. Of course I said, “No, because I reported as much in 2005.” Through the course of the conversation, however, it became clear that comScore had the ability to shed new light on our understanding of cookie-based measurement; specifically they had the ability to measure the rate of deletion associated with first-party cookies.

comScore published the results of that study today.

I will fight the temptation to smugly say, “Ah ha! I told you so …” since the comScore data shows that I was both right and wrong when I first wrote about cookie deletion when I was with JupiterResearch. I was right in my assessment that this is happening far more frequently than those of us in the web analytics field particularly want to believe. But I was wrong in my assumption that cookie deletion was largely limited to third-party cookies.

The comScore data reports that over 30 percent of their panel of 400,000 home user computers deleted both first- and third-party cookies. Now, when I talked to Andrew Lipsman and Gian Fulgoni from comScore I repeatedly encouraged them to check and double-check these findings since especially their number for first-party cookies is much, much higher than I think any of us expected to see.

That said, I have no reason to believe that comScore would make this claim frivolously (okay, except for the fact that they provide a competing methodology to cookies) … I have asked comScore for a deeper briefing on their research but nothing has been scheduled as of this posting. Perhaps on my urging comScore took their research a step further and surveyed a subset of their panel asking about their stated behavior towards cookies. In the press release, Dr. Magrid Abraham addresses this in the context of the conventional wisdom that assigns greater risk to third- than first-party cookies:

“There is a common perception that third-party cookie deletion rates should be significantly higher than first-party cookie deletion rates,” continued Dr. Abraham. “Because many PC users reset or delete their cookies using security protection programs, conventional wisdom dictates that people are more likely to selectively expunge third-party cookies – which are generally deemed more invasive – while maintaining their first-party cookies. But these findings suggest that selective cookie management is not prevalent, a fact that comScore confirmed via a survey, with only 4 percent of Internet users indicating that they delete third-party but not first-party cookies.”

Yikes. When you look at the tables in the comScore study you can see where the problem is coming from: serial cookie deleters, the 7% of site visitors (measured via the comScore panel) that are repeatedly removing their cookies and thusly will appear as a new site visitor with every visit. I addressed the idea of serial deleters in my final JupiterResearch report on “The Crumbling Cookie” and, at the time speculated that some of the more nefarious activities available through the Internet were to blame.

Still, I never would have put the number as high as 7 percent.

It’s interesting to me that cookies are back in the news. It will be more interesting to see how all of this is digested in the coming days, weeks, and months. I wonder if Seth Godin will comment on the comScore study? I mean, I’m not sure that the “echo chamber” argument applies to comScore’s panel of 400,000 measured, identified individuals.

This seems to be a topic ripe for commentary and conversation. What do you think? Is comScore crazy? Is this report flawed? Or are we just fooling ourselves when we believe that “unique visitor” counts are an accurate representation of the number of real human beings coming to our web sites over long periods of time?

Posted Monday, April 16th, 2007 | 19 responses | Add a Comment | Share, Save or Email

Michael Leuchtenburg

How would someone only delete 3rd party cookies? I can’t tell a 3rd party cookie from a 1st party cookie by looking at it – neither can any program, except perhaps my web browser. Firefox used to have an option to deny 3rd party cookies, but it never worked properly, and it seems to have been removed from the preferences dialog now.

So most people who delete cookies are going to either delete them all or selectively delete the ones that aren’t from sites they care about. Since the ones most users care about are the sites where they have an account, tracking their usage is not that difficult anyway – they log in, you associate their session with the appropriate user.

I wonder what percentage of users delete their cookies specifically to be disruptive to people collecting statistics. That’s my reason.

eric

Some anti-spyware applications keep lists of “blacklisted” domains that are essentially third-party tracking domains. Domains like “ehg-hitbox.com”, “2o7.net”, “statse.webtrendslive.com”, etc. that have been (perhaps wrongly) identified as being some type of threat to a computer.

Your question about motivation is interesting. I wonder how we could get to the answer to that question …

Thanks for the comments, Michael.

Ian Houston

I don’t think the rise in the percentage of 1st party deletion to meet 3rd party deletion is really that surprising when you consider how the clients have progressed in this area over the past couple years.

Take Firefox2’s “Clear Private Data” menu and IE7’s “Delete Browsing History” which make deleting all your cookies(and anything else that might be used to see where you have gone) a 2 click process. It takes work (and that is a four letter word) to keep some cookies. The fastest easiest way to get rid of the “bad” cookies is to get rid of them all and I don’t think there is a lot of incentive for most consumers to do anything more complicated than that.

In the end I think it was pretty clear this was coming. I’m actually thankful that the overall percentage of users hasn’t really increased too much.

I think this new report is a good opportunity to talk more about how to deal with the issue rather than debate if this Nth study is right in claiming it exists. To that end I have posted some papers on my site that talk about some technical strategies I have worked on over the past couple years but they are certainly not without their faults.

Beyond the technical approach there is also cause to look at how numbers are reported. In the systems I work with we don’t put out the raw cookies based number for any length of time greater than 7 days. For monthly numbers we employ a more conservative approach through looking at the amount of activity per cookie ID as a filter of what to include in the longer term numbers of what we consider to be true “users” of the sites. Interestingly, that approach generally has led our monthly reported totals to be approximately 40% lower than what the raw numbers would be.

-Ian

Jacques Warren

OK, that’s it! I’m recommending that we now all put a button on web sites that says “Yep, I’ve been here in the past”, whith a universal look and found at the same place on ALL sites. With all the marketing genius out there we’ll figure out an incentive for people to click…

OK, sorry about the daydreaming. I think Michael has put his finger on something interesting. We WILL have to address the visitor counting issue as a professional community, especially with the coming integration with enterprise customer data systems. I mean, those systems will say “Whaaat?? a customer is two visitors who are sometimes a customer sometimes a browser and have five twins??”. You can imagine the fun…

Is the solution in looking at shorter periods (daily, weekly) and leave the deep and true relationship analysis to other systems? Systems that will need customers, or anybody interested in entertaining a relationship with the site, to say “Hello”, people who care about us as Michael says? Maybe, and why not? Can we imagine an experience that would treat anonymous visitors and known customers/prospects differently? Well, it’s already out there: personalization is based on giving a better experience to recognized visitors.

Personally, I don’t delete my cookies for two reasons (OK, maybe three if you count the fact that I’m not a technology obsessed man): to help my fellow analysts, and to be greeted when I go to Amazon Canada which take a large chunk of my income. I mean, I am a sucker: I hate to sign in on my favorite sites.

Seth Godin says people are not stupid, just very busy. I sure hope they’re at least suckers like me…

Linda

People are becoming more Internet savvy every day. And I don’t’ think it’s because they are disgruntled toward statistics; they just are more attune to privacy issues – right, wrong, or otherwise.

The Visitor Internet Intelligence effect needs to be taken into consideration when determining our output accuracy.

Linda

Enrique Gonzales

Eric,

Do you question comScore’s study at all? The results do seem to explain the perennial difference between internal and panel service numbers very nicely. By the way, comScore is about to go public. Does that factor in at all?

If you don’t question the study, then I think comScore has stated the problem pretty clearly.

Then, what’s the solution? Do we now account for the subset of the population who delete cookies frequently? Do we give up on cookies altogether. Do we rely on expensive panel-based services? Do we do sample-based studies of our site to come up with a unique visitor number? Does everyone go to full-registration, allowing advertisers to read our registration-based 1st party cookies?

Enrique Gonzales
NPR.org

eric

Ian: Agreed. I am, of course, a big fan of your CUID approach to cookie replacement but I wonder if that is practical given the default settings that browsers give to remove “stuff” … when I hit CTRL-SHIFT-DELETE in Firefox I believe that my browser history is one of the defaults selected for clearing (which would limit the CUID approach)

Still, props to you for coming up with the approach!

Jacques: Can I get that button for my web site too? ;-) Seriously, I think the problem with cookie deletion isn’t really about short-term issues so much as our general tendency to want to talk about web sites in terms of the number of “people” that visit them. Especially for sites that function independent of some type of UUID or login event (think CNN or NPR or CBSNews) and have little chance for personalization, this news must be particularly unwelcome, even if we all kind of suspected it.

Linda: Agreed. What is the “Visitor Internet Intelligence” effect?

Thanks to all three of you for your comments!

David

Interesting discussion. I’d be interested in seeing a list of the sites that are most popular with these serial cookie deleters. I suspect that deletion rates might vary _very_ widely from site to site, which means that an average rate (either for serial cookie deleters or for all internet users) may not be particularly useful in this context.

eric

David: You’re absolutely correct I suspect, that the serial deleter effect will vary site-to-site. There is a relevant post in the Yahoo! group that addresses the issue you might want to check out:

http://tech.groups.yahoo.com/group/webanalytics/message/10420

Thanks for your comments!

Ted McDonald

Not sure about comScore, but Nielsen/NetRatings provides an interesting “sessions per person” stat. They don’t, however, provide the actual number of sessions (gotta do some quickie math for that) and I suspect comScore doesn’t either since there’s no mention of sessions in their press release. So why is that? My guess is that it’s because number of sessions is not reliant on cookies, so comScore/Nielsen won’t have that crutch to lean on when analysts note that there’s still a huge discrepancy in sessions data.

For instance, our in-house UV counts via Omniture are typically about 3 times higher than what Nielsen reports. I agree that our numbers are somewhat inflated because of the cookie issue, but not to the extent estimated by the panel-based measurement companies. Our Visits per UV are only 1.15 while Nielsen reports a 1.4 – using that to calculate sessions data from Nielsen (and thus leaving cookies out of the equation) we find that our in-house sessions numbers are still at least TWICE what Nielsen reports.

Am I missing something? Does anyone else notice similar sessions discrepancies between their in-house data and the panel-based?

Brownie

I’m highly suspicious of this study. Look at the details here:

http://www.comscore.com/press/release.asp?press=1389

Now, I can buy that there are millions of users running local firewalls and anti-spyware apps that purge third-party cookies every week, or whatever, and this could easily produce the figures we see in the comScore report for third party cookies. But first party, too? Does LavaSoft’s Adaware purge first party cookies by default? I didn’t think any of these products did that. Which would mean users would have to manually purge all their cookies to flush first party cookies at the rate comScore are reporting.

In a nutshell, the conventional wisdom has been that most cookie deletion is a result of anti-spyware products cleaning pcs, and not manual intervention by users, even in the new, internet-savvy, privacy-aware environment in which we all operate. So why no delta between first and third party cookie purge rates? Does not compute.

I’ll wait to see these study findings corroborated by a company that doesn’t deal exclusively in panel-based measurement before I swallow it. I’m not casting aspersions, but when I read Abraham saying:

“But these findings suggest that selective cookie management is not prevalent, a fact that comScore confirmed via a survey, with only 4 percent of Internet users indicating that they delete third-party but not first-party cookies.”

…I ask myself how many users know that they are just deleting third and not first party cookies when they run their anti-spyware apps in default mode? Maybe very few, yet that is precisely what they will be doing (in most cases, anyway).

If comScore don’t come up with the missing detail, I think I’m entitled to remain sceptical, and I think everyone else should, too.

DB.

Linda Stacy

The Visitor Intelligence Effect is something that I use when determining how accurate my metric deliverables are. My thought process is that I need to first determine how Internet savvy my visitors are, then based on this subtract a percentage based on the probability that they are deleting their cookies (in this particular ‘cookie’ circumstance). The visitor could do this in the form of using Spyware, and/or manually configuring their Internet Options Settings — manual or automatic.

Please note that I establish a baseline for the campaign so I only have to go thru this particular scenario a couple of times before I can get a reasonable accurate percentage (albeit this is still an estimate). Example: if I have a visitor base that is high on the VIE, then I determine the percentage that are returning visitors based on the daily metrics. Yes, this takes crawling thru the analytics output for awhile, and the crawling depth depends on how much you are scrutinized by management to prove your estimates. I’ve found that doing this once or twice is enough to give a good estimate, and also to show that you know what you’re talking about if someone asks for the logic behind the percentage. That certainly goes a long way.

In B2B analytics, an area that we need to look in to is the VIE of the corporation is as well. Many businesses configure their employee’s towers, and do not enable changes. If the company blocks this from their end then it does not matter how savvy their employees are in most cases.

Linda

eric

Ted: An excellent question, one I alluded to in my post on the “death” of the page view (http://blog.webanalyticsdemystified.com/weblog/2007/02/worried-about-page-views-dying-dont-be.html) I’m still waiting for more comparative data on sessions (=visits) looking at panel vs. cookie-based data.

Anyone reading Ted’s comment willing to share what they’re seeing?

David: I wrote to comScore last night and asked a series of questions, one of which was about the survey methodology they used. I am also slightly suspicious of the statement you allude to but it’s not unreasonable to believe that they did conduct reasonable and well-crafted research to ask the question. Hopefully I’ll hear back from them before long.

Incidentally, the non-panel research you’re looking for that more-or-less mirrors what comScore is reporting was published by JupiterResearch in 2005. I wrote a report on the “Decline of Cookie-Based Measurement” in which 39% of a fairly large U.S.-based sample told us they deleted cookies on at least a monthly basis. We didn’t ask about first- vs. third- because it is pretty hard to get to that level of detail.

Linda: An excellent explanation! I would love to hear more about how you determine how savvy your visitors are since that level of detail is what confounds most sites. I can imagine the process for B2B but it seems more difficult for B2C, marketing, etc.

Thanks to all of you for your comments! To everyone else, regarding Ted’s comment, I’d love to hear what you have to say if you’ve compared session counts between cookie-based and panel-based systems.

Brownie

Oh, I don’t dispute the incidence of cookie deletion, but I am sceptical that there is no delta between 1st and 3rd party cookie deletion rates given:

a) most cookie deletion is powered by security/anti-spyware apps, and
b) most of the apps quarantine and purge 3rd party cookies *only* in default mode.

Going back to Abraham’s

only 4 percent of Internet users indicating that they delete third-party but not first-party cookies.

I would strongly suspect this 4pc is made up of users manually deleting 3rd party cookies only using whichever browser option, and those users running anti-spyware apps that do exactly the same thing automatically are not represented.

Yes, I realise this is a guess :-)

Regards,

DB.

Cookies o el futuro del usuario único » WebAnalytics.es

[...] La noticia ha tenido especial eco entre los bloggers sobre web analytics, evidentemente. Personalmente, estoy de acuerdo con Eric T. Peterson cuando dice que este dato no significa necesariamente que haya que medir con técnicas diferentes, sino que quizás la medida de “cuántas personas” nos visitan pueda dejar de tener sentido en un futuro cercano. [...]

wanalytics » Blog Archive » Emetrics Summit 2007 Düsseldorf, Teil I

[...] Was wir weiter gelernt haben: Lycos und Google verstehen sich auf analytischer Ebene prächtig und Erstere können Erfolge in der Optimierung von Nutzersegmenten in der Online Community vorweisen, durchaus interessant. Ein lohnenswerter Blick auf die Cookie Debatte (siehe auch comscore und Eric Petersson ) durch Frank Reese und die Grundlagen des neuen Telemediengesetzes im Bezug auf den Datenschutz im Web Analytics Umfeld durch Fred Türling im Doppelpack bereicherte die Bandbreite der Veranstaltung. Die WestLB beleuchtete in einem anderen Track die Einflüsse und Auswirkungen crossmedialer Werbekampagnen und deren Meßbarkeit mit Web Analytics Methoden und Tools. [...]

Web Analytics Demystified » Blog Archive » comScore answers a few of my questions about their recent report

[...] As I mentioned a few times in the Yahoo! group, I have been talking to the folks at comScore about their recent report on cookie deletion. I got an email back from Andrew Lipsman with some more information and partial answers to questions of mine and a few passed to me by other cookie-savvy folk. [...]

Sébastien Brodeur

I don’t believe it.

We use both methodology (1st party cookie and panelist) and we got more unique visitor (25% more) from our panelist that unique visitor calculate from 1st party cookies.

One can argue that maybe we didn’t implement well our cookie based solution, but I doubt. I believe those number from comScore are exaggerated.

This is why you need multiple source to validate your number. Still, no solution are perfect, but it’s all about the trend baby :-)

————————–

Je n’y crois pas.

Nous utilisons les deux méthodologies (1st party cookie et des panelistes) et nous avons plus de visiteurs unique (25% plus) rapporter par notre solution par paneliste que de visiteurs uniques identifiés par notre solution utilisant des 1st party cookie.

On pourrais supossé que notre implémentation de notre solution par 1st party cookie n’est pas bien faite, mais j’en doute. Je pense que les chiffres de comScore sont exagérés.

Voilà pourquoi il est important d’avoir plusieurs sources pour être en mesure de valider nos chiffres. Malgré tout, aucune solution n’est parfaite, mais tout est dans les tendances bébé (c’est drôle ça sonne pas aussi bien en français :-)

Blog-conversion.com : optimisation du taux de conversion sur internet : e-commerce, services, editeurs ...

[...] ComScore sort une étude sur l’impact de la suppression des cookies par leurs utilisateurs. Cela pourrait entrainer selon l’étude une surestimation de 2,5 fois le trafic réel !!! Ce chiffre paraît absolument énorme. Une petite analyse chez Eric Peterson, en anglais, ou chez Emmanuel Parody, en français. Nous referons clairement un article sur ce sujet ! [...]

Add a Comment