On NetRatings and time spent on site
In all of the fuss about NetRatings dropping page views as a metric used to calculate site popularity is the fact that the company actually did a pretty smart thing: they took my advice from February 15th of this year and rolled in a very valuable and useful “sessions” metric. Well, maybe it wasn’t my advice they took, but I think it was a great idea either way to drop page views since they’ve become increasingly inconsistent to instead focus on the one metric that is consistently applied and well defined, sessions.
Unfortunately NetRatings chose to focus their announcement on “total minutes” saying that time was a better measure of engagement. Personally I’ve never been a very big fan of the time spent metrics — I guess I’ve just looked too long and too hard at all the problems associated with how time is collected and recorded in the web analytics realm.
There is a really engaged thread at the Web Analytics Forum at Yahoo! Groups on this subject that is definitely worth a read if you’re interested.
And I’ll admit, I don’t have all the details associated with how panel-based services like Neilsen and comScore track time spent. If they’re actively tracking the user and only counting time when the browser window is active and the mouse is moving, well that would be a good use of the panel. My suspicion is that, like in web analytics, they’re simply recording the delta between the first and last request for a page in the domain — a strategy that suffers from a litany of well-described problems.
The two I see as most problematic are:
- Single page visits are either difficult to count or not counted in time spent calculations
- The amount of time a web page is open is likely only poorly correlated to their actual engagement with the page
Some have already noted that the fact that very popular sites like Google will do poorly in time spent on site because one of the dominant use cases involves only a single page (I search and I go.) Conversely, depending on how time spent on site is calculated, the search engines may have inordinately long times spent based on a search leading to a long browse time on a discovered site, leading back to the search results (same session, clock is presumably still ticking), leading to the next discovered site, etc.
I for one use iGoogle in exactly this way: I load the page frequently throughout the day and do nothing more than look at a single page view. In fact, unless Nielsen is either tracking the AJAX-interaction with the iGoogle interface, or counting single page view sessions, it is likely that my interaction with iGoogle is not counted at all. But let me assure you, I am quite engaged with the content in my Google portal (something that would be well evidenced by the total session count I generate at the site each day.)
As I looked back through the plethora of comments that my original post on using sessions to compare sites I noticed that I had made this statement in response to a comment from Jacques Warren:
- If you want to compare two or more web sites, use sessions because of the reasons I outlined in my original post.
- If you’re interested in the number of people coming to one web site (presumably yours), use de-duplicated unique visitors but be mindful of cookie deletion.
- If you’re interested in the activity of people on your web site, and if you have a “Web 1.0″ web site, use page views but be mindful of issues like code coverage, proxies, robots, etc.
- If you’re interested in the activity of people on your web site, and if you have a “Web 2.0″ web site built around RIAs, etc., use some form of event model.
I’ll stand by this. Until I know more about how N/NR and comScore calculate their time spent on site metrics it’s hard to believe their numbers to be any more useful or accurate than those provided by direct measurement systems. That said, I’d welcome a briefing on the subject from either company if they’re reading this and are interested in having me pick apart their methodology spending some time with me.
If companies really need to use time spent on site, they should consider using better key performance indicators for time such as Percent Low/Medium/High Time Spent on Site categories (something I talk about at length in The Big Book of Key Performance Indicators.) That way N/NR could report on the percent of all tracked sessions that were “30 seconds or less”, “31 seconds to 5 minutes”, and “More than 5 minutes” (as an example) which would give us a more powerful view into the relationship between visitors and the time they spend on site.
At the end of the day I like that N/NR has provided a consistent and easily compared metric to their customers in “total sessions” which is what I will inevitably focus on as a measure of site popularity. Having devoted quite a bit of time to describing what I believe to be a solid measure of visitor engagement, it’s difficult for me to think about “time spent on site” (or even “total sessions”) as a good proxy. Time spent, recency, depth of session, session number, etc. are all components of engagement, not direct measures.
What do you think? Is Nielsen right and I’m crazy? Have you been looking closely at your time spent on site metric for years and are delighted that the rest of the world has finally caught up? Or are you like me and spend far too much time browsing from site to site, flipping from task to task, and thusly confounding clocks and counters on every site you visit?
I welcome your comments.
Bill Bruno added the following ...
Eric, after reading your post, we are in complete agreement. I have a difficult time stomaching Time Spent on Site as being a primary metric in any situation. It can be largely inaccurate, and could be due to a variety of factors, such as:
Confusion
Improper Navigation
Slow Loading Times
Etc.
Instead of being a direct correlation to being an active visitor. In addition, it is difficult to dissect all of these metrics on their own from a panel perspective. When we are conducting an analysis for a client, all of these metrics play into the story we are trying to tell to drive optimization. From a panel perspective, you are losing some of that to generalities. For example, let’s say you are simply looking at Page Views. Well, your page views could be inflated for many of the same reasons that I listed for Time spent on Site in addition to pure site size in terms of content. It’s simply a different view of the same issue.
When it comes down to it, I think a site can only truly be judged by Actions, but then you could be comparing apples and oranges in a Panel-based environment. Supporting total sessions is definitely a huge step up though, as that gets rid of some of the issues I’ve listed above and from a panel perspective gives you an idea of Total Site Volumes.
Kim Weller added the following ...
I think looking at time spent metrics as a measure of site engagement is a really bad idea. Even if the data wasn’t questionable, any IA or Usability person will tell you that the users’ goal is to move efficiently through a site. So an increase in time spent on site or on page for that matter may actually be an indication of a problem, not engagement.
eric added the following ...
Bill: Agreed on time spent on site and agreed on the need for “actions” but the difficulty in identifying them with services like NNR, comScore, etc. It’s too bad that the panels don’t take the time to identify critical actions for, say, the top 100 sites, and track those as well. So for GYM it would be searches, at CNN and AOL it would be ad clicks, and at BBY and Walmart it would be purchases.
Would that not be cool data to have?!
Kim: Yep! When I was talking to online retailers about “how to measure engagement” on their sites many commented that they already had a pretty good measure — purchases and revenue! I think the problem you describe and the problem I described in the post re: Google alone makes time spent far less useful of a metric than I think some people believe (or want to believe.)
Thanks for commenting!
Jacques Warren added the following ...
I agree with Kim. Let us not forget that 10 years of reading Jakob Nielsen and Jared Spool has taught us a lot about making sure visitors quickly found what they were looking for. Shorter visits became thus a byproduct of being efficient at usability. Does this mean those satisfied visitors are less engaged?
One is quite surprised that a company such as Nielsen would confuse such elementary interactive marketing stuff.
Eric: I followed with great interst all the work you did on engagement when you were at VS. You were on something very valuable. Are you going to still explore it?
Jim Novo added the following ...
Folks, let’s not forget what this N/NR - comScore thing is about:
1. The display advertising model
2. A ranking of web sites for that model
It’s not hard to imagine the longer you were on a site, the more likely it would be the display advertising would be seen and an impression registered in the brain.
This really has nothing to do with how you analyze a web site for navigation / actions / conversions and so forth.
It’s about being engaged with *display advertising*, not being engaged with your web site or your business. Buyers of that kind of media would like to know duration, because it makes sense to them and is used with other electronic media.
More if interested:
http://blog.jimnovo.com/2007/07/16/nielsen-netratings-comscore/
Harald Conradi added the following ...
Like a hot knife thru butter, Jim.
The last thing I want to see from a conversion standpoint is a visitor exiting my site by clicking on a display ad.
Can we really compare the value of websites via a standardized set of stats, when the conversion goals are specific to each site? At the end of the day, I want a visitor to buy- whether first visit or a gradual multi-step engagement/conversion process.
eric added the following ...
Jim: I agree with your assessment but wonder about your statement “the longer you were on a site, the more likely it would be the display advertising would be seen and an impression registered in the brain” being validation for using a metric that is known to be problematic. Do you know anything about how N/NR is calculating time spent? For some reason nobody from Nielsen has responded to my offer to hear more about their strategy.
Harold: Good point, but don’t online retailers take some pride in being the most popular/most visited? Many of the retailers I have worked with in the past have been almost preoccupied with benchmarking and this type of comparison — sometimes to the point of distraction.
I’m sure this debate will continue but I’m looking forward to Shop.ORG in Las Vegas in September to discuss this more with such a learned and engaged audience.
Thanks to you both for your comments and Jim I’ll see you next week in Los Angeles!
Dorian Benkoil added the following ...
Eric (et al),
In a recent discussion I had with Nielsen Net Ratings
(see http://www.mediavillage.com/jmr/2007/06/11/jmr-06-11-07/ for my writeup)
Nielsen told me they’re working on a technology to measure engagement as time someone is actually on a page and engaged with it. So, for example, when someone has something in a browser tab that is not in the forefront it won’t get counted toward time on site. They also acknowledged it’s a work in progress.
Whatever the system, publishers/content providers will try to game it, and the measurers will try to keep it accurate. But I do think Nielsen is taking a bold step, even if what they’ve got right now isn’t nearly perfect
eric added the following ...
Jacques: Man, I totally spaced your comment, I am so sorry! Yes, I am currently working on a white paper on measuring visitor engagement, something more easily consumed than 50K words in my blog ;-)
For those of you wondering what Mr. Warren is referring to, start reading at the bottom of this page:
http://blog.webanalyticsdemystified.com/weblog/category/engagement/page/3/
That said, my measure of visitor engagement is far from appropriate for N/NR or comScore — they simply don’t have the technology available (IMHO) to make the calculation. And, many believe that every site’s measure of engagement is different (see Bill Gassman’s comments in the posts above.)
Dorian: If N/NR has that technology it would be great and it would honestly solve a bunch of questions. I don’t know if there is anything in the DOM that allows for direct measure of physical engagement with the page (e.g., tab is open, tab is in focus, user is not interacting with other applications on the desktop, etc.) Do you?
It’s funny you should mention “gaming” N/NR’s new metrics. I had a nice conversation with an analyst for a very large media company today who basically said they were going to re-think their web analytics strategy to drive increased time on site. I jokingly said “well, all you have to do is add huge images of Paris Hilton to every page and everything well render more slowly, thusly lengthening the overall session …”
He didn’t laugh. Hopefully that was because Paris Hilton isn’t funny, not because he’s considering my idea.
Thanks to you both for your thoughts and keep ‘em coming!
Jacques Warren added the following ...
Hi Eric,
No harm done!
Great! I am really looking forward to reading it in a “better” formatted version. Sure, that is out-of-scope stuff for N/NR-comScore. Is it even within reach of most WA applications? Doubt it frankly, but definitely something that should constitute a product development roadmap. Anyway, we’ll have the opportunity to discuss it again when the WP is out.
The Software Abstractions Blog added the following ...
Top 8 Reasons to Implement Tracking and Measurement for your Web Site…
With the easy availability of full-featured, free online services like Google Analytics, an increasing number of companies are implementing Web Analytics tools for their web sites. Many of these implementations still focus on older metrics like page vi…
shally added the following ...
If website opened on the desktop all day, does this count as time on Google analytics? Any suggestion?
Danica added the following ...
@shally: My impression from looking around at information about Google Analytics and analytics programs in general is that if a website is opened and just sits there, the session “times out” and it’s considered a “bounce.”
I don’t know for sure how GA handles these, but I notice that I have a WHOLE lot of visitors who apparently - even when coming to my site from, for example, their RSS feeds, which I assume is on purpose and out of interest - spend “0.00″ seconds on my site. Which I find entirely unnerving.
I suspect that that is just their way of telling me the person “bounced,” rather than appearing to interact with the site. Maybe I’ll try switching to have it show only part of one entry at a time so people can’t just scroll down….
I’m happy that they have adopted the “0-30/31-59/60+” seconds method of counting time spent on a site, but I wish they’d make information about how they track this stuff a little more transparent!
John Sloan added the following ...
If I wanted to become an expert at web analytics what would you say is the best way to get started? Is there a holy grail of Web Analytics I need to read other than this blog?
John
share.websitemagazine.com added the following ...
On NetRatings and time spent on site…
In all of the fuss about NetRatings dropping page views as a metric used to calculate site popularity is the fact that the company rolled in a very valuable and useful “sessions” metric…