Archive for 'Engagement'

In all of the fuss about NetRatings dropping page views as a metric used to calculate site popularity is the fact that the company actually did a pretty smart thing: they took my advice from February 15th of this year and rolled in a very valuable and useful “sessions” metric. Well, maybe it wasn’t my advice they took, but I think it was a great idea either way to drop page views since they’ve become increasingly inconsistent to instead focus on the one metric that is consistently applied and well defined, sessions.

Unfortunately NetRatings chose to focus their announcement on “total minutes” saying that time was a better measure of engagement. Personally I’ve never been a very big fan of the time spent metrics — I guess I’ve just looked too long and too hard at all the problems associated with how time is collected and recorded in the web analytics realm.

There is a really engaged thread at the Web Analytics Forum at Yahoo! Groups on this subject that is definitely worth a read if you’re interested.

And I’ll admit, I don’t have all the details associated with how panel-based services like Neilsen and comScore track time spent. If they’re actively tracking the user and only counting time when the browser window is active and the mouse is moving, well that would be a good use of the panel. My suspicion is that, like in web analytics, they’re simply recording the delta between the first and last request for a page in the domain — a strategy that suffers from a litany of well-described problems.

The two I see as most problematic are:

• Single page visits are either difficult to count or not counted in time spent calculations
• The amount of time a web page is open is likely only poorly correlated to their actual engagement with the page

Some have already noted that the fact that very popular sites like Google will do poorly in time spent on site because one of the dominant use cases involves only a single page (I search and I go.) Conversely, depending on how time spent on site is calculated, the search engines may have inordinately long times spent based on a search leading to a long browse time on a discovered site, leading back to the search results (same session, clock is presumably still ticking), leading to the next discovered site, etc.

I for one use iGoogle in exactly this way: I load the page frequently throughout the day and do nothing more than look at a single page view. In fact, unless Nielsen is either tracking the AJAX-interaction with the iGoogle interface, or counting single page view sessions, it is likely that my interaction with iGoogle is not counted at all. But let me assure you, I am quite engaged with the content in my Google portal (something that would be well evidenced by the total session count I generate at the site each day.)

As I looked back through the plethora of comments that my original post on using sessions to compare sites I noticed that I had made this statement in response to a comment from Jacques Warren:

• If you want to compare two or more web sites, use sessions because of the reasons I outlined in my original post.
• If you’re interested in the number of people coming to one web site (presumably yours), use de-duplicated unique visitors but be mindful of cookie deletion.
• If you’re interested in the activity of people on your web site, and if you have a “Web 1.0″ web site, use page views but be mindful of issues like code coverage, proxies, robots, etc.
• If you’re interested in the activity of people on your web site, and if you have a “Web 2.0″ web site built around RIAs, etc., use some form of event model.

I’ll stand by this. Until I know more about how N/NR and comScore calculate their time spent on site metrics it’s hard to believe their numbers to be any more useful or accurate than those provided by direct measurement systems. That said, I’d welcome a briefing on the subject from either company if they’re reading this and are interested in having me pick apart their methodology spending some time with me.

If companies really need to use time spent on site, they should consider using better key performance indicators for time such as Percent Low/Medium/High Time Spent on Site categories (something I talk about at length in The Big Book of Key Performance Indicators.)  That way N/NR could report on the percent of all tracked sessions that were “30 seconds or less”, “31 seconds to 5 minutes”, and “More than 5 minutes” (as an example) which would give us a more powerful view into the relationship between visitors and the time they spend on site.
At the end of the day I like that N/NR has provided a consistent and easily compared metric to their customers in “total sessions” which is what I will inevitably focus on as a measure of site popularity. Having devoted quite a bit of time to describing what I believe to be a solid measure of visitor engagement, it’s difficult for me to think about “time spent on site” (or even “total sessions”) as a good proxy. Time spent, recency, depth of session, session number, etc. are all components of engagement, not direct measures.

What do you think? Is Nielsen right and I’m crazy? Have you been looking closely at your time spent on site metric for years and are delighted that the rest of the world has finally caught up? Or are you like me and spend far too much time browsing from site to site, flipping from task to task, and thusly confounding clocks and counters on every site you visit?

I welcome your comments.

About a month ago, just before I started Web Analytics Demystified, I had the pleasure of sitting down for an interview with Jeremiah Owyang of PodTech.net. Clint first introduced me to Jeremiah when I was talking about measuring visitor engagement and how social media might be best measured. Jeremiah is very much connected in the Bay Area and I though the interview went really well (but you can judge for yourself by watching the interview at Jeremiah’s web site.)

A number of folks have commented on the interview at Jeremiah’s site and the comments are well worth a read.

More recently I wrote a post on the 10/20/70 Rule for Achievable Web Analytics Success in which I outlined the importance of process to web analytics. A number of folks have since commented on the post but Rene Dechamps from OX2 was kind enough to post a video from the conversation that got me thinking about 10/20/70 (thanks Rene!)

Since Rene was about as tired as I was at 7:00 AM local time, and he’d been kind enough to bring me a coffee, I recommend ** not ** trying to watch the video and just listening instead.

What do you think?  Should I stick to writing and stay off the tele?  As always, I welcome your comments.

[ I'm really happy to have my first "guest post" from blogger Robbin Steif from Lunametrics. Robbin really liked my "gradual building of context" post from awhile back and she and I have been discussing a related metric that she thinks builds nicely on my visitor engagement metric. Without further ado, Robbin Steif ... ]

On the one hand, I thought that Eric’s recent post, The Gradual Building of Context, was just awesome. Although every site has to define visitor engagement for itself, every site is still capable of pulling together similar numbers (which is why I loved it.)

On the other hand, I disagreed with Eric’s final conclusion, “I need to reach out folks like Matt, Marshall, and Clint and see if there is some way I can get them to more passionately advocate for my books in their weblogs. Given that their visitors are more highly engaged than the “average visitor’, I have to believe their is an opportunity to sell more books.”

I took one look at the numbers in the last chart and thought, well, that doesn’t make sense. Sure, Matt’s visitors (or Clint’s or Marshall’s) are somewhat more engaged than the average visitor, but their “start the purchase cycle” numbers are pitiful. If Eric were to put effort into this, the place to put effort in is where both those metrics are strong.

Eric was good enough to send me the spreadsheet, and I pushed the numbers. (Well ok, technically he and I pushed the numbers at the same time over the phone …) On the phone, he called it “Robbin’s Metric” and I left it that way. It is the product of his Visitor Engagement and Percent Buy Path Sessions:

[ Ugh! Yes, I know that image is hard to read! I will correct ASAP!!! ]

By multiplying the two metrics and then ranking all the referring blogs by that metric, you see where Eric should put in extra effort. I agree witrh the first conclusion that Eric already came to in his blog, i.e. that he needs to work out some kind of deal with Anil. However, the two blogs where he should put time/effort would be Justin Cutroni’s and ROI Revolution’s. Interestingly, they are both Google Analytic blogs, so there is a decent chance that the reader is newer at analytics and probably could really benefit from Eric’s books. I didn’t highlight Steve Jackson’s blog, Xavier’s or Aurelie’s because they are already converting well (if there is such thing as converting well.)

Finaly thoughts: An engaged visitor to a site that is also content rich, like Eric’s, doesn’t necessarily make a good customer. In fact, Clint did a survey on his blog and saw that many of his visitors already own many, if not all, of Eric’s books. When visitors go to the beginning of the checkout, we can actually see interest in the purchase, as well as interest in the content — and as direct marketers know, you should always pursue the customer who already has a propensity to buy.

[ Thanks to Robbin for taking the time to take visitor engagement to the next level! What do you think? Is Robbin on the right track? Did I miss the mark? As always, your comments are greatly appreciated! ]

(If you need to catch up on where we are to date, have a look at my last post in this series on measuring visitor engagement.)

I had a nice conversation a few days ago with Jeremiah Owyang, Web Strategist at PodTech.net, on how I have been measuring engagement. Jeremiah has been thinking about how engagement is defined for some time and had a very fresh perspective on the subject which has somewhat expanded my thinking on the subject. Jeremiah, by virtue of being an “A-list” blogger (IMHO) gets great critical feedback from folks like Forrester’s Charlene Li (who says that my measurement is too explicit, oh well …) After we talked, I realized that I really needed to get the promised post on measuring “social engagement in a Web 2.0 world” out the door. So here it is.

One of the links that Jeremiah references is this one from Wiredset, published in November of last year. In their post, Wiredset gives a definition of engagement as “a consumer based measurement that regards interaction with an aspect of a brand or media property” and goes on to say that “Web 2.0 Engagement” could include activities (Jeremiah refers to these as “gestures”) like:

• Publishing
• Creating and Publishing to a Group
• Posting
• Subscribing
• Favoriting
• Adding Friends
• Bookmarking
• Emailing
• Distributing
• Streaming
• Networking
• Creating Mash-up Content

I absolutely agree with Wiredset, and they go on to say:

“When measuring engagement, the level of user interaction (i.e. 200 vs. 2,000,000 streams) is an obvious and important component. Yet engagement is complex in that it is not comprised solely by clicks, but also a range of involved user actions.“

If you’ve been reading along the entire time, you’ll note that my current definition of visitor engagement is derived exclusively from click-stream data and it tries to be as independent of content as possible. While this makes sense for a lot of reasons, the larger conversation (as Clint and Jeremiah wisely point out) is about how a visitor engagement metric can help us better understand the value of emerging Internet technologies.

While Web Analytics Demystified is not your typical Web 2.0 or social community site, I have enough of the activities listed above on my site to apply a social media filter to my measurement calculation and look at the effects. Again, if you’ve been reading along, I covered many of these in Part III of this series.

Here is the list of things that I am tracking vis-a-vis social media/Web 2.0 on my site:

Now, up until this point I have basically fought applying any weighting to the visitor engagement metric, mostly because I think it’s pretty difficult to rationalize any particular weighting over another and it will complicate what has already been described as “the mother of all KPIs”. That said, I am scoring these social activities into what I call an “interaction index” (ratio of sessions with one of the activities above vs. sessions without) and using the interaction index to weight the visitor engagement metric.

So instead of the existing definition of visitor engagement:

We have the new definition of “Social Engagement”:

Both metrics are the sum of component indices divided by seven, so you can hopefully see that the latter metric is weighted by any contribution made by the “Interaction Index”. For definitions of the component indices, please see Part IV in this series.

So what does this give us? Well, if you were interested in tracking individual users based on their level of visitor or social engagement, you would be able to drill-down along each Web 2.0 activity and perhaps learn something interesting:

There is Frank Faubert from Unica again, not much more socially engaged with my site than he is otherwise engaged. Remember that Frank initially complained about his only having a 21 percent engagement score, to which I responded that I had lost him in my data. Well, I found him, and based on the evolving calculation, Frank is over 31 percent engaged but little of his measured engagement is “social” in nature.

But what if I drill-down along each of my defined social activities, what can I learn?

First we can see my good friend Jeff Katz, formerly of WebTrends, who is a regular reader of my blog and whose social engagement score is much higher than his visitor engagement score. Jeff has repeatedly joined the community (Web 2.0 Measurement Working Group, Web Analytics Wednesday attendee) and has also hosted a WAW event here in Portland, OR.

Looking at direct engagement via email, we can see the great Aurelie Pols from OX2 Belgium who has also submitted comments to my blog.

I can also apply the visitor and social engagement scores to other relevant dimensions like referrers:

Here you can see that I’ve calculated the variance between visitor and social engagement and am color-coding that against my site referrers. O’Reilly’s XML.com, E-consultancy, and Jim Sterne’s Emetrics web site all are sending visitors who are well-engaged socially.

Finally, you can see the difference between visitor and social engagement applied to the various blog posts I am tracking for Clint Ivy, Ian Houston, Robbin Steif, and Avinash Kaushik. Clint’s open letter to Jeff Jarvis (a controversial piece if ever there was one) is driving a great deal of Web 2.0 engagement amongst Clint’s readers. Nice work, Clint!

Hopefully you get the picture here. By weighting the visitor engagement metric with these social media activities, I am able to easily identify individuals, referring sources, marketing campaigns, rich Internet applications, etc. that are actively interacting, both on my site (join community, engage directly, submit a comment, contribute content) and off (host an event, share a social bookmark).

Wiredset’s proposes a distilled definition of “Engagement = Interaction/Attention” which makes sense to me … you have attention by virtue of their coming to the site, but can you drive interaction? I would propose that the visitor and social engagement metrics I have described in this series of blog posts describes this equation practically applied.

As always, I welcome your comments and criticism.

Man, it seems like I just cannot get away from Gary Angel lately. He and I are engaged in some kind of crazy mutual-admiration society thing, which would normally worry me, but I know few people as into all this as Gary. He recently posted about “that darn engagement metric” where he expanded on some of my ideas and his response to my ideas and my response to his response to my ideas, etc. One thing Gary said really stuck out in my mind. Regarding my use of the visitor engagement metric to tell a story about the traffic that Marshall Sponder sends to my web site, Gary commented:

“This gradual building of a context around a measure is exactly how I think reporting actually works – and how analysis drives to actionable understanding.”

Exactly! The gradual building of context is what this is all about. The reason I’ve defined a visitor engagement metric is to provide another firm basis for the establishment of said context, another indicator on which we can draw to better understand a dimension or set of dimensions of data we collect.

Much has been written about the value of bloggers to business; it seems like you can’t open Business Week, Fortune, or even Newsweek anymore without having to read about the next big thing that bloggers and blogging are doing to change both business and society. But what can we know about the traffic bloggers send us? And what actions can we take based on that information? Let’s have a look …

Here I’ve rank-ordered the folks I list in my blogroll by percent of sessions they drive back to my web site. No great insights here based on my KPI “percent of sessions” but I suppose if I wanted to I could add Clint, Steve, Tim, and Eric Butler to my holiday card list.

Now I’ve added the session conversion KPI to my list of bloggers. Immediately I see two sets of actions I could possibly take: The first would be to send “much love” to Steve, Mike, Gary, Xavier, Manoj, Aurelie, and Tim for helping me pay for my children’s college education (all book proceeds go to my kids’s Fidelity 529 plans). The second would be to see what I could do to get the rest of the bloggers to say something like “Hey, go buy Peterson’s books, they’re great!”

Still, I should probably check to see first that these folks aren’t referring me traffic that later returns to the site and makes a purchase, right? I need to roll in a visitor-based conversion metric:

Ah ha, now I can see that I owe some serious thanks to Steven Jackson and the folks at the Blackbeak Blog! Better than one in ten people Steve has sent my way have made a book purchase, which is awesome. But it looks like I have a problem with bloggers like Anil Batra, Matt Jacobs, Robbin Steif and even Marshall Sponder. Zero percent visitor-based conversion to purchases on my site … must be some problem with how those folks are talking about me, right?

Oh, or maybe not, at least not in every case. I added a KPI for percent buy path sessions, basically the percentage of sessions in which a visitor at least starts down the book purchase path. Now I can see that Anil, Justin, and Robbin are all doing a pretty good job of getting people into the purchase consideration process, but for some reason those folks aren’t completing the purchase on my site. It’s not their fault, it’s my fault!

Damn.

But hey, maybe it’s still not my fault. Maybe even though the bloggers are sending me traffic that hits the buy path, maybe those folks aren’t really all that engaged with my site and content. Maybe the visitor’s they refer me are just looking at one page in the buy path and leaving, never to return.

Okay, or maybe not. Anil, Matt Jacobs, Marshall, and Clint are all sending me visitors that I consider to be “well engaged” with my site (my site-wide visitor engagement average is 30 percent.) Now I can see two clear action items:

1. I need to reach out to Anil and see if he and I can work out a deal to help further encourage his readers to completing the book purchase on my site. Anil, if you’re reading this, call me, we need to talk.
2. I need to reach out folks like Matt, Marshall, and Clint and see if there is some way I can get them to more passionately advocate for my books in their weblogs. Given that their visitors are more highly engaged than the “average visitor”, I have to believe their is an opportunity to sell more books.

But wait, I’m not done. In fact, I’ve only just begun to mine for the true opportunity here. But hopefully you can see, this gradual building of context is well-supported by each of the key performance indicators I keep in my arsenal, both simple metrics like “percent of sessions” and the more complex “visitor engagement”.

This post was a really long way of saying I agree with Gary about no one KPI driving a specific and easily understood action. All of our efforts are ultimately designed to help the online business better mine for opportunity and understand how that opportunity might potentially be leveraged. There are no easy answers, there are no silver bullets, there is no magic, nor mystery, nor puzzles …

There is only the gradual building of context …