Archive for 'Web 2.0'

I found myself thinking, “Are we really having this conversation?” today after reading Steve Rubel’s post today on “What will replace the allmighty page view?” where Mr. Rubel commented:

“The page view is on life support. It fails to capture all of the myriad of ways consumers engage in online activities without ever leaving a web page.“

Okaaaaaaaaaaaaay.

I suppose Steve is coming at this from a different perspective than anyone who works in the web analytics field, more-or-less looking at page views as a basis for comparing the relative value of one advertising opportunity to another. If that’s the case then yeah, page views are becoming increasingly limited in their utility.

But damn, as a web analytics professional, doesn’t all this talk about page views going the way of the Dodo bird just make your stomach feel all funny? Like, you know there are problems with the metric, but A) when compared to the other problems web site operators have vis-a-vis counting (cookie deletion, cookie blocking, poor implementations, caching, robots, lack of understanding, lack of interest) and B) when put in the context of the number of sites that still rely on good old fashioned HTML, don’t these proclamations seem a bit premature?

Is it just me? Maybe it’s just me …

Anyway, we can stop worrying about dying pages and dying page views now since the answer has been with us the whole time. It’s not unique visitors … too many problems with how unique visitors are counted, what with cookie deletion and some of the inaccuracies ascribed to panel-based services. It’s not time spent on site … the problems with this metric as the basis of comparison are many (connection speed, amount of content, quality of content, bathroom breaks, etc.)

It’s sessions.

Yep, sessions. Good old “start ‘em with the first page view and stop ‘em after 30 minutes of inactivity” sessions. And while they don’t necessarily solve the problem of how many impressions a site can serve (you need old fashioned web analytics for that), they provide a stable basis for comparison across sites:

• Sessions are defined by a widely-used and widely-understood standard, the 30 minute timeout between subsequent page views. Heck, in the web analytics industry, it’s pretty much the only standard we have …
• Sessions are counted once and only once when a visitor goes to a web site in a single web browser and are thusly not subject to inflation due to crappy web design or RIAs. No more complaints about MySpace!
• Sessions are time independent, except for the session timeout. You can click away all day and you’ll still only count one session, unless you walk away for 30 minutes and one second …
• Sessions mitigate out issues associated with error pages and the such, because again, the number of pages viewed is irrelevant after the visitor views the first page. Again, no more complaints about MySpace …
• Sessions are not affected by cookie deletion and are not always affected by cookie blocking. Whoopie! We can stop bugging out about cookie deletion …
• Sessions are not affected by users visiting sites from multiple web browsers, since regardless of location (home, work, etc.) the session is counted. Hurrah! No more massive over-counting of unique visitors during Fantasy Football season …
• Sessions can be counted even when the visitor is not on your web site, depending on what tracking technology you’re using and how it’s deployed. For example, a session can be counted when someone reads a post in their RSS reader …
• Sessions are easily tied back to relevant referring sources, such as advertising units, RSS feeds, search terms, etc. Yippie! Not only do we get more accurate counts, we know from where the sessions are originating …

Yep, good old fashioned sessions … who’da thunk it? You can call them “visits” if you’d like!

What’s better is that the reporting networks should just as easily be able to report on sessions as they do unique visitors. If they can report on “unique searches” and “time per person” and “page views” and all that, nothing should theoretically stop them from using “sessions” as the basis for reporting.

Clint Ivy pointed out to me that Hitwise uses sessions as the basis for their reporting platform, only they report however on percent market share and not the actual number of sessions which is almost certainly what advertisers would prefer to see. Neither of us were sure why they don’t give raw session counts, do any of you?

Just think of all the problems we can solve by using sessions to compare the popularity of web sites! No more complaints about newspaper sites reporting more unique visitors than live in the entire state. No more complaints about huge differences in reported numbers ascribed to cookie deletion. No more freaking out about inanimate objects dying …

What do you think? Am I crazy? Is it just me? As always, I welcome your comments.

(If you need to catch up on where we are to date, have a look at my last post in this series on measuring visitor engagement.)

I had a nice conversation a few days ago with Jeremiah Owyang, Web Strategist at PodTech.net, on how I have been measuring engagement. Jeremiah has been thinking about how engagement is defined for some time and had a very fresh perspective on the subject which has somewhat expanded my thinking on the subject. Jeremiah, by virtue of being an “A-list” blogger (IMHO) gets great critical feedback from folks like Forrester’s Charlene Li (who says that my measurement is too explicit, oh well …) After we talked, I realized that I really needed to get the promised post on measuring “social engagement in a Web 2.0 world” out the door. So here it is.

One of the links that Jeremiah references is this one from Wiredset, published in November of last year. In their post, Wiredset gives a definition of engagement as “a consumer based measurement that regards interaction with an aspect of a brand or media property” and goes on to say that “Web 2.0 Engagement” could include activities (Jeremiah refers to these as “gestures”) like:

• Publishing
• Creating and Publishing to a Group
• Posting
• Subscribing
• Favoriting
• Adding Friends
• Bookmarking
• Emailing
• Distributing
• Streaming
• Networking
• Creating Mash-up Content

I absolutely agree with Wiredset, and they go on to say:

“When measuring engagement, the level of user interaction (i.e. 200 vs. 2,000,000 streams) is an obvious and important component. Yet engagement is complex in that it is not comprised solely by clicks, but also a range of involved user actions.“

If you’ve been reading along the entire time, you’ll note that my current definition of visitor engagement is derived exclusively from click-stream data and it tries to be as independent of content as possible. While this makes sense for a lot of reasons, the larger conversation (as Clint and Jeremiah wisely point out) is about how a visitor engagement metric can help us better understand the value of emerging Internet technologies.

While Web Analytics Demystified is not your typical Web 2.0 or social community site, I have enough of the activities listed above on my site to apply a social media filter to my measurement calculation and look at the effects. Again, if you’ve been reading along, I covered many of these in Part III of this series.

Here is the list of things that I am tracking vis-a-vis social media/Web 2.0 on my site:

Now, up until this point I have basically fought applying any weighting to the visitor engagement metric, mostly because I think it’s pretty difficult to rationalize any particular weighting over another and it will complicate what has already been described as “the mother of all KPIs”. That said, I am scoring these social activities into what I call an “interaction index” (ratio of sessions with one of the activities above vs. sessions without) and using the interaction index to weight the visitor engagement metric.

So instead of the existing definition of visitor engagement:

We have the new definition of “Social Engagement”:

Both metrics are the sum of component indices divided by seven, so you can hopefully see that the latter metric is weighted by any contribution made by the “Interaction Index”. For definitions of the component indices, please see Part IV in this series.

So what does this give us? Well, if you were interested in tracking individual users based on their level of visitor or social engagement, you would be able to drill-down along each Web 2.0 activity and perhaps learn something interesting:

There is Frank Faubert from Unica again, not much more socially engaged with my site than he is otherwise engaged. Remember that Frank initially complained about his only having a 21 percent engagement score, to which I responded that I had lost him in my data. Well, I found him, and based on the evolving calculation, Frank is over 31 percent engaged but little of his measured engagement is “social” in nature.

But what if I drill-down along each of my defined social activities, what can I learn?

First we can see my good friend Jeff Katz, formerly of WebTrends, who is a regular reader of my blog and whose social engagement score is much higher than his visitor engagement score. Jeff has repeatedly joined the community (Web 2.0 Measurement Working Group, Web Analytics Wednesday attendee) and has also hosted a WAW event here in Portland, OR.

Looking at direct engagement via email, we can see the great Aurelie Pols from OX2 Belgium who has also submitted comments to my blog.

I can also apply the visitor and social engagement scores to other relevant dimensions like referrers:

Here you can see that I’ve calculated the variance between visitor and social engagement and am color-coding that against my site referrers. O’Reilly’s XML.com, E-consultancy, and Jim Sterne’s Emetrics web site all are sending visitors who are well-engaged socially.

Finally, you can see the difference between visitor and social engagement applied to the various blog posts I am tracking for Clint Ivy, Ian Houston, Robbin Steif, and Avinash Kaushik. Clint’s open letter to Jeff Jarvis (a controversial piece if ever there was one) is driving a great deal of Web 2.0 engagement amongst Clint’s readers. Nice work, Clint!

Hopefully you get the picture here. By weighting the visitor engagement metric with these social media activities, I am able to easily identify individuals, referring sources, marketing campaigns, rich Internet applications, etc. that are actively interacting, both on my site (join community, engage directly, submit a comment, contribute content) and off (host an event, share a social bookmark).

Wiredset’s proposes a distilled definition of “Engagement = Interaction/Attention” which makes sense to me … you have attention by virtue of their coming to the site, but can you drive interaction? I would propose that the visitor and social engagement metrics I have described in this series of blog posts describes this equation practically applied.

As always, I welcome your comments and criticism.

Man, it seems like I just cannot get away from Gary Angel lately. He and I are engaged in some kind of crazy mutual-admiration society thing, which would normally worry me, but I know few people as into all this as Gary. He recently posted about “that darn engagement metric” where he expanded on some of my ideas and his response to my ideas and my response to his response to my ideas, etc. One thing Gary said really stuck out in my mind. Regarding my use of the visitor engagement metric to tell a story about the traffic that Marshall Sponder sends to my web site, Gary commented:

“This gradual building of a context around a measure is exactly how I think reporting actually works – and how analysis drives to actionable understanding.”

Exactly! The gradual building of context is what this is all about. The reason I’ve defined a visitor engagement metric is to provide another firm basis for the establishment of said context, another indicator on which we can draw to better understand a dimension or set of dimensions of data we collect.

Much has been written about the value of bloggers to business; it seems like you can’t open Business Week, Fortune, or even Newsweek anymore without having to read about the next big thing that bloggers and blogging are doing to change both business and society. But what can we know about the traffic bloggers send us? And what actions can we take based on that information? Let’s have a look …

Here I’ve rank-ordered the folks I list in my blogroll by percent of sessions they drive back to my web site. No great insights here based on my KPI “percent of sessions” but I suppose if I wanted to I could add Clint, Steve, Tim, and Eric Butler to my holiday card list.

Now I’ve added the session conversion KPI to my list of bloggers. Immediately I see two sets of actions I could possibly take: The first would be to send “much love” to Steve, Mike, Gary, Xavier, Manoj, Aurelie, and Tim for helping me pay for my children’s college education (all book proceeds go to my kids’s Fidelity 529 plans). The second would be to see what I could do to get the rest of the bloggers to say something like “Hey, go buy Peterson’s books, they’re great!”

Still, I should probably check to see first that these folks aren’t referring me traffic that later returns to the site and makes a purchase, right? I need to roll in a visitor-based conversion metric:

Ah ha, now I can see that I owe some serious thanks to Steven Jackson and the folks at the Blackbeak Blog! Better than one in ten people Steve has sent my way have made a book purchase, which is awesome. But it looks like I have a problem with bloggers like Anil Batra, Matt Jacobs, Robbin Steif and even Marshall Sponder. Zero percent visitor-based conversion to purchases on my site … must be some problem with how those folks are talking about me, right?

Oh, or maybe not, at least not in every case. I added a KPI for percent buy path sessions, basically the percentage of sessions in which a visitor at least starts down the book purchase path. Now I can see that Anil, Justin, and Robbin are all doing a pretty good job of getting people into the purchase consideration process, but for some reason those folks aren’t completing the purchase on my site. It’s not their fault, it’s my fault!

Damn.

But hey, maybe it’s still not my fault. Maybe even though the bloggers are sending me traffic that hits the buy path, maybe those folks aren’t really all that engaged with my site and content. Maybe the visitor’s they refer me are just looking at one page in the buy path and leaving, never to return.

Okay, or maybe not. Anil, Matt Jacobs, Marshall, and Clint are all sending me visitors that I consider to be “well engaged” with my site (my site-wide visitor engagement average is 30 percent.) Now I can see two clear action items:

1. I need to reach out to Anil and see if he and I can work out a deal to help further encourage his readers to completing the book purchase on my site. Anil, if you’re reading this, call me, we need to talk.
2. I need to reach out folks like Matt, Marshall, and Clint and see if there is some way I can get them to more passionately advocate for my books in their weblogs. Given that their visitors are more highly engaged than the “average visitor”, I have to believe their is an opportunity to sell more books.

But wait, I’m not done. In fact, I’ve only just begun to mine for the true opportunity here. But hopefully you can see, this gradual building of context is well-supported by each of the key performance indicators I keep in my arsenal, both simple metrics like “percent of sessions” and the more complex “visitor engagement”.

This post was a really long way of saying I agree with Gary about no one KPI driving a specific and easily understood action. All of our efforts are ultimately designed to help the online business better mine for opportunity and understand how that opportunity might potentially be leveraged. There are no easy answers, there are no silver bullets, there is no magic, nor mystery, nor puzzles …

There is only the gradual building of context …

Some interesting things I saw in the blogosphere the last few days:

• Robbin Steif interviewed me on behalf of the American Marketing Association for whom I’m doing a free webinar on web analytics in early March. Robbin always asks really good questions but sometimes my answers get lost in translation.
• Eric Enge has a really good interview with Jim Sterne. Jim is such a great leader of our industry and Eric Enge is really knocking ‘em down interviewing Brett Crosby, Dennis Mortenson (IndexTools), and Jim.
• Ian Thomas is busy telling us what Microsoft Gatineau will be without actually saying the words. Expect to see something outcome focused that is easy to use that provides people in a May/June timeline. Or, if you’re anxious to figure Gatineau out now, just grab the JavaScript code out of Ian’s blog and have a look-see.
• My very bright friend Ian Houston is starting a series breaking down the web analytics data model by focusing on the canonical data unit, the “event”. WARNING: Eat your Wheaties before you read Ian Houston. Ian makes perfectly logical statements like “refining the definition of Events to the concept of an Event Super Class where the sub-dimensions by type are not children of the Events dimensions but rather sub-classes of the Events class that inherit their properties and relations within the data model from the Super Class.”
• There are new jobs posted in my premium job board from PayPal, Stratigent, Cox Newspapers, Staples, and the World Wrestling Entertainment group.

Congrats to the Colts on yesterday’s Super Bowl win. I’m from outside of Chicago so my loyalties lie elsewhere but it’s nice to not see the freaking Patriots in the big game for a change.

While I have not had time to write Part V of my series on measuring visitor engagement, I wanted to take a few minutes to address some comments folks have made about the metric recently. It’s very encouraging to see folks like Gary Angel and Daniel Markus pushing the conversation about measuring engagement along as I can think of few more qualified to critique this work.

Gary Angel, who had very nice things to say about the metric, commented on how in some areas the metric is biased, specifically towards search engines and specific types of content. Gary is concerned that the Brand Index will unfairly bias towards search engines (given that one component is searches for brand-specific terms like “eric t. peterson” and “web analytics demystified”.) I examined this effect and it turns out that “branded searches” make up only a small part of the index for my site but Gary makes an excellent point, unnecessary bias should be removed from the index whenever possible. As such, in my current calculation I have removed this weighting from the Brand Index, redefining said index to only be direct sessions (non-search, non-referred.)

Score one for Gary.

Gary also commented that:

“… if I’m using my metric to measure the “engagement” produced by visitors who used a specific part of a site (like the blog or the press releases), it’s vitally important that my metric not include a strong built in bias toward one of the areas (like blogging). Some analysts might argue that this represents a flaw in the metric Eric proposes. I don’t think so. Every metric carries with it some biases – and no metric is appropriate to every situation.”

This is a good point, one that had been made by a handful of other folks who critiqued the metric early on. The problem I have with removing the Blog Index (ratio of blog reading sessions to all sessions) is the evidence that my weblog is a prime driver of engagement with my site and overall web analytics brand: Over the last 12 months, weblog subscribers are nearly 400 percent more likely to have returned to the site recently than non-readers; those visitors not subscribed to my blog (e.g., in Bloglines or Google Reader) but who are still reading blog content are 300 percent more likely to have returned recently.

Score one for Eric.

One thing worth noting, the way I am using Visual Site to measure weblog readership and subscription, this activity does not show up as traditional “page views” unless the reader A) reads the post on my web site or B) clicks through to the web site (at which time the post appears as a session “referrer”) — Visual Site is able to track external RSS and XML-based content using a non-page view event (something I call “reads”.) Not all web analytics systems afford their operators this flexibility so I thought it would be worth bringing up. This is part of the reason that the Blog Index needs to be a separate index, not part of the Click Depth Index as some have questioned.

But enough about Gary … Daniel Markus posted what I surmise to be a nice post about my visitor engagement metric at Marketing Facts late last week in which he called my calculation “the mother of all Web Analytics KPIs.” The post is entirely in Dutch and my Dutch is horrible so I wrote to Daniel and asked for a rough translation . While there were many good comments about the metric, they raised two concerns:

1. The calculation is complicated and difficult to understand.
2. There was some question of the utility of this metric, essentially calling into question the overall “actionability” (not a word) of visitor engagement.

Regarding the complexity of the calculation, as Gary has so eloquently stated any number of times, no indicator or metric is any use without understanding its components, its definition, and its inherent biases. Clearly the onus is on the web analyst to explain the metric and it’s definition to any audience they present engagement data, especially given the complete lack of formality around measuring “engagement” (at least until you started reading my posts on the subject.)

Given the complexity of the calculation, the latter concern is valid but one that misses the point of the metric. There are any number of loose definitions of “engagement” floating around in our community — duration, page views, average page views per session, sessions per visitor, etc. But none of these more easily understood (note: not easily interpreted) metrics, in my mind, captures the essence of an engaged visitor.

Visitor engagement has to be examined over diverse criteria, simple assessments simply do not work. To wit:

• To say that session duration is a good measure of engagement is fine, unless the visitor never returns to the site.
• To say that a high number of page views is a good measure of engagement is fine, unless the visitor runs up those page views in a very short period of time and was unlikely able to actually read content.
• To say that recency of visit is a good measure of engagement is fine, unless the visitor has only looked at your home page and left.
• To say that direct visits are a good measure of engagement is fine, unless those direct visits lead to short sessions of few pages viewed and the visitors never return.

I believe that the complexity of the calculation is where visitor engagement derives its value. For practitioners who are lucky enough to have access to a platform that can actually make this calculation and who are willing to take the time to explain to their audience what the metric measures and what its limitations and biases are, the metric can yield insights that would be unlikely to fall out of “traditional” web analytics.

I will leave you with an example of how I am deriving small insights from my measurement of visitor engagement.

Marshall Sponder is the WebMetricsGuru blogger and all-in-all a pretty nice guy. He and I had a little tiff awhile back over Avinash’s web analytics blogger index (something Avinash has stopped doing for some reason …) when I was less than complimentary about the volume of web analytics posts that he produced relative to his blogging in general. Examining traffic metrics from Marshall’s blog I would interpret the value of having a good relationship with him based on a set of commonly understood data:

Almost no volume and no books sold. Come on Marshall, let’s see a nice recommendation for Web Analytics Demystified already! ;-)

But wait, what if I have a closer look at the measured engagement of the visitors he’s been sending to my site:

While my “average” visitor to the site is only 24.2 percent engaged, visitors from Marshall’s posts are nearly 40 percent engaged with my site and, more importantly, of these visitors almost 10 percent are “highly engaged” (50 percent engagement or better.)

Marshall may not be selling books yet, but I have the nagging feeling if he tried even just a little, he could probably drive pretty good numbers given the engagement of the audience he referrers.

Now just imagine that you were running a million or billion dollar business, looking for new opportunities on the Internet. You have hundreds-if-not-thousands of sites sending you visitor traffic all day, every day. Maybe some of these people make purchases, but maybe you have nothing for them to purchase … how do you decide who to spend more time with and who to ignore?

Me, I’m going to write nice things about Marshall Sponder and if the folks from e-consultancy call me and want to do another interview, I’m taking that call right away! How’s that for a KPI defining an action?