Archive for 'Engagement'

[ I'm really happy to have my first "guest post" from blogger Robbin Steif from Lunametrics. Robbin really liked my "gradual building of context" post from awhile back and she and I have been discussing a related metric that she thinks builds nicely on my visitor engagement metric. Without further ado, Robbin Steif ... ]

On the one hand, I thought that Eric’s recent post, The Gradual Building of Context, was just awesome. Although every site has to define visitor engagement for itself, every site is still capable of pulling together similar numbers (which is why I loved it.)

On the other hand, I disagreed with Eric’s final conclusion, “I need to reach out folks like Matt, Marshall, and Clint and see if there is some way I can get them to more passionately advocate for my books in their weblogs. Given that their visitors are more highly engaged than the “average visitor’, I have to believe their is an opportunity to sell more books.”

I took one look at the numbers in the last chart and thought, well, that doesn’t make sense. Sure, Matt’s visitors (or Clint’s or Marshall’s) are somewhat more engaged than the average visitor, but their “start the purchase cycle” numbers are pitiful. If Eric were to put effort into this, the place to put effort in is where both those metrics are strong.

Eric was good enough to send me the spreadsheet, and I pushed the numbers. (Well ok, technically he and I pushed the numbers at the same time over the phone …) On the phone, he called it “Robbin’s Metric” and I left it that way. It is the product of his Visitor Engagement and Percent Buy Path Sessions:

[ Ugh! Yes, I know that image is hard to read! I will correct ASAP!!! ]

By multiplying the two metrics and then ranking all the referring blogs by that metric, you see where Eric should put in extra effort. I agree witrh the first conclusion that Eric already came to in his blog, i.e. that he needs to work out some kind of deal with Anil. However, the two blogs where he should put time/effort would be Justin Cutroni’s and ROI Revolution’s. Interestingly, they are both Google Analytic blogs, so there is a decent chance that the reader is newer at analytics and probably could really benefit from Eric’s books. I didn’t highlight Steve Jackson’s blog, Xavier’s or Aurelie’s because they are already converting well (if there is such thing as converting well.)

Finaly thoughts: An engaged visitor to a site that is also content rich, like Eric’s, doesn’t necessarily make a good customer. In fact, Clint did a survey on his blog and saw that many of his visitors already own many, if not all, of Eric’s books. When visitors go to the beginning of the checkout, we can actually see interest in the purchase, as well as interest in the content — and as direct marketers know, you should always pursue the customer who already has a propensity to buy.

[ Thanks to Robbin for taking the time to take visitor engagement to the next level! What do you think? Is Robbin on the right track? Did I miss the mark? As always, your comments are greatly appreciated! ]

(If you need to catch up on where we are to date, have a look at my last post in this series on measuring visitor engagement.)

I had a nice conversation a few days ago with Jeremiah Owyang, Web Strategist at PodTech.net, on how I have been measuring engagement. Jeremiah has been thinking about how engagement is defined for some time and had a very fresh perspective on the subject which has somewhat expanded my thinking on the subject. Jeremiah, by virtue of being an “A-list” blogger (IMHO) gets great critical feedback from folks like Forrester’s Charlene Li (who says that my measurement is too explicit, oh well …) After we talked, I realized that I really needed to get the promised post on measuring “social engagement in a Web 2.0 world” out the door. So here it is.

One of the links that Jeremiah references is this one from Wiredset, published in November of last year. In their post, Wiredset gives a definition of engagement as “a consumer based measurement that regards interaction with an aspect of a brand or media property” and goes on to say that “Web 2.0 Engagement” could include activities (Jeremiah refers to these as “gestures”) like:

• Publishing
• Creating and Publishing to a Group
• Posting
• Subscribing
• Favoriting
• Adding Friends
• Bookmarking
• Emailing
• Distributing
• Streaming
• Networking
• Creating Mash-up Content

I absolutely agree with Wiredset, and they go on to say:

“When measuring engagement, the level of user interaction (i.e. 200 vs. 2,000,000 streams) is an obvious and important component. Yet engagement is complex in that it is not comprised solely by clicks, but also a range of involved user actions.“

If you’ve been reading along the entire time, you’ll note that my current definition of visitor engagement is derived exclusively from click-stream data and it tries to be as independent of content as possible. While this makes sense for a lot of reasons, the larger conversation (as Clint and Jeremiah wisely point out) is about how a visitor engagement metric can help us better understand the value of emerging Internet technologies.

While Web Analytics Demystified is not your typical Web 2.0 or social community site, I have enough of the activities listed above on my site to apply a social media filter to my measurement calculation and look at the effects. Again, if you’ve been reading along, I covered many of these in Part III of this series.

Here is the list of things that I am tracking vis-a-vis social media/Web 2.0 on my site:

Now, up until this point I have basically fought applying any weighting to the visitor engagement metric, mostly because I think it’s pretty difficult to rationalize any particular weighting over another and it will complicate what has already been described as “the mother of all KPIs”. That said, I am scoring these social activities into what I call an “interaction index” (ratio of sessions with one of the activities above vs. sessions without) and using the interaction index to weight the visitor engagement metric.

So instead of the existing definition of visitor engagement:

We have the new definition of “Social Engagement”:

Both metrics are the sum of component indices divided by seven, so you can hopefully see that the latter metric is weighted by any contribution made by the “Interaction Index”. For definitions of the component indices, please see Part IV in this series.

So what does this give us? Well, if you were interested in tracking individual users based on their level of visitor or social engagement, you would be able to drill-down along each Web 2.0 activity and perhaps learn something interesting:

There is Frank Faubert from Unica again, not much more socially engaged with my site than he is otherwise engaged. Remember that Frank initially complained about his only having a 21 percent engagement score, to which I responded that I had lost him in my data. Well, I found him, and based on the evolving calculation, Frank is over 31 percent engaged but little of his measured engagement is “social” in nature.

But what if I drill-down along each of my defined social activities, what can I learn?

First we can see my good friend Jeff Katz, formerly of WebTrends, who is a regular reader of my blog and whose social engagement score is much higher than his visitor engagement score. Jeff has repeatedly joined the community (Web 2.0 Measurement Working Group, Web Analytics Wednesday attendee) and has also hosted a WAW event here in Portland, OR.

Looking at direct engagement via email, we can see the great Aurelie Pols from OX2 Belgium who has also submitted comments to my blog.

I can also apply the visitor and social engagement scores to other relevant dimensions like referrers:

Here you can see that I’ve calculated the variance between visitor and social engagement and am color-coding that against my site referrers. O’Reilly’s XML.com, E-consultancy, and Jim Sterne’s Emetrics web site all are sending visitors who are well-engaged socially.

Finally, you can see the difference between visitor and social engagement applied to the various blog posts I am tracking for Clint Ivy, Ian Houston, Robbin Steif, and Avinash Kaushik. Clint’s open letter to Jeff Jarvis (a controversial piece if ever there was one) is driving a great deal of Web 2.0 engagement amongst Clint’s readers. Nice work, Clint!

Hopefully you get the picture here. By weighting the visitor engagement metric with these social media activities, I am able to easily identify individuals, referring sources, marketing campaigns, rich Internet applications, etc. that are actively interacting, both on my site (join community, engage directly, submit a comment, contribute content) and off (host an event, share a social bookmark).

Wiredset’s proposes a distilled definition of “Engagement = Interaction/Attention” which makes sense to me … you have attention by virtue of their coming to the site, but can you drive interaction? I would propose that the visitor and social engagement metrics I have described in this series of blog posts describes this equation practically applied.

As always, I welcome your comments and criticism.

Man, it seems like I just cannot get away from Gary Angel lately. He and I are engaged in some kind of crazy mutual-admiration society thing, which would normally worry me, but I know few people as into all this as Gary. He recently posted about “that darn engagement metric” where he expanded on some of my ideas and his response to my ideas and my response to his response to my ideas, etc. One thing Gary said really stuck out in my mind. Regarding my use of the visitor engagement metric to tell a story about the traffic that Marshall Sponder sends to my web site, Gary commented:

“This gradual building of a context around a measure is exactly how I think reporting actually works – and how analysis drives to actionable understanding.”

Exactly! The gradual building of context is what this is all about. The reason I’ve defined a visitor engagement metric is to provide another firm basis for the establishment of said context, another indicator on which we can draw to better understand a dimension or set of dimensions of data we collect.

Much has been written about the value of bloggers to business; it seems like you can’t open Business Week, Fortune, or even Newsweek anymore without having to read about the next big thing that bloggers and blogging are doing to change both business and society. But what can we know about the traffic bloggers send us? And what actions can we take based on that information? Let’s have a look …

Here I’ve rank-ordered the folks I list in my blogroll by percent of sessions they drive back to my web site. No great insights here based on my KPI “percent of sessions” but I suppose if I wanted to I could add Clint, Steve, Tim, and Eric Butler to my holiday card list.

Now I’ve added the session conversion KPI to my list of bloggers. Immediately I see two sets of actions I could possibly take: The first would be to send “much love” to Steve, Mike, Gary, Xavier, Manoj, Aurelie, and Tim for helping me pay for my children’s college education (all book proceeds go to my kids’s Fidelity 529 plans). The second would be to see what I could do to get the rest of the bloggers to say something like “Hey, go buy Peterson’s books, they’re great!”

Still, I should probably check to see first that these folks aren’t referring me traffic that later returns to the site and makes a purchase, right? I need to roll in a visitor-based conversion metric:

Ah ha, now I can see that I owe some serious thanks to Steven Jackson and the folks at the Blackbeak Blog! Better than one in ten people Steve has sent my way have made a book purchase, which is awesome. But it looks like I have a problem with bloggers like Anil Batra, Matt Jacobs, Robbin Steif and even Marshall Sponder. Zero percent visitor-based conversion to purchases on my site … must be some problem with how those folks are talking about me, right?

Oh, or maybe not, at least not in every case. I added a KPI for percent buy path sessions, basically the percentage of sessions in which a visitor at least starts down the book purchase path. Now I can see that Anil, Justin, and Robbin are all doing a pretty good job of getting people into the purchase consideration process, but for some reason those folks aren’t completing the purchase on my site. It’s not their fault, it’s my fault!

Damn.

But hey, maybe it’s still not my fault. Maybe even though the bloggers are sending me traffic that hits the buy path, maybe those folks aren’t really all that engaged with my site and content. Maybe the visitor’s they refer me are just looking at one page in the buy path and leaving, never to return.

Okay, or maybe not. Anil, Matt Jacobs, Marshall, and Clint are all sending me visitors that I consider to be “well engaged” with my site (my site-wide visitor engagement average is 30 percent.) Now I can see two clear action items:

1. I need to reach out to Anil and see if he and I can work out a deal to help further encourage his readers to completing the book purchase on my site. Anil, if you’re reading this, call me, we need to talk.
2. I need to reach out folks like Matt, Marshall, and Clint and see if there is some way I can get them to more passionately advocate for my books in their weblogs. Given that their visitors are more highly engaged than the “average visitor”, I have to believe their is an opportunity to sell more books.

But wait, I’m not done. In fact, I’ve only just begun to mine for the true opportunity here. But hopefully you can see, this gradual building of context is well-supported by each of the key performance indicators I keep in my arsenal, both simple metrics like “percent of sessions” and the more complex “visitor engagement”.

This post was a really long way of saying I agree with Gary about no one KPI driving a specific and easily understood action. All of our efforts are ultimately designed to help the online business better mine for opportunity and understand how that opportunity might potentially be leveraged. There are no easy answers, there are no silver bullets, there is no magic, nor mystery, nor puzzles …

There is only the gradual building of context …

While I have not had time to write Part V of my series on measuring visitor engagement, I wanted to take a few minutes to address some comments folks have made about the metric recently. It’s very encouraging to see folks like Gary Angel and Daniel Markus pushing the conversation about measuring engagement along as I can think of few more qualified to critique this work.

Gary Angel, who had very nice things to say about the metric, commented on how in some areas the metric is biased, specifically towards search engines and specific types of content. Gary is concerned that the Brand Index will unfairly bias towards search engines (given that one component is searches for brand-specific terms like “eric t. peterson” and “web analytics demystified”.) I examined this effect and it turns out that “branded searches” make up only a small part of the index for my site but Gary makes an excellent point, unnecessary bias should be removed from the index whenever possible. As such, in my current calculation I have removed this weighting from the Brand Index, redefining said index to only be direct sessions (non-search, non-referred.)

Score one for Gary.

Gary also commented that:

“… if I’m using my metric to measure the “engagement” produced by visitors who used a specific part of a site (like the blog or the press releases), it’s vitally important that my metric not include a strong built in bias toward one of the areas (like blogging). Some analysts might argue that this represents a flaw in the metric Eric proposes. I don’t think so. Every metric carries with it some biases – and no metric is appropriate to every situation.”

This is a good point, one that had been made by a handful of other folks who critiqued the metric early on. The problem I have with removing the Blog Index (ratio of blog reading sessions to all sessions) is the evidence that my weblog is a prime driver of engagement with my site and overall web analytics brand: Over the last 12 months, weblog subscribers are nearly 400 percent more likely to have returned to the site recently than non-readers; those visitors not subscribed to my blog (e.g., in Bloglines or Google Reader) but who are still reading blog content are 300 percent more likely to have returned recently.

Score one for Eric.

One thing worth noting, the way I am using Visual Site to measure weblog readership and subscription, this activity does not show up as traditional “page views” unless the reader A) reads the post on my web site or B) clicks through to the web site (at which time the post appears as a session “referrer”) — Visual Site is able to track external RSS and XML-based content using a non-page view event (something I call “reads”.) Not all web analytics systems afford their operators this flexibility so I thought it would be worth bringing up. This is part of the reason that the Blog Index needs to be a separate index, not part of the Click Depth Index as some have questioned.

But enough about Gary … Daniel Markus posted what I surmise to be a nice post about my visitor engagement metric at Marketing Facts late last week in which he called my calculation “the mother of all Web Analytics KPIs.” The post is entirely in Dutch and my Dutch is horrible so I wrote to Daniel and asked for a rough translation . While there were many good comments about the metric, they raised two concerns:

1. The calculation is complicated and difficult to understand.
2. There was some question of the utility of this metric, essentially calling into question the overall “actionability” (not a word) of visitor engagement.

Regarding the complexity of the calculation, as Gary has so eloquently stated any number of times, no indicator or metric is any use without understanding its components, its definition, and its inherent biases. Clearly the onus is on the web analyst to explain the metric and it’s definition to any audience they present engagement data, especially given the complete lack of formality around measuring “engagement” (at least until you started reading my posts on the subject.)

Given the complexity of the calculation, the latter concern is valid but one that misses the point of the metric. There are any number of loose definitions of “engagement” floating around in our community — duration, page views, average page views per session, sessions per visitor, etc. But none of these more easily understood (note: not easily interpreted) metrics, in my mind, captures the essence of an engaged visitor.

Visitor engagement has to be examined over diverse criteria, simple assessments simply do not work. To wit:

• To say that session duration is a good measure of engagement is fine, unless the visitor never returns to the site.
• To say that a high number of page views is a good measure of engagement is fine, unless the visitor runs up those page views in a very short period of time and was unlikely able to actually read content.
• To say that recency of visit is a good measure of engagement is fine, unless the visitor has only looked at your home page and left.
• To say that direct visits are a good measure of engagement is fine, unless those direct visits lead to short sessions of few pages viewed and the visitors never return.

I believe that the complexity of the calculation is where visitor engagement derives its value. For practitioners who are lucky enough to have access to a platform that can actually make this calculation and who are willing to take the time to explain to their audience what the metric measures and what its limitations and biases are, the metric can yield insights that would be unlikely to fall out of “traditional” web analytics.

I will leave you with an example of how I am deriving small insights from my measurement of visitor engagement.

Marshall Sponder is the WebMetricsGuru blogger and all-in-all a pretty nice guy. He and I had a little tiff awhile back over Avinash’s web analytics blogger index (something Avinash has stopped doing for some reason …) when I was less than complimentary about the volume of web analytics posts that he produced relative to his blogging in general. Examining traffic metrics from Marshall’s blog I would interpret the value of having a good relationship with him based on a set of commonly understood data:

Almost no volume and no books sold. Come on Marshall, let’s see a nice recommendation for Web Analytics Demystified already! ;-)

But wait, what if I have a closer look at the measured engagement of the visitors he’s been sending to my site:

While my “average” visitor to the site is only 24.2 percent engaged, visitors from Marshall’s posts are nearly 40 percent engaged with my site and, more importantly, of these visitors almost 10 percent are “highly engaged” (50 percent engagement or better.)

Marshall may not be selling books yet, but I have the nagging feeling if he tried even just a little, he could probably drive pretty good numbers given the engagement of the audience he referrers.

Now just imagine that you were running a million or billion dollar business, looking for new opportunities on the Internet. You have hundreds-if-not-thousands of sites sending you visitor traffic all day, every day. Maybe some of these people make purchases, but maybe you have nothing for them to purchase … how do you decide who to spend more time with and who to ignore?

Me, I’m going to write nice things about Marshall Sponder and if the folks from e-consultancy call me and want to do another interview, I’m taking that call right away! How’s that for a KPI defining an action?

I wish I had more time right now to address all the valuable points that Gary Angel of SEMphonic makes in his critique of my visitor engagement metric but I’m heading out early in the AM tomorrow. Suffice to say, Gary took the time to really drill-down into my work on measuring engagement and his analysis is great. Personally, when someone of Gary’s caliber says, “… while I’d quibble with one or two of his choices, I think it’s one of the best metrics for this that I’ve ever seen in web analytics” I get a big smile on my face.

Check out Gary’s great post on “Eric Peterson’s Engagement Metric“, and be sure to read it in the context of his last post on “the myth of actionability” … something sure to raise some hackles (mine are up, but I have not had a chance to respond!)