Archive for 'Engagement'

While I have not had time to write Part V of my series on measuring visitor engagement, I wanted to take a few minutes to address some comments folks have made about the metric recently. It’s very encouraging to see folks like Gary Angel and Daniel Markus pushing the conversation about measuring engagement along as I can think of few more qualified to critique this work.

Gary Angel, who had very nice things to say about the metric, commented on how in some areas the metric is biased, specifically towards search engines and specific types of content. Gary is concerned that the Brand Index will unfairly bias towards search engines (given that one component is searches for brand-specific terms like “eric t. peterson” and “web analytics demystified”.) I examined this effect and it turns out that “branded searches” make up only a small part of the index for my site but Gary makes an excellent point, unnecessary bias should be removed from the index whenever possible. As such, in my current calculation I have removed this weighting from the Brand Index, redefining said index to only be direct sessions (non-search, non-referred.)

Score one for Gary.

Gary also commented that:

“… if I’m using my metric to measure the “engagement” produced by visitors who used a specific part of a site (like the blog or the press releases), it’s vitally important that my metric not include a strong built in bias toward one of the areas (like blogging). Some analysts might argue that this represents a flaw in the metric Eric proposes. I don’t think so. Every metric carries with it some biases – and no metric is appropriate to every situation.”

This is a good point, one that had been made by a handful of other folks who critiqued the metric early on. The problem I have with removing the Blog Index (ratio of blog reading sessions to all sessions) is the evidence that my weblog is a prime driver of engagement with my site and overall web analytics brand: Over the last 12 months, weblog subscribers are nearly 400 percent more likely to have returned to the site recently than non-readers; those visitors not subscribed to my blog (e.g., in Bloglines or Google Reader) but who are still reading blog content are 300 percent more likely to have returned recently.

Score one for Eric.

One thing worth noting, the way I am using Visual Site to measure weblog readership and subscription, this activity does not show up as traditional “page views” unless the reader A) reads the post on my web site or B) clicks through to the web site (at which time the post appears as a session “referrer”) — Visual Site is able to track external RSS and XML-based content using a non-page view event (something I call “reads”.) Not all web analytics systems afford their operators this flexibility so I thought it would be worth bringing up. This is part of the reason that the Blog Index needs to be a separate index, not part of the Click Depth Index as some have questioned.

But enough about Gary … Daniel Markus posted what I surmise to be a nice post about my visitor engagement metric at Marketing Facts late last week in which he called my calculation “the mother of all Web Analytics KPIs.” The post is entirely in Dutch and my Dutch is horrible so I wrote to Daniel and asked for a rough translation . While there were many good comments about the metric, they raised two concerns:

1. The calculation is complicated and difficult to understand.
2. There was some question of the utility of this metric, essentially calling into question the overall “actionability” (not a word) of visitor engagement.

Regarding the complexity of the calculation, as Gary has so eloquently stated any number of times, no indicator or metric is any use without understanding its components, its definition, and its inherent biases. Clearly the onus is on the web analyst to explain the metric and it’s definition to any audience they present engagement data, especially given the complete lack of formality around measuring “engagement” (at least until you started reading my posts on the subject.)

Given the complexity of the calculation, the latter concern is valid but one that misses the point of the metric. There are any number of loose definitions of “engagement” floating around in our community — duration, page views, average page views per session, sessions per visitor, etc. But none of these more easily understood (note: not easily interpreted) metrics, in my mind, captures the essence of an engaged visitor.

Visitor engagement has to be examined over diverse criteria, simple assessments simply do not work. To wit:

• To say that session duration is a good measure of engagement is fine, unless the visitor never returns to the site.
• To say that a high number of page views is a good measure of engagement is fine, unless the visitor runs up those page views in a very short period of time and was unlikely able to actually read content.
• To say that recency of visit is a good measure of engagement is fine, unless the visitor has only looked at your home page and left.
• To say that direct visits are a good measure of engagement is fine, unless those direct visits lead to short sessions of few pages viewed and the visitors never return.

I believe that the complexity of the calculation is where visitor engagement derives its value. For practitioners who are lucky enough to have access to a platform that can actually make this calculation and who are willing to take the time to explain to their audience what the metric measures and what its limitations and biases are, the metric can yield insights that would be unlikely to fall out of “traditional” web analytics.

I will leave you with an example of how I am deriving small insights from my measurement of visitor engagement.

Marshall Sponder is the WebMetricsGuru blogger and all-in-all a pretty nice guy. He and I had a little tiff awhile back over Avinash’s web analytics blogger index (something Avinash has stopped doing for some reason …) when I was less than complimentary about the volume of web analytics posts that he produced relative to his blogging in general. Examining traffic metrics from Marshall’s blog I would interpret the value of having a good relationship with him based on a set of commonly understood data:

Almost no volume and no books sold. Come on Marshall, let’s see a nice recommendation for Web Analytics Demystified already! ;-)

But wait, what if I have a closer look at the measured engagement of the visitors he’s been sending to my site:

While my “average” visitor to the site is only 24.2 percent engaged, visitors from Marshall’s posts are nearly 40 percent engaged with my site and, more importantly, of these visitors almost 10 percent are “highly engaged” (50 percent engagement or better.)

Marshall may not be selling books yet, but I have the nagging feeling if he tried even just a little, he could probably drive pretty good numbers given the engagement of the audience he referrers.

Now just imagine that you were running a million or billion dollar business, looking for new opportunities on the Internet. You have hundreds-if-not-thousands of sites sending you visitor traffic all day, every day. Maybe some of these people make purchases, but maybe you have nothing for them to purchase … how do you decide who to spend more time with and who to ignore?

Me, I’m going to write nice things about Marshall Sponder and if the folks from e-consultancy call me and want to do another interview, I’m taking that call right away! How’s that for a KPI defining an action?

I wish I had more time right now to address all the valuable points that Gary Angel of SEMphonic makes in his critique of my visitor engagement metric but I’m heading out early in the AM tomorrow. Suffice to say, Gary took the time to really drill-down into my work on measuring engagement and his analysis is great. Personally, when someone of Gary’s caliber says, “… while I’d quibble with one or two of his choices, I think it’s one of the best metrics for this that I’ve ever seen in web analytics” I get a big smile on my face.

Check out Gary’s great post on “Eric Peterson’s Engagement Metric“, and be sure to read it in the context of his last post on “the myth of actionability” … something sure to raise some hackles (mine are up, but I have not had a chance to respond!)

Wow, once again I manage to fall woefully behind and am forced to play catch-up. Likely you’ve seen all of this but just in case:

• Mike Keyes at On The Trail references my engagement metric but wonders if the all-mighty dollar is not a better measure of web site quality. He wonders aloud about the engagement profile for people who submit leads to the site, something I can easily measure but will have to follow-up on later. I don’t think Mike’s idea is all that goofy!
• Daniel Riveong at Emergence Media describes my engagement metric as “pretty raw” but wisely points out measuring “buzz” is only a component of measuring engagement and that external data needs to be incorporated into site-based measurement. Sounds like Daniel is connected in the Bay Area so perhaps I’ll get invited to the next Social Media roundtable to talk about my metric.
• I am on the Online Marketing Blog’s list of blogs found useful to TopRank Online Marketing, along with many of the other bloggers I regularly read. Thanks!
• Robbin Steif at LunaMetrics blogs a conversation she had with Bob Chatham of Visual Sciences. It’s a little dated at this point but still a good read IMHO.
• There is an interesting “tags vs. logs” debate happening in the Yahoo! group. No great surprise that people have pretty strong opinions on either side of this debate. Me, I say “why choose? Why not have both if that’s what you need?!”
• Finally, two of my favorite people have finally bust forth into the blogosphere, Ian Houston of Visioactive and Dylan Lewis of Intuit. Both Ian and Dylan contributed to my O’Reilly book, Web Site Measurement Hacks and are among the brightest people I know in the web analytics arena.

Since I lined all these links up two days ago I’m sure I’m behind again but you do what you can. Think I missed something big? Send it to me in your comments!

For those of you keeping track at home, this is the fourth in what will likely be a five-part series on calculating an “engagement metric”. The first three posts are here:

• Part I
• Part II
• Part III

Originally I had postulated that an engaged visitor, at least on my web site, can be characterized as follows:

1. The visitor views “critical” content on the web site
2. The visitor has returned to the web site recently
3. The visitor returns directly to the web site some of the time
4. Some high percentage of the visitor’s sessions are “long” sessions
5. If available, the visitor is subscribed to at least one available site feed

Basically, the final calculation, one revised thanks to the valuable feedback of dozens of folks, is essentially the same with a few slight modifications. The final goals for my site, goals easily tweaked for any site, are as follows.

Well-engaged visitors will:

1. View a relatively large number of page views in a given session
2. Have visited the site in the last four weeks
3. Have relatively long sessions
4. Come directly to my site or come from a “Eric Peterson” branded search
5. Be reading my weblog in addition to non-blog content
6. Buy one or more of my books through my web site

As you can hopefully see, the first item in my original list (view “critical” content) has been softened somewhat. While the act of purchasing is necessitated by viewing critical content (my “thank you” page) ultimately I agreed with several reader comments that the a priori definition of visitor goals would skew the metric and reduce the metric’s ability to tell me about all of the content on my web site. Thanks to Victor and others for hammering this home.

Given all this, the visitor engagement metric is composed of six sub-metrics, each of which can be examined individually to provide context to the larger calculation. The six sub-metrics are:

1. Click-Depth Index: Percent of visitor sessions of “n” or more pages
2. Recency Index: Percent of visitor sessions occurring in the last “small n” weeks
3. Duration Index: Percent of visitor sessions of “n” or more minutes
4. Brand Index: Percent of visitor sessions originating directly or originating from search engine searches for terms like “eric t. peterson” and “web analytics demystified”, etc.
5. Blog Index: Ratio of blog reading sessions to all sessions
6. Conversion Index: In this case, session- or order-based conversion

Keep in mind, engagement is a visitor-based calculation, one designed to look at the lifetime of visitor sessions to the web site. So that the engagement of any visitor is a function of their lifetime of visits. Yeah, this assumes some stability in cookies so always use first-party cookies.

The final calculation is simply a summation of the component indices divided by the total number of components which yields a simple percentage:

If you’re looking across multiple visitors, you would read this as “the average visitor is just under 27 percent engaged, as defined by X, Y, and Z.” If you’re looking at a single visitor you can break engagement down on a session-by-session basis, watching for increases and decreases in the visitor’s engagement over time. In aggregate, visitor engagement becomes a very powerful but elegant key performance indicator that tells you a great deal about the make-up of your audience.

Once you decide that you need more information about the basis for an increase or decrease in visitor engagement, and assuming you have the right technology powering your analysis, you would simply visualize each of the core components over time:

As Clint commented in my last post, there is a surprising stability in each of the components, which is in my mind what you’re looking for. I want to see the variation show up when I examine engagement against my business-critical dimensions (referrer, campaign, page, search term, etc.)

When you analyze the visitor engagement calculation against all of your site visitors, you’re looking for a more-or-less normal distribution. This distribution is spiky because of the calculation, but if you’re able to drill-down, you should see something like this:

(The bars that exceed the visualization’s scale represent peaks that occur as visitors achieve 100% of sessions for 1, 2, 3, 4, and 5 of the engagement calculation’s core components. If you want to see this image at 100% scale let me know …)

Another way I can think about this is to use a scatter-plot, basically showing the same thing but easier to visualize differences as you drill-down into specific dimensions:

All of these calculations actually become relevant when you actually apply them to a dimension of data. Here, for example, is visitor engagement mapped to blog posts from my and Avinash Kaushik’s weblog:

Pretty cool, huh? I mean, it’s no great surprise that Avinash’s 2007 Web Analytics Predictions post has the highest visitor engagement score in this image when you think about all of the follow-up predictions his original post spawned. But boy-howdy, isn’t it nice to see that in a metric that you can understand and actually use?!

Here is the visitor engagement metric applied to some of the referrers to my web site:

No great surprise again that Feedburner, Technorati, and WordPress are driving visitor engagement given that they are likely to be driving visitors maxing out their blog index score. But what about the folks at ROI Revolution, sending me visitors who are on average over 30% engaged, or Blackbeak and the folks at Conversion Chronicles, sending me visitors who are as engaged as my 27 percent site-wide average?

Arrrrrrr, indeed!

Finally, and I know that I showed this already but I just think it’s damn sexy, I can map visitor engagement against any geographic dimension in my system (continent, country, city, state, zip code, DMA, etc.) to see where I might want to focus my local marketing efforts in the future:

You two people in Midland, Michigan, get ready for an onslaught of Web Analytics Demystified promotions!

Oh, some random notes:

• You can add non-page view events (RIAs, AJAX, Flash, etc.) into the calculation easily. I don’t have much of that on my site but I have an “Event Index” calculation that can be added for sites heavily leveraging these types of applications.
• You can add the “social media index” that I discussed in my last post just as easily as you can add content-based indices for retail, customer support, business-to-business, or content.
• You can take or leave my idea of scoring the brand index against specific search terms. Visual Site gives me a really easy way to do that and I believe that engagement is very much a function of brand awareness, something notoriously difficult to measure in any practical way.
• Visual Sciences customers interested in deploying the visitor engagement metric should contact me directly via normal company channels. I have pretty much everything you need to get up and running with this in a ZIP file and I’d be happy to talk you through the process.

As always, I welcome your comments and feedback on the engagement calculation and anything else that comes to mind. In the next (and perhaps final) installment I will cover Clint’s inevitable complaint of “What the heck happened to all the great ‘social media’ stuff?!?” as well as talk about some specific applications of the engagement metric.

Curse Clint Ivy, curse him for being right some of the time! I mean, of course, Clint’s diatribe about my engagement calculation and it’s lack of social (media) value. In his post, Clint gives me credit for at least trying to work out how we can measure engagement, then proceeds to chop to pieces for forgetting about the everyday blogger in my calculations.

Maybe he wasn’t that mean, but it’s late and I’m cranky … and he makes a good point. In my previous posts, I have been assigning some value in my engagement calculation directly to the viewing of specific content on my web site. But, given my respect for Mr. Ivy, and the fact that others have commented about this, I took out the high- and moderate-value content scoring and have substituted (experimentally) a Social Media Index. After looking at my site, I am now scoring the following “social media” activities one can engage in at Web Analytics Demystified:

• Reading my weblog
• Reading other user generated content on the web site
• Participating in a truly social activity facilitated by my site
• Joining a social network of web analytics people
• Contributing content directly to the web site
• Submitting a comment to my weblog
• Emailing me directly

Yeah, I know that my site is no Digg.com, nor is it Friendster or YouTube, but hopefully you get the gist. The measurement works pretty much the same, regardless of the volume of traffic. The net effect is to, at least in my mind, remove some of the content-specificity from the calculation while improving the metrics ability to help sites understand visitor attraction to activities designed to draw the visitor in.

This list could just as easily include providing a rating, tagging, Digging, etc. Depending on the technology you use, the measurements don’t even need to be direct. Think of my list as a strawman, one that can be brutally beaten into better shape (but, unlike almost everything else I’ve seen so far, one that actually functions now …)

One thing that Clint and I talked about off-line that wasn’t represented in his post (or maybe it was, it is getting later by the minute) was whether the engagement calculation would provide any additional value, relative to “corporate” measurements like conversion rate. I took a look at that, mapping my buyer conversion rate and engagement against the visitor’s session number. I got this:

While it clearly looks like if I don’t get ‘em to buy one of my books pretty quickly after they first come to the site my opportunity to convert goes down pretty fast, the opposite is true for engagement. It actually appears that, at least on my site, there is a sweet spot for visitor engagement between about 40 and 50 sessions … heck, I even sold a few books to folks well after their initial visit once their engagement ran up to over 55 percent!

The nice thing about the engagement metric is that it helps resolve the problem that Gary describes in his recent post on visitor classification. Gary, in talking about the need to capture and visualize both absolute and relative content usage on a site says this:

The problem is that heavily engaged users of your site will show up (and often drive the statistics for) virtually every area of your site. For publishing clients, a small segment of heavily engaged users inevitably show up in every single content area. And the smaller the overall usage of that area, the more the heavily engaged component influences the results.

Yep, so wouldn’t it be nice if you could not only create on-the-fly visitor segments that are inclusive of any different number of content areas and pages on your site plus easily determine how much of an influence highly engaged visitors are on your absolute content usage measurements? If you could do that, it would probably look something like this:

I know it’s hard to see, but I simply dragged a bunch of pages, groups of pages, and content groups onto the page visualization map and told Visual Site to color the nodes by visitor engagement (the height of the bars represents the relative number of sessions to each node.) I could then select-in or select-out visitors based on their relative level of engagement to identify the special kinds of customers Gary refers to.

Anyway, I’m going to have beers with the good Mr. Ivy next week and I didn’t want that whole “social media” thing hanging over my head. And while I recognize that this metric (which I still have yet to share the calculation) doesn’t capture fully the elaborate needs of the really smart folks working to pound out Social Media Measurement, I heartily agree with Clint’s friend Jeremiah Owyang when he says that “Social Media is about people. People connecting to other people to build better relationships, fostering communities and increasing collective knowledge” and “Measurement and Metrics are one way to help to tell the story of Social Media.”

Measurement and Metrics, indeed.