Archive for January, 2007

For those of you keeping track at home, this is the fourth in what will likely be a five-part series on calculating an “engagement metric”. The first three posts are here:

• Part I
• Part II
• Part III

Originally I had postulated that an engaged visitor, at least on my web site, can be characterized as follows:

1. The visitor views “critical” content on the web site
2. The visitor has returned to the web site recently
3. The visitor returns directly to the web site some of the time
4. Some high percentage of the visitor’s sessions are “long” sessions
5. If available, the visitor is subscribed to at least one available site feed

Basically, the final calculation, one revised thanks to the valuable feedback of dozens of folks, is essentially the same with a few slight modifications. The final goals for my site, goals easily tweaked for any site, are as follows.

Well-engaged visitors will:

1. View a relatively large number of page views in a given session
2. Have visited the site in the last four weeks
3. Have relatively long sessions
4. Come directly to my site or come from a “Eric Peterson” branded search
5. Be reading my weblog in addition to non-blog content
6. Buy one or more of my books through my web site

As you can hopefully see, the first item in my original list (view “critical” content) has been softened somewhat. While the act of purchasing is necessitated by viewing critical content (my “thank you” page) ultimately I agreed with several reader comments that the a priori definition of visitor goals would skew the metric and reduce the metric’s ability to tell me about all of the content on my web site. Thanks to Victor and others for hammering this home.

Given all this, the visitor engagement metric is composed of six sub-metrics, each of which can be examined individually to provide context to the larger calculation. The six sub-metrics are:

1. Click-Depth Index: Percent of visitor sessions of “n” or more pages
2. Recency Index: Percent of visitor sessions occurring in the last “small n” weeks
3. Duration Index: Percent of visitor sessions of “n” or more minutes
4. Brand Index: Percent of visitor sessions originating directly or originating from search engine searches for terms like “eric t. peterson” and “web analytics demystified”, etc.
5. Blog Index: Ratio of blog reading sessions to all sessions
6. Conversion Index: In this case, session- or order-based conversion

Keep in mind, engagement is a visitor-based calculation, one designed to look at the lifetime of visitor sessions to the web site. So that the engagement of any visitor is a function of their lifetime of visits. Yeah, this assumes some stability in cookies so always use first-party cookies.

The final calculation is simply a summation of the component indices divided by the total number of components which yields a simple percentage:

If you’re looking across multiple visitors, you would read this as “the average visitor is just under 27 percent engaged, as defined by X, Y, and Z.” If you’re looking at a single visitor you can break engagement down on a session-by-session basis, watching for increases and decreases in the visitor’s engagement over time. In aggregate, visitor engagement becomes a very powerful but elegant key performance indicator that tells you a great deal about the make-up of your audience.

Once you decide that you need more information about the basis for an increase or decrease in visitor engagement, and assuming you have the right technology powering your analysis, you would simply visualize each of the core components over time:

As Clint commented in my last post, there is a surprising stability in each of the components, which is in my mind what you’re looking for. I want to see the variation show up when I examine engagement against my business-critical dimensions (referrer, campaign, page, search term, etc.)

When you analyze the visitor engagement calculation against all of your site visitors, you’re looking for a more-or-less normal distribution. This distribution is spiky because of the calculation, but if you’re able to drill-down, you should see something like this:

(The bars that exceed the visualization’s scale represent peaks that occur as visitors achieve 100% of sessions for 1, 2, 3, 4, and 5 of the engagement calculation’s core components. If you want to see this image at 100% scale let me know …)

Another way I can think about this is to use a scatter-plot, basically showing the same thing but easier to visualize differences as you drill-down into specific dimensions:

All of these calculations actually become relevant when you actually apply them to a dimension of data. Here, for example, is visitor engagement mapped to blog posts from my and Avinash Kaushik’s weblog:

Pretty cool, huh? I mean, it’s no great surprise that Avinash’s 2007 Web Analytics Predictions post has the highest visitor engagement score in this image when you think about all of the follow-up predictions his original post spawned. But boy-howdy, isn’t it nice to see that in a metric that you can understand and actually use?!

Here is the visitor engagement metric applied to some of the referrers to my web site:

No great surprise again that Feedburner, Technorati, and WordPress are driving visitor engagement given that they are likely to be driving visitors maxing out their blog index score. But what about the folks at ROI Revolution, sending me visitors who are on average over 30% engaged, or Blackbeak and the folks at Conversion Chronicles, sending me visitors who are as engaged as my 27 percent site-wide average?

Arrrrrrr, indeed!

Finally, and I know that I showed this already but I just think it’s damn sexy, I can map visitor engagement against any geographic dimension in my system (continent, country, city, state, zip code, DMA, etc.) to see where I might want to focus my local marketing efforts in the future:

You two people in Midland, Michigan, get ready for an onslaught of Web Analytics Demystified promotions!

Oh, some random notes:

• You can add non-page view events (RIAs, AJAX, Flash, etc.) into the calculation easily. I don’t have much of that on my site but I have an “Event Index” calculation that can be added for sites heavily leveraging these types of applications.
• You can add the “social media index” that I discussed in my last post just as easily as you can add content-based indices for retail, customer support, business-to-business, or content.
• You can take or leave my idea of scoring the brand index against specific search terms. Visual Site gives me a really easy way to do that and I believe that engagement is very much a function of brand awareness, something notoriously difficult to measure in any practical way.
• Visual Sciences customers interested in deploying the visitor engagement metric should contact me directly via normal company channels. I have pretty much everything you need to get up and running with this in a ZIP file and I’d be happy to talk you through the process.

As always, I welcome your comments and feedback on the engagement calculation and anything else that comes to mind. In the next (and perhaps final) installment I will cover Clint’s inevitable complaint of “What the heck happened to all the great ‘social media’ stuff?!?” as well as talk about some specific applications of the engagement metric.

I got word a few days back that W. Greg Dowling, my replacement at JupiterResearch covering among other things web analytics, has left JupiterResearch for Modem Media. Greg started last week as the Vice President/Associate Director of Digital Marketing Analytics. According to Greg, his new job will have him:

“[Focusing] on developing Modem’s capabilities in Web analytics, growing Web analytics consulting revenue, and building cross-capability integration of Web analytics into all Modem deliverables.”

While I am quite bummed out to hear that Greg has left JupiterResearch, after talking to him for awhile I’m excited for him in his new opportunity. Best of luck, Greg!

UPDATED: I am a bad blogger. I referenced my friend Dylan Lewis in my original post and didn’t link to his blog. Dylan can be read at http://www.passionateanalyst.com/ and I encourage you to ask about the Platypus thing. My sincere apologies, Mr. Lewis!

UPDATED: I totally forgot about all the 2007 web analytics predictions! You can see a nice summary list posted to the Yahoo! group by Lars but this list does not include one of my favorite set of predictions from Craig Danuloff. Craig I think makes the boldest predictions of anyone playing the game.

It occurred to me last night that I’ve been so engaged with measuring engagement that some interesting stuff has passed me by. Let’s catch up over coffee, shall we?

• After being “tagged” by Gary Angel I tagged a few folks. So far Clint Ivy and Eric Butler have responded to my tagging, but perhaps the most interesting tag response comes from Dylan Lewis who postulates that the game of tag is either a thin disguise to increase our page ranks for searches for things like “web analytics and the grateful dead” or some intense navel gazing.
• The “death of the page view” conversation, while interesting, is starting to go too far when otherwise smart people begin to predict things like “we’ll no longer bother to collect pageviews by end of 2007.” While you can make the case for using unique visitors in comparative situations, I sincerely question statements like “[the] page is no longer considered a metric worth looking at.” Is it me?
• Justin Cutroni had a really good post a few weeks back titled “Web Analytics: It’s About Process” that I loved. I’m still well-engaged thinking about the processes behind the successful “doing” of web analytics at the Enterprise-level and have had several enlightening conversations lately. One of my favorite comments was “people often consider “process” to be a dirty word …” Ouch!
• There is a new job posted on my job board, which is slow to take off likely because I have been too busy to bug all the recruiters posting to the Yahoo! group to give it a try! If you’re in Utah and have experience with the local analytics technology, have a look at this posting.
• Finally, and those of us thinking about how “Web 2.0″ is going to be measured knew this was bound to happen, measurement tools are coming to Second Life. While I’m not a Second-Lifer (I barely get everything done in my first life) I am dying to see what kinds of metrics Electric Sheep are able to come up with. Can you imagine the KPIs? “Percent Avatars propositioning sex” and “Percent Avatars pretending to be adults who were probably eleven-year-olds” and the such. Seriously though, if you have access to these reports, I would LOVE to see them.

As usual, I welcome your comments.

Curse Clint Ivy, curse him for being right some of the time! I mean, of course, Clint’s diatribe about my engagement calculation and it’s lack of social (media) value. In his post, Clint gives me credit for at least trying to work out how we can measure engagement, then proceeds to chop to pieces for forgetting about the everyday blogger in my calculations.

Maybe he wasn’t that mean, but it’s late and I’m cranky … and he makes a good point. In my previous posts, I have been assigning some value in my engagement calculation directly to the viewing of specific content on my web site. But, given my respect for Mr. Ivy, and the fact that others have commented about this, I took out the high- and moderate-value content scoring and have substituted (experimentally) a Social Media Index. After looking at my site, I am now scoring the following “social media” activities one can engage in at Web Analytics Demystified:

• Reading my weblog
• Reading other user generated content on the web site
• Participating in a truly social activity facilitated by my site
• Joining a social network of web analytics people
• Contributing content directly to the web site
• Submitting a comment to my weblog
• Emailing me directly

Yeah, I know that my site is no Digg.com, nor is it Friendster or YouTube, but hopefully you get the gist. The measurement works pretty much the same, regardless of the volume of traffic. The net effect is to, at least in my mind, remove some of the content-specificity from the calculation while improving the metrics ability to help sites understand visitor attraction to activities designed to draw the visitor in.

This list could just as easily include providing a rating, tagging, Digging, etc. Depending on the technology you use, the measurements don’t even need to be direct. Think of my list as a strawman, one that can be brutally beaten into better shape (but, unlike almost everything else I’ve seen so far, one that actually functions now …)

One thing that Clint and I talked about off-line that wasn’t represented in his post (or maybe it was, it is getting later by the minute) was whether the engagement calculation would provide any additional value, relative to “corporate” measurements like conversion rate. I took a look at that, mapping my buyer conversion rate and engagement against the visitor’s session number. I got this:

While it clearly looks like if I don’t get ‘em to buy one of my books pretty quickly after they first come to the site my opportunity to convert goes down pretty fast, the opposite is true for engagement. It actually appears that, at least on my site, there is a sweet spot for visitor engagement between about 40 and 50 sessions … heck, I even sold a few books to folks well after their initial visit once their engagement ran up to over 55 percent!

The nice thing about the engagement metric is that it helps resolve the problem that Gary describes in his recent post on visitor classification. Gary, in talking about the need to capture and visualize both absolute and relative content usage on a site says this:

The problem is that heavily engaged users of your site will show up (and often drive the statistics for) virtually every area of your site. For publishing clients, a small segment of heavily engaged users inevitably show up in every single content area. And the smaller the overall usage of that area, the more the heavily engaged component influences the results.

Yep, so wouldn’t it be nice if you could not only create on-the-fly visitor segments that are inclusive of any different number of content areas and pages on your site plus easily determine how much of an influence highly engaged visitors are on your absolute content usage measurements? If you could do that, it would probably look something like this:

I know it’s hard to see, but I simply dragged a bunch of pages, groups of pages, and content groups onto the page visualization map and told Visual Site to color the nodes by visitor engagement (the height of the bars represents the relative number of sessions to each node.) I could then select-in or select-out visitors based on their relative level of engagement to identify the special kinds of customers Gary refers to.

Anyway, I’m going to have beers with the good Mr. Ivy next week and I didn’t want that whole “social media” thing hanging over my head. And while I recognize that this metric (which I still have yet to share the calculation) doesn’t capture fully the elaborate needs of the really smart folks working to pound out Social Media Measurement, I heartily agree with Clint’s friend Jeremiah Owyang when he says that “Social Media is about people. People connecting to other people to build better relationships, fostering communities and increasing collective knowledge” and “Measurement and Metrics are one way to help to tell the story of Social Media.”

Measurement and Metrics, indeed.