Archive for 'Engagement'

Given that my last post on measuring engagement generated a fair amount of feedback, I wanted to follow-up with the post that in retrospect I should have published first, the nuts and bolts behind the engagement calculation.

Since there are numerous definitions of “engagement” that could be applied to the online channel, I choose to use the following definition:

Engagement is an estimate of the degree and depth of visitor interaction on the site against a clearly defined set of goals.

My definition sounds like conversion rate except engagement is a more flexible concept, one that can accommodate a variety of business needs such as those described by Bill Gassman in his comments to my last post and those of Craig Danuloff who is looking for a metric that accommodates a variety of visitor activities.

Based on my knowledge of my site visitors and their long-term usage patterns, my engagement goals are as follows:

1. I would like that visitors would view and interact with certain content on my site;
2. I would like visitors to subscribe to this weblog to stay connected;
3. I would like visitors to maintain a low recency with my content, regardless of whether they’re reading blog posts or viewing pages on my site;
4. When visitors are on my web site, I would like them to spend a reasonable amount of time interacting with my content;
5. When visitors return to my site, I prefer they remember my domain name and return to my site directly, either via a bookmark or by directly entering my URL into their browser.

Now, many of you will likely argue with these criteria, and fairly so. It’s fine that you may have a different definition of engagement for your site; I think that Bill put it best when he commented:

“Each organization’s version of engagement will be unique. It will be derived from a number of root metrics, probably under a dozen. Common root metrics will be frequency, recency, length of visit, purchases and lifetime value. Some organizations may include visitor actions, such as subscribing, providing personal information, writing a comment, or participating in a blog.”

I’m using the criteria I listed above based on my knowledge of my site visitors, mined from a variety of channels including site activity, email, comments, personal conversations, etc., juxtaposed against my site’s business objectives (see below.) Given a sufficiently flexible analytics package you can build your engagement metric using any goals you like …

Regarding item #1 in the list above, wanting visitors to interact with certain content on my site, here are the activities I am tracking broken down by moderate- and high-value:

Moderate-Value Activities

• Read my weblog
• Read about the Web Analytics Business Process
• Research web analytics jobs
• Add a link to my link database
• Read comments about my books
• Give me an email address
• Host a Web Analytics Wednesday
• Join the Web 2.0 Measurement Working Group

High-Value Activities

• Consider buying one or more of my books
• Buy one or more of my books
• Read about any of my books
• Read about my Key Performance Indicator Worksheets
• Download a sample copy of one of my books
• Email me directly
• Submit a comment to my weblog
• Go to Amazon.com to check out my books

Because it is very difficult to know a visitor’s intent when they visit a web site, these activities are designed to allow me to examine the visitor not in the context of their intent but rather in the context of my site’s specific objectives. I maintain Web Analytics Demystified for three primary reasons:

1. To sell my books
2. To maintain my visibility in the web analytics field
3. To have a channel through which I can continue to contribute ideas to our community

You may argue that tracking a visitor’s interaction with specific contents is a poor measure of engagement given that visitors may be looking at an entirely different set of content and are intensely engaged … fair enough. But these lists represent the activities that visitors can perform on my web site that are in-line with my stated business objectives.

If highly-engaged visitors are interacting with some other content on my site, that would prompt me to reconsider that contents contribution to my engagement calculation and perhaps add it to one of the lists above. My belief is that any engagement estimate must take content consumption into account given that it is the content that drives visitor engagement in the first place.

This post is getting long so it’s clear I’ll need a “Part III” (and maybe a “Part IV”) but here is something tangible to chew on until I have time to post again. Based on my five business goals stated above, my engagement calculation is essentially this:

(Pct High-Value Content Consumption Sessions + Pct Moderate-Value Content Consumption Sessions + Blog Subscriber Reads per Session + Pct Recent Sessions + Pct “Long” Sessions + Pct Direct Sessions) / 6

I am calculating the percentage of sessions on a per-visitor basis and summing those percentages to generate an “engagement score” between 0.0 and 6.0. I convert this score to a percentage itself to make it easier to read and voila! I can apply my engagement metric to any dimension I am tracking in Visual Site.

By clearly defining my engagement goals and then systematically scoring visitors against that framework, I can build a metric that can be objectively applied regardless of whether visitors buy a book. I can apply my engagement estimate to any dimension I am tracking on my site, allowing me to discover patterns of visitor behavior that would not be obvious based on more traditional metrics such as conversion rate, session duration, or page view count.

Just so I don’t lose you, here is one of the visualizations I am using to better understand visitor engagement showing visitor engagement by percent of visitors by visitor city:

It’s hard to see with the scale I’m providing in this image but I can assure you that the long-tail is there. And sure, with my $50 book I’m unlikely to launch a geo-targeted marketing campaign in markets where visitors are, on average, twice as engaged as my site-wide population … but maybe you would!

Until next time, I welcome your comments and criticism.

Frank Faubert, who I referenced in my last post, wrote in and had this to say:

“I think you have a flaws in both your data collection, and in the way you are thinking about engagement. I have been subscribed to your RSS feed for quite a long time, and I read it (along with many others) on a daily basis. I also have been to your web site in the past 90 days, though admittedly I don’t visit very often — mostly just to look at the status of the current Web Analytics Wednesday events.

If asked, I would consider myself an engaged visitor of your site, as I consume all of the content that you are generating, on a daily basis. Given that you push the entirety of your post in your RSS feed, I have no need to visit your site directly. (And in fact, if you were to change this to only publish snippets of your posts via RSS to force me onto the site, I would unsubscribe.) Does the fact that I am very busy, yet I can leverage the technology of a good RSS aggregator to consume your content anyway, really make me any less engaged?”

Frank makes a few really good points so I wanted to clarify something. I chose Frank as an example in my last post since I know him to be a really good guy and someone who was likely to provide thoughtful feedback. Frank didn’t let me down.

In fact, at the present time, I can’t actually track Frank’s engagement score and tie it directly to him personally (via his email address). Here is a view of Frank’s activity on my web site over time:

As you can see, Frank must have deleted the cookie that tied his email address to his site activity back in late 2005. Fair enough, Frank–I sometimes delete cookies too!

But Frank raises valid points about whether his engagement with my RSS feeds should be counted as “engaged”, and he’s absolutely correct! I’m pretty sure that Frank is located in Waltham, MA based on his attendance at the Web Analytics Wednesday event that Akin Arikan (also of Unica) hosted there awhile back. Here is a quick snapshot of activity to my site and RSS feeds from Waltham over the last eight weeks:

Clearly a few folks in Waltham are visiting the site and reading the blog, and the average engagement for visitors from Waltham is about 18 percent higher than average.

But Frank raises another really excellent point, albeit indirectly, should visitors who are actively interacting with RSS or XML-based content get a “push” in their engagement score? I mean, based on his response, Frank is doing two of the six things I have identified as “most important” on my site from a content and activity perspective (more on that in my next post), so if I could uniquely identify Frank I could vet whether his score is correct based on his description of his interaction with my site.

In reality I only provided Bill and Frank as examples to show how ultimately engagement needs to take real people’s activities into consideration, at least as the metrics are being defined and worked out. The engagement metric is really designed to be applied to dimensions other than people given that at the individual level, well, cookie deletion happens and thusly problems like the one that Frank highlights occur. Now, I suppose if you could positively identify individuals every time based on a login or customer ID that would change.

I hope I didn’t offend Bill, Frank, Robbin, or any of the other folks listed in that table as having visited my site from time-to-time. If I did, please accept my most humble of apologies. I don’t usually do this type of analysis but it helped to prove a point.

Anyway, thanks for writing in Frank and I’m glad to see you’re still reading! Hopefully we’ll get to connect next time I’m in Waltham and please give my regards to Akin, Ed, and everyone on the Unica analytics product team.

My good friend Clint Ivy and I were talking awhile back and he asked me, “So what do you think about Scoble’s call for an engagement metric?” I said, “Huh?” since I had long since stopped reading Robert Scoble, but apparently he had rubbed Clint the wrong way.

Anyway, I had been working on a project for a customer and we had been talking about how to measure engagement on their web site. We’d gone round-and-round on ideas about what constitutes an “engaged” visitor and narrowed it down to a few key areas:

1. The visitor views “critical” content on the web site
2. The visitor has returned to the web site recently
3. The visitor returns directly to the web site some of the time
4. Some high percentage of the visitor’s sessions are “long” sessions
5. If available, the visitor is subscribed to at least one available site feed

So, with this in mind, visitors that are consuming content slowly and methodically and returning directly to the site are well-engaged. Visitors who have also subscribed to some type of “push” feed are more engaged, and even more so if they’ve returned to the site recently.

Sounds reasonable, doesn’t it?

Using this model, sites like Yahoo! and Digg will have very engaged visitors, whereas sites like mine will have slightly less engaged visitors. That also sounds reasonable, given that Yahoo! and Digg are social networks and Web Analytics Demystified is more or less a weblog, a geek hub, and a job board (in that order).

It turns out that my audience is, on the whole, 32.3 percent engaged.

Perhaps more importantly, visitors that I get from the following sources are engaged at the following rates:

You can see there that my friend Avinash is sending me pretty good folks but Avinash’s people are slightly less engaged with my site than the “average” visitor. That and they hopefully already have my books because pretty much none of them are buying Web Analytics Demystified or The Big Book of Key Performance Indicators from my site! They’re not even taking advantage of the great combo-offer I have on both books!

Perhaps most interesting and wonderful is that my engagement metric allows me to build wonderful visualizations like this scatter-plot to compare the volume of referred visitors to engagement in a way that more easy on the eyes than your basic table or line graph.

Most importantly, because I am running the industry’s most easy-to-use yet powerful web analytics application supporting multi-source and multi-channel data analysis, I can vet my engagement index against real people who have come or are coming to my web site!

Cool, huh?

So Bill Gassman from the Gartner Group is among the most engaged visitors I have (I am quite honored, Bill!) Bill is consuming the content I deem most important to creating a relationship with my visitors, he is subscribed to my weblog, he keeps coming back, his sessions are of reasonable length, and he comes directly to my site or feed over 2/3rds of the time.

Bill is nearly 54 percent engaged with my site, approaching twice the average!

Compare Bill to Frank Faubert from Sane Solutions. Frank is seeing all of the content I believe to be most important and he also is well retained. However, Frank is not subscribed to my RSS feed so most of the time he is getting to my content indirectly then only spending a short period of time reading that content. Moreover, Frank hasn’t been to my site in the last 90 days.

Frank is only 21 percent engaged with my site so I guess maybe he doesn’t like me now that I’m not an objective, third-party anymore.

I’m interested in your thoughts about my engagement metric. Do you think I’m using the right inputs? Or am I missing something critical to how this metric should be calculated? I’d love you input since I know that the folks reading my weblog are among the brightest in the web analytics industry …

Next Time: Being a big fan of “showing my work”, I’ll provide the calculations behind my engagement metric so that you can calculate your site’s engagement in the safety of your own home. My hope is that through your comments and criticism I’ll be able to refine this metric down to something that any vendor can implement and any practitioner can use.