Curse Clint Ivy, curse him for being right some of the time! I mean, of course, Clint’s diatribe about my engagement calculation and it’s lack of social (media) value. In his post, Clint gives me credit for at least trying to work out how we can measure engagement, then proceeds to chop to pieces for forgetting about the everyday blogger in my calculations.

Maybe he wasn’t that mean, but it’s late and I’m cranky … and he makes a good point. In my previous posts, I have been assigning some value in my engagement calculation directly to the viewing of specific content on my web site. But, given my respect for Mr. Ivy, and the fact that others have commented about this, I took out the high- and moderate-value content scoring and have substituted (experimentally) a Social Media Index. After looking at my site, I am now scoring the following “social media” activities one can engage in at Web Analytics Demystified:

• Reading my weblog
• Reading other user generated content on the web site
• Participating in a truly social activity facilitated by my site
• Joining a social network of web analytics people
• Contributing content directly to the web site
• Submitting a comment to my weblog
• Emailing me directly

Yeah, I know that my site is no Digg.com, nor is it Friendster or YouTube, but hopefully you get the gist. The measurement works pretty much the same, regardless of the volume of traffic. The net effect is to, at least in my mind, remove some of the content-specificity from the calculation while improving the metrics ability to help sites understand visitor attraction to activities designed to draw the visitor in.

This list could just as easily include providing a rating, tagging, Digging, etc. Depending on the technology you use, the measurements don’t even need to be direct. Think of my list as a strawman, one that can be brutally beaten into better shape (but, unlike almost everything else I’ve seen so far, one that actually functions now …)

One thing that Clint and I talked about off-line that wasn’t represented in his post (or maybe it was, it is getting later by the minute) was whether the engagement calculation would provide any additional value, relative to “corporate” measurements like conversion rate. I took a look at that, mapping my buyer conversion rate and engagement against the visitor’s session number. I got this:

While it clearly looks like if I don’t get ‘em to buy one of my books pretty quickly after they first come to the site my opportunity to convert goes down pretty fast, the opposite is true for engagement. It actually appears that, at least on my site, there is a sweet spot for visitor engagement between about 40 and 50 sessions … heck, I even sold a few books to folks well after their initial visit once their engagement ran up to over 55 percent!

The nice thing about the engagement metric is that it helps resolve the problem that Gary describes in his recent post on visitor classification. Gary, in talking about the need to capture and visualize both absolute and relative content usage on a site says this:

The problem is that heavily engaged users of your site will show up (and often drive the statistics for) virtually every area of your site. For publishing clients, a small segment of heavily engaged users inevitably show up in every single content area. And the smaller the overall usage of that area, the more the heavily engaged component influences the results.

Yep, so wouldn’t it be nice if you could not only create on-the-fly visitor segments that are inclusive of any different number of content areas and pages on your site plus easily determine how much of an influence highly engaged visitors are on your absolute content usage measurements? If you could do that, it would probably look something like this:

I know it’s hard to see, but I simply dragged a bunch of pages, groups of pages, and content groups onto the page visualization map and told Visual Site to color the nodes by visitor engagement (the height of the bars represents the relative number of sessions to each node.) I could then select-in or select-out visitors based on their relative level of engagement to identify the special kinds of customers Gary refers to.

Anyway, I’m going to have beers with the good Mr. Ivy next week and I didn’t want that whole “social media” thing hanging over my head. And while I recognize that this metric (which I still have yet to share the calculation) doesn’t capture fully the elaborate needs of the really smart folks working to pound out Social Media Measurement, I heartily agree with Clint’s friend Jeremiah Owyang when he says that “Social Media is about people. People connecting to other people to build better relationships, fostering communities and increasing collective knowledge” and “Measurement and Metrics are one way to help to tell the story of Social Media.”

Measurement and Metrics, indeed.

One of my favorite bloggers, Mr. Avinash Kaushik, offered up his web analytics predictions for 2007 in a recent post. At the end of his post he offered this:

In turn I’ll tag our esteemed world leader Mr. Peterson to offer up his top secret list of 2007 web analytics predictions (I am positive it will be super!).

When Avinash calls me “our esteemed world leader” I have to wonder if he’s drinking stronger egg nog than I am at this time of the year, but since Mr. Kaushik is so nice (and positive it will be super!) here goes:

1. The many smart companies that have invested already in technology and people will increasingly realize that without addressing their Web Analytics Business Process they will still under-appreciate the full value of their investment in web analytics.
2. Smart people will stop freaking out about how “Web 2.0″ is going to be measured and will begin to develop rational and reasonable models for tracking emerging Internet technologies and business models.
3. Those companies who have deployed sufficiently powerful applications to identify search marketing click-fraud, especially those paying for highly-competitive search terms, may not particularly like what they find.
4. Emetrics San Francisco will be widely proclaimed the “best ever” thanks to the combination of an interstellar line-up of speakers and a location that (finally!) says “Go out at night and PARTY!” [ Important: I have no knowledge about the speakers at Emetrics San Francisco, I just really trust Jim Sterne! ]
5. The web analytics analyst community will shake again, although not quite the same way it did when Bob Chatham and I both joined Visual Sciences within a month of each other.

One Bonus:

1. Avinash, hot off the critical success of his new book, will finally accept his “guru” status in our community and stop being so damn humble all the time. Personally, I want to hear Mr. Kaushik say, “Damn straight, I’m the man!” on stage in San Francisco … Who’s with me?

Damn, it took me waaaaaaaaay longer than six minutes to write five not-so-surprising predictions. I guess I didn’t want to limit my career quite as much as my friend Avinash …

Since we’re still playing games, I’m going to go ahead and tag Mr. Gary Angel at SEMphonic, someone who has always cut straight to the chase, and see if he’ll offer up his own web analytics predictions for 2007. Gary?

Given that my last post on measuring engagement generated a fair amount of feedback, I wanted to follow-up with the post that in retrospect I should have published first, the nuts and bolts behind the engagement calculation.

Since there are numerous definitions of “engagement” that could be applied to the online channel, I choose to use the following definition:

Engagement is an estimate of the degree and depth of visitor interaction on the site against a clearly defined set of goals.

My definition sounds like conversion rate except engagement is a more flexible concept, one that can accommodate a variety of business needs such as those described by Bill Gassman in his comments to my last post and those of Craig Danuloff who is looking for a metric that accommodates a variety of visitor activities.

Based on my knowledge of my site visitors and their long-term usage patterns, my engagement goals are as follows:

1. I would like that visitors would view and interact with certain content on my site;
2. I would like visitors to subscribe to this weblog to stay connected;
3. I would like visitors to maintain a low recency with my content, regardless of whether they’re reading blog posts or viewing pages on my site;
4. When visitors are on my web site, I would like them to spend a reasonable amount of time interacting with my content;
5. When visitors return to my site, I prefer they remember my domain name and return to my site directly, either via a bookmark or by directly entering my URL into their browser.

Now, many of you will likely argue with these criteria, and fairly so. It’s fine that you may have a different definition of engagement for your site; I think that Bill put it best when he commented:

“Each organization’s version of engagement will be unique. It will be derived from a number of root metrics, probably under a dozen. Common root metrics will be frequency, recency, length of visit, purchases and lifetime value. Some organizations may include visitor actions, such as subscribing, providing personal information, writing a comment, or participating in a blog.”

I’m using the criteria I listed above based on my knowledge of my site visitors, mined from a variety of channels including site activity, email, comments, personal conversations, etc., juxtaposed against my site’s business objectives (see below.) Given a sufficiently flexible analytics package you can build your engagement metric using any goals you like …

Regarding item #1 in the list above, wanting visitors to interact with certain content on my site, here are the activities I am tracking broken down by moderate- and high-value:

Moderate-Value Activities

• Read my weblog
• Read about the Web Analytics Business Process
• Research web analytics jobs
• Add a link to my link database
• Read comments about my books
• Give me an email address
• Host a Web Analytics Wednesday
• Join the Web 2.0 Measurement Working Group

High-Value Activities

• Consider buying one or more of my books
• Buy one or more of my books
• Read about any of my books
• Read about my Key Performance Indicator Worksheets
• Download a sample copy of one of my books
• Email me directly
• Submit a comment to my weblog
• Go to Amazon.com to check out my books

Because it is very difficult to know a visitor’s intent when they visit a web site, these activities are designed to allow me to examine the visitor not in the context of their intent but rather in the context of my site’s specific objectives. I maintain Web Analytics Demystified for three primary reasons:

1. To sell my books
2. To maintain my visibility in the web analytics field
3. To have a channel through which I can continue to contribute ideas to our community

You may argue that tracking a visitor’s interaction with specific contents is a poor measure of engagement given that visitors may be looking at an entirely different set of content and are intensely engaged … fair enough. But these lists represent the activities that visitors can perform on my web site that are in-line with my stated business objectives.

If highly-engaged visitors are interacting with some other content on my site, that would prompt me to reconsider that contents contribution to my engagement calculation and perhaps add it to one of the lists above. My belief is that any engagement estimate must take content consumption into account given that it is the content that drives visitor engagement in the first place.

This post is getting long so it’s clear I’ll need a “Part III” (and maybe a “Part IV”) but here is something tangible to chew on until I have time to post again. Based on my five business goals stated above, my engagement calculation is essentially this:

(Pct High-Value Content Consumption Sessions + Pct Moderate-Value Content Consumption Sessions + Blog Subscriber Reads per Session + Pct Recent Sessions + Pct “Long” Sessions + Pct Direct Sessions) / 6

I am calculating the percentage of sessions on a per-visitor basis and summing those percentages to generate an “engagement score” between 0.0 and 6.0. I convert this score to a percentage itself to make it easier to read and voila! I can apply my engagement metric to any dimension I am tracking in Visual Site.

By clearly defining my engagement goals and then systematically scoring visitors against that framework, I can build a metric that can be objectively applied regardless of whether visitors buy a book. I can apply my engagement estimate to any dimension I am tracking on my site, allowing me to discover patterns of visitor behavior that would not be obvious based on more traditional metrics such as conversion rate, session duration, or page view count.

Just so I don’t lose you, here is one of the visualizations I am using to better understand visitor engagement showing visitor engagement by percent of visitors by visitor city:

It’s hard to see with the scale I’m providing in this image but I can assure you that the long-tail is there. And sure, with my $50 book I’m unlikely to launch a geo-targeted marketing campaign in markets where visitors are, on average, twice as engaged as my site-wide population … but maybe you would!

Until next time, I welcome your comments and criticism.

Frank Faubert, who I referenced in my last post, wrote in and had this to say:

“I think you have a flaws in both your data collection, and in the way you are thinking about engagement. I have been subscribed to your RSS feed for quite a long time, and I read it (along with many others) on a daily basis. I also have been to your web site in the past 90 days, though admittedly I don’t visit very often — mostly just to look at the status of the current Web Analytics Wednesday events.

If asked, I would consider myself an engaged visitor of your site, as I consume all of the content that you are generating, on a daily basis. Given that you push the entirety of your post in your RSS feed, I have no need to visit your site directly. (And in fact, if you were to change this to only publish snippets of your posts via RSS to force me onto the site, I would unsubscribe.) Does the fact that I am very busy, yet I can leverage the technology of a good RSS aggregator to consume your content anyway, really make me any less engaged?”

Frank makes a few really good points so I wanted to clarify something. I chose Frank as an example in my last post since I know him to be a really good guy and someone who was likely to provide thoughtful feedback. Frank didn’t let me down.

In fact, at the present time, I can’t actually track Frank’s engagement score and tie it directly to him personally (via his email address). Here is a view of Frank’s activity on my web site over time:

As you can see, Frank must have deleted the cookie that tied his email address to his site activity back in late 2005. Fair enough, Frank–I sometimes delete cookies too!

But Frank raises valid points about whether his engagement with my RSS feeds should be counted as “engaged”, and he’s absolutely correct! I’m pretty sure that Frank is located in Waltham, MA based on his attendance at the Web Analytics Wednesday event that Akin Arikan (also of Unica) hosted there awhile back. Here is a quick snapshot of activity to my site and RSS feeds from Waltham over the last eight weeks:

Clearly a few folks in Waltham are visiting the site and reading the blog, and the average engagement for visitors from Waltham is about 18 percent higher than average.

But Frank raises another really excellent point, albeit indirectly, should visitors who are actively interacting with RSS or XML-based content get a “push” in their engagement score? I mean, based on his response, Frank is doing two of the six things I have identified as “most important” on my site from a content and activity perspective (more on that in my next post), so if I could uniquely identify Frank I could vet whether his score is correct based on his description of his interaction with my site.

In reality I only provided Bill and Frank as examples to show how ultimately engagement needs to take real people’s activities into consideration, at least as the metrics are being defined and worked out. The engagement metric is really designed to be applied to dimensions other than people given that at the individual level, well, cookie deletion happens and thusly problems like the one that Frank highlights occur. Now, I suppose if you could positively identify individuals every time based on a login or customer ID that would change.

I hope I didn’t offend Bill, Frank, Robbin, or any of the other folks listed in that table as having visited my site from time-to-time. If I did, please accept my most humble of apologies. I don’t usually do this type of analysis but it helped to prove a point.

Anyway, thanks for writing in Frank and I’m glad to see you’re still reading! Hopefully we’ll get to connect next time I’m in Waltham and please give my regards to Akin, Ed, and everyone on the Unica analytics product team.

My good friend Clint Ivy and I were talking awhile back and he asked me, “So what do you think about Scoble’s call for an engagement metric?” I said, “Huh?” since I had long since stopped reading Robert Scoble, but apparently he had rubbed Clint the wrong way.

Anyway, I had been working on a project for a customer and we had been talking about how to measure engagement on their web site. We’d gone round-and-round on ideas about what constitutes an “engaged” visitor and narrowed it down to a few key areas:

1. The visitor views “critical” content on the web site
2. The visitor has returned to the web site recently
3. The visitor returns directly to the web site some of the time
4. Some high percentage of the visitor’s sessions are “long” sessions
5. If available, the visitor is subscribed to at least one available site feed

So, with this in mind, visitors that are consuming content slowly and methodically and returning directly to the site are well-engaged. Visitors who have also subscribed to some type of “push” feed are more engaged, and even more so if they’ve returned to the site recently.

Sounds reasonable, doesn’t it?

Using this model, sites like Yahoo! and Digg will have very engaged visitors, whereas sites like mine will have slightly less engaged visitors. That also sounds reasonable, given that Yahoo! and Digg are social networks and Web Analytics Demystified is more or less a weblog, a geek hub, and a job board (in that order).

It turns out that my audience is, on the whole, 32.3 percent engaged.

Perhaps more importantly, visitors that I get from the following sources are engaged at the following rates:

You can see there that my friend Avinash is sending me pretty good folks but Avinash’s people are slightly less engaged with my site than the “average” visitor. That and they hopefully already have my books because pretty much none of them are buying Web Analytics Demystified or The Big Book of Key Performance Indicators from my site! They’re not even taking advantage of the great combo-offer I have on both books!

Perhaps most interesting and wonderful is that my engagement metric allows me to build wonderful visualizations like this scatter-plot to compare the volume of referred visitors to engagement in a way that more easy on the eyes than your basic table or line graph.

Most importantly, because I am running the industry’s most easy-to-use yet powerful web analytics application supporting multi-source and multi-channel data analysis, I can vet my engagement index against real people who have come or are coming to my web site!

Cool, huh?

So Bill Gassman from the Gartner Group is among the most engaged visitors I have (I am quite honored, Bill!) Bill is consuming the content I deem most important to creating a relationship with my visitors, he is subscribed to my weblog, he keeps coming back, his sessions are of reasonable length, and he comes directly to my site or feed over 2/3rds of the time.

Bill is nearly 54 percent engaged with my site, approaching twice the average!

Compare Bill to Frank Faubert from Sane Solutions. Frank is seeing all of the content I believe to be most important and he also is well retained. However, Frank is not subscribed to my RSS feed so most of the time he is getting to my content indirectly then only spending a short period of time reading that content. Moreover, Frank hasn’t been to my site in the last 90 days.

Frank is only 21 percent engaged with my site so I guess maybe he doesn’t like me now that I’m not an objective, third-party anymore.

I’m interested in your thoughts about my engagement metric. Do you think I’m using the right inputs? Or am I missing something critical to how this metric should be calculated? I’d love you input since I know that the folks reading my weblog are among the brightest in the web analytics industry …

Next Time: Being a big fan of “showing my work”, I’ll provide the calculations behind my engagement metric so that you can calculate your site’s engagement in the safety of your own home. My hope is that through your comments and criticism I’ll be able to refine this metric down to something that any vendor can implement and any practitioner can use.