Archive for December, 2006

One of my favorite bloggers, Mr. Avinash Kaushik, offered up his web analytics predictions for 2007 in a recent post. At the end of his post he offered this:

In turn I’ll tag our esteemed world leader Mr. Peterson to offer up his top secret list of 2007 web analytics predictions (I am positive it will be super!).

When Avinash calls me “our esteemed world leader” I have to wonder if he’s drinking stronger egg nog than I am at this time of the year, but since Mr. Kaushik is so nice (and positive it will be super!) here goes:

1. The many smart companies that have invested already in technology and people will increasingly realize that without addressing their Web Analytics Business Process they will still under-appreciate the full value of their investment in web analytics.
2. Smart people will stop freaking out about how “Web 2.0″ is going to be measured and will begin to develop rational and reasonable models for tracking emerging Internet technologies and business models.
3. Those companies who have deployed sufficiently powerful applications to identify search marketing click-fraud, especially those paying for highly-competitive search terms, may not particularly like what they find.
4. Emetrics San Francisco will be widely proclaimed the “best ever” thanks to the combination of an interstellar line-up of speakers and a location that (finally!) says “Go out at night and PARTY!” [ Important: I have no knowledge about the speakers at Emetrics San Francisco, I just really trust Jim Sterne! ]
5. The web analytics analyst community will shake again, although not quite the same way it did when Bob Chatham and I both joined Visual Sciences within a month of each other.

One Bonus:

1. Avinash, hot off the critical success of his new book, will finally accept his “guru” status in our community and stop being so damn humble all the time. Personally, I want to hear Mr. Kaushik say, “Damn straight, I’m the man!” on stage in San Francisco … Who’s with me?

Damn, it took me waaaaaaaaay longer than six minutes to write five not-so-surprising predictions. I guess I didn’t want to limit my career quite as much as my friend Avinash …

Since we’re still playing games, I’m going to go ahead and tag Mr. Gary Angel at SEMphonic, someone who has always cut straight to the chase, and see if he’ll offer up his own web analytics predictions for 2007. Gary?

For Christmas this year, Gary Angel from SEMphonic got me … work! Gary tagged me as part of a blogger’s chain letter that has been going around, kindly noting that he thinks of me as a “kindred spirit” which I take as a great compliment.

Not wanting to be “that guy” who gets a nice compliment and then fails to respond, here are five things you may not know about me:

1. I have two beautiful children, one girl, one boy, for whom I would do anything. I consider myself incredibly lucky to work for a company that is supportive of my need to work from home and actively participate in my kid’s lives, especially when they’re this young.
2. I hold a Masters Degree in Botany & Plant Pathology from Oregon State University (GO BEAVS!) My Master’s thesis was titled “Systematics of the genus Otidea in the Pacific Northwest” Don’t know the ascomycete genus Otidea? You’re kidding me. Really!?
3. For most of my college years I played drums in a “psychedelic bluegrass” band called The Sugar Beets. We played throughout the Northwest, often at large festivals, which contributed to my willingness to present to large audiences. I mean hey, after you’ve dropped a drum stick in front of 5,000 people, talking about web analytics is a piece of cake!
4. To get around in Portland, I drive one of these, except when the weather is really nice (three days a year in Oregon) when I ride one of these. I owe my small vehicle collection entirely to 1) a very understanding wife and 2) all the nice folks who bought copies of Web Analytics Demystified in 2004 and 2005. THANKS!
5. After high school I wasn’t as focused as many of my friends so instead of going straight to college I followed the Grateful Dead around the country on-and-off for two years. You really learn something about yourself when you have to sell organic bagels with cream cheese and sprouts to afford gas to get to the next town … What a long, strange trip that was!

There you have it! Per the rules of the “game”, I now tag five other bloggers that I enjoy:

1. Xavier Casanova, former co-founder of Fireclick and current CEO at Perenety. Xavier was an inspiration to me as I wrote Web Analytics Demystified, he shares my taste in cars, and he is one of the brightest guys I know.
2. Ian Thomas from Microsoft, who I share an interest in cookie deletion and who turned me on to the comics of Jessica Hagy.
3. A. Clinton Ivy, a co-worker of mine and someone who never hesitates to tell me I’m wrong, regardless of whether I’m wrong or not.
4. Eric Butler from WebTrends, a former co-worker of mine and a great advocate for both his company’s technology and the web analytics space in general.
5. Chris Boring, a friend from waaaaaaay back in the day who has always been one of the strongest people I know. Chris is currently fighting Acute Lymphoblastic Leukemia (ALL) and has been blogging about his and his partner Beth’s experience.

Finally, I want to wish everyone who reads my weblog the happiest of holidays. Regardless of what you celebrate or where you are in the world, I sincerely hope that your year ends in a relaxing and enjoyable fashion and you’re charged and ready to go in 2007!

Given that my last post on measuring engagement generated a fair amount of feedback, I wanted to follow-up with the post that in retrospect I should have published first, the nuts and bolts behind the engagement calculation.

Since there are numerous definitions of “engagement” that could be applied to the online channel, I choose to use the following definition:

Engagement is an estimate of the degree and depth of visitor interaction on the site against a clearly defined set of goals.

My definition sounds like conversion rate except engagement is a more flexible concept, one that can accommodate a variety of business needs such as those described by Bill Gassman in his comments to my last post and those of Craig Danuloff who is looking for a metric that accommodates a variety of visitor activities.

Based on my knowledge of my site visitors and their long-term usage patterns, my engagement goals are as follows:

1. I would like that visitors would view and interact with certain content on my site;
2. I would like visitors to subscribe to this weblog to stay connected;
3. I would like visitors to maintain a low recency with my content, regardless of whether they’re reading blog posts or viewing pages on my site;
4. When visitors are on my web site, I would like them to spend a reasonable amount of time interacting with my content;
5. When visitors return to my site, I prefer they remember my domain name and return to my site directly, either via a bookmark or by directly entering my URL into their browser.

Now, many of you will likely argue with these criteria, and fairly so. It’s fine that you may have a different definition of engagement for your site; I think that Bill put it best when he commented:

“Each organization’s version of engagement will be unique. It will be derived from a number of root metrics, probably under a dozen. Common root metrics will be frequency, recency, length of visit, purchases and lifetime value. Some organizations may include visitor actions, such as subscribing, providing personal information, writing a comment, or participating in a blog.”

I’m using the criteria I listed above based on my knowledge of my site visitors, mined from a variety of channels including site activity, email, comments, personal conversations, etc., juxtaposed against my site’s business objectives (see below.) Given a sufficiently flexible analytics package you can build your engagement metric using any goals you like …

Regarding item #1 in the list above, wanting visitors to interact with certain content on my site, here are the activities I am tracking broken down by moderate- and high-value:

Moderate-Value Activities

• Read my weblog
• Read about the Web Analytics Business Process
• Research web analytics jobs
• Add a link to my link database
• Read comments about my books
• Give me an email address
• Host a Web Analytics Wednesday
• Join the Web 2.0 Measurement Working Group

High-Value Activities

• Consider buying one or more of my books
• Buy one or more of my books
• Read about any of my books
• Read about my Key Performance Indicator Worksheets
• Download a sample copy of one of my books
• Email me directly
• Submit a comment to my weblog
• Go to Amazon.com to check out my books

Because it is very difficult to know a visitor’s intent when they visit a web site, these activities are designed to allow me to examine the visitor not in the context of their intent but rather in the context of my site’s specific objectives. I maintain Web Analytics Demystified for three primary reasons:

1. To sell my books
2. To maintain my visibility in the web analytics field
3. To have a channel through which I can continue to contribute ideas to our community

You may argue that tracking a visitor’s interaction with specific contents is a poor measure of engagement given that visitors may be looking at an entirely different set of content and are intensely engaged … fair enough. But these lists represent the activities that visitors can perform on my web site that are in-line with my stated business objectives.

If highly-engaged visitors are interacting with some other content on my site, that would prompt me to reconsider that contents contribution to my engagement calculation and perhaps add it to one of the lists above. My belief is that any engagement estimate must take content consumption into account given that it is the content that drives visitor engagement in the first place.

This post is getting long so it’s clear I’ll need a “Part III” (and maybe a “Part IV”) but here is something tangible to chew on until I have time to post again. Based on my five business goals stated above, my engagement calculation is essentially this:

(Pct High-Value Content Consumption Sessions + Pct Moderate-Value Content Consumption Sessions + Blog Subscriber Reads per Session + Pct Recent Sessions + Pct “Long” Sessions + Pct Direct Sessions) / 6

I am calculating the percentage of sessions on a per-visitor basis and summing those percentages to generate an “engagement score” between 0.0 and 6.0. I convert this score to a percentage itself to make it easier to read and voila! I can apply my engagement metric to any dimension I am tracking in Visual Site.

By clearly defining my engagement goals and then systematically scoring visitors against that framework, I can build a metric that can be objectively applied regardless of whether visitors buy a book. I can apply my engagement estimate to any dimension I am tracking on my site, allowing me to discover patterns of visitor behavior that would not be obvious based on more traditional metrics such as conversion rate, session duration, or page view count.

Just so I don’t lose you, here is one of the visualizations I am using to better understand visitor engagement showing visitor engagement by percent of visitors by visitor city:

It’s hard to see with the scale I’m providing in this image but I can assure you that the long-tail is there. And sure, with my $50 book I’m unlikely to launch a geo-targeted marketing campaign in markets where visitors are, on average, twice as engaged as my site-wide population … but maybe you would!

Until next time, I welcome your comments and criticism.

Frank Faubert, who I referenced in my last post, wrote in and had this to say:

“I think you have a flaws in both your data collection, and in the way you are thinking about engagement. I have been subscribed to your RSS feed for quite a long time, and I read it (along with many others) on a daily basis. I also have been to your web site in the past 90 days, though admittedly I don’t visit very often — mostly just to look at the status of the current Web Analytics Wednesday events.

If asked, I would consider myself an engaged visitor of your site, as I consume all of the content that you are generating, on a daily basis. Given that you push the entirety of your post in your RSS feed, I have no need to visit your site directly. (And in fact, if you were to change this to only publish snippets of your posts via RSS to force me onto the site, I would unsubscribe.) Does the fact that I am very busy, yet I can leverage the technology of a good RSS aggregator to consume your content anyway, really make me any less engaged?”

Frank makes a few really good points so I wanted to clarify something. I chose Frank as an example in my last post since I know him to be a really good guy and someone who was likely to provide thoughtful feedback. Frank didn’t let me down.

In fact, at the present time, I can’t actually track Frank’s engagement score and tie it directly to him personally (via his email address). Here is a view of Frank’s activity on my web site over time:

As you can see, Frank must have deleted the cookie that tied his email address to his site activity back in late 2005. Fair enough, Frank–I sometimes delete cookies too!

But Frank raises valid points about whether his engagement with my RSS feeds should be counted as “engaged”, and he’s absolutely correct! I’m pretty sure that Frank is located in Waltham, MA based on his attendance at the Web Analytics Wednesday event that Akin Arikan (also of Unica) hosted there awhile back. Here is a quick snapshot of activity to my site and RSS feeds from Waltham over the last eight weeks:

Clearly a few folks in Waltham are visiting the site and reading the blog, and the average engagement for visitors from Waltham is about 18 percent higher than average.

But Frank raises another really excellent point, albeit indirectly, should visitors who are actively interacting with RSS or XML-based content get a “push” in their engagement score? I mean, based on his response, Frank is doing two of the six things I have identified as “most important” on my site from a content and activity perspective (more on that in my next post), so if I could uniquely identify Frank I could vet whether his score is correct based on his description of his interaction with my site.

In reality I only provided Bill and Frank as examples to show how ultimately engagement needs to take real people’s activities into consideration, at least as the metrics are being defined and worked out. The engagement metric is really designed to be applied to dimensions other than people given that at the individual level, well, cookie deletion happens and thusly problems like the one that Frank highlights occur. Now, I suppose if you could positively identify individuals every time based on a login or customer ID that would change.

I hope I didn’t offend Bill, Frank, Robbin, or any of the other folks listed in that table as having visited my site from time-to-time. If I did, please accept my most humble of apologies. I don’t usually do this type of analysis but it helped to prove a point.

Anyway, thanks for writing in Frank and I’m glad to see you’re still reading! Hopefully we’ll get to connect next time I’m in Waltham and please give my regards to Akin, Ed, and everyone on the Unica analytics product team.

My good friend Clint Ivy and I were talking awhile back and he asked me, “So what do you think about Scoble’s call for an engagement metric?” I said, “Huh?” since I had long since stopped reading Robert Scoble, but apparently he had rubbed Clint the wrong way.

Anyway, I had been working on a project for a customer and we had been talking about how to measure engagement on their web site. We’d gone round-and-round on ideas about what constitutes an “engaged” visitor and narrowed it down to a few key areas:

1. The visitor views “critical” content on the web site
2. The visitor has returned to the web site recently
3. The visitor returns directly to the web site some of the time
4. Some high percentage of the visitor’s sessions are “long” sessions
5. If available, the visitor is subscribed to at least one available site feed

So, with this in mind, visitors that are consuming content slowly and methodically and returning directly to the site are well-engaged. Visitors who have also subscribed to some type of “push” feed are more engaged, and even more so if they’ve returned to the site recently.

Sounds reasonable, doesn’t it?

Using this model, sites like Yahoo! and Digg will have very engaged visitors, whereas sites like mine will have slightly less engaged visitors. That also sounds reasonable, given that Yahoo! and Digg are social networks and Web Analytics Demystified is more or less a weblog, a geek hub, and a job board (in that order).

It turns out that my audience is, on the whole, 32.3 percent engaged.

Perhaps more importantly, visitors that I get from the following sources are engaged at the following rates:

You can see there that my friend Avinash is sending me pretty good folks but Avinash’s people are slightly less engaged with my site than the “average” visitor. That and they hopefully already have my books because pretty much none of them are buying Web Analytics Demystified or The Big Book of Key Performance Indicators from my site! They’re not even taking advantage of the great combo-offer I have on both books!

Perhaps most interesting and wonderful is that my engagement metric allows me to build wonderful visualizations like this scatter-plot to compare the volume of referred visitors to engagement in a way that more easy on the eyes than your basic table or line graph.

Most importantly, because I am running the industry’s most easy-to-use yet powerful web analytics application supporting multi-source and multi-channel data analysis, I can vet my engagement index against real people who have come or are coming to my web site!

Cool, huh?

So Bill Gassman from the Gartner Group is among the most engaged visitors I have (I am quite honored, Bill!) Bill is consuming the content I deem most important to creating a relationship with my visitors, he is subscribed to my weblog, he keeps coming back, his sessions are of reasonable length, and he comes directly to my site or feed over 2/3rds of the time.

Bill is nearly 54 percent engaged with my site, approaching twice the average!

Compare Bill to Frank Faubert from Sane Solutions. Frank is seeing all of the content I believe to be most important and he also is well retained. However, Frank is not subscribed to my RSS feed so most of the time he is getting to my content indirectly then only spending a short period of time reading that content. Moreover, Frank hasn’t been to my site in the last 90 days.

Frank is only 21 percent engaged with my site so I guess maybe he doesn’t like me now that I’m not an objective, third-party anymore.

I’m interested in your thoughts about my engagement metric. Do you think I’m using the right inputs? Or am I missing something critical to how this metric should be calculated? I’d love you input since I know that the folks reading my weblog are among the brightest in the web analytics industry …

Next Time: Being a big fan of “showing my work”, I’ll provide the calculations behind my engagement metric so that you can calculate your site’s engagement in the safety of your own home. My hope is that through your comments and criticism I’ll be able to refine this metric down to something that any vendor can implement and any practitioner can use.