Archive for 'Engagement'

The same guys that want you all to believe web analytics is easy has now declared that “Visitor engagement formulas are largely another fad, just like parachute pants and the Hollywood diet. It’s a measure some consultants and vendors can pitch like snake oil.”

Omniture’s point that Visitor Engagement is a bad idea because it has subjective components fails to understand the work that folks like Jim Novo, Steve Jackson, Theo Papadakis, Joseph Carrabis and others have done; it makes me wonder if the author bothered to read anyone’s work on the subject.

Worse, it makes me question Omniture’s long-term commitment to Visual Site customers since Visual (= Omniture Discover OnPremise) is, at least for now, the industry’s leading solution for creating derived measures and experimenting with visitor-level data.  The point seems to be that simple measures of success, such as those provided by SiteCatalyst, are all that are required.

Hmmm …

We pretty much had this same debate a year ago when Avinash Kaushik disagreed with the use of calculated metrics to measure engagement, and I can see that Steve Jackson has already commented as such.  I wouldn’t normally have written about this except the author said one smart thing when he commented you shouldn’t “try to build a better mouse trap, when you’re not taking advantage of the one you’ve got today.”

Agreed.

If you’re thinking about trying to leverage any measure of visitor engagement, regardless of which measure you choose, you should definitely make sure your web analytics house is in order first.  Despite Omniture’s assertion, most people believe that web analytics is hard and requires a sometimes intense focus on people, process, and technology.  If you’re not staffed appropriately, if you haven’t defined your key performance indicators, if you haven’t established core web analytics business processes, and if you haven’t worked to optimize your web analytics implementation then trust me, Visitor Engagement is not for you.

A good analogy is the one provided in Tom Davenport’s book “Competing on Analytics” where he describes how baseball teams like the Oakland A’s and my friend Judah’s beloved Boston Red Sox, and football teams like the New England Patriots have used new and innovative metrics to evaluate the performance of players, concessions workers, and the entire fan experience.  Visitor Engagement is a new measure in web analytics, and thusly it will take a special type of analytical competitor to recognize the opportunity that this “uber measure” can potentially provide.  And just like some teams have shown that they are not ready to adopt new measures to evaluate their business, some companies are simply not ready to explore complex key performance indicators in an effort to “Compete on Web Analytics.”

If you’re like most companies doing web analytics today, it is likely that you will benefit more from focusing internally and learning more about how to leverage people, process, and technology more effectively, rather than look externally for new metrics of success.  You could get a good book on the subject of fundamental key performance indicators and spend a great deal of time implementing what you learn.

But if you’re interested in learning more about an innovative metric that describes the behavior and opportunity that exists with the 97% that don’t convert, a measure that you can apply to your advertising, content, B2B, marketing, or lead generation site that will compliment your otherwise robust key performance indicator suite, and a calculation that describes the level of Attention that visitors are paying to your site, your content, your testing and targeting, etc… well then I guess you’ll have to keep reading my blog (and Jim, and Steve, and Joseph, and a whole host of other people’s work who are committed to working these ideas out rather than just saying “balderdash!”)

If you’re not content to just keep reading and want to know more about my thoughts on Visitor Engagement, know this: I have been exceedingly clear that my measures of Visitor and Audience Engagement are new, and in their newness there is risk in the level of insight they may be able to provide you.  I am not promising you better skin, new hair, or more friends, despite the validation that the measurement of engagement recently received when NextStage was granted a patent for their work on the subject.  But, unlike some people, I have done my homework on the subject, and I continue to have conversations with some of the best companies in the world about how they can use new measures to improve their overall use of web analytic technology.

In the meantime, I guess I’ll put on my parachute pants, grab a glass of “Miracle juice”, and bust out the ol’ Snake Oil.

Since I am on the record as being supportive of the web analytics community around the world I wanted to make European readers aware of a webcast I will be doing next week.  Coremetrics has asked me to reprise the presentation I did on measuring visitor engagement that I did at their client summit last Fall.  The good news (for Europeans) is that the webcast is open to everyone and will be presented at 10 AM London time!

You can register for this free event at WebEx.

For those of you who can’t make the event because of holiday, or because like me the presentation will happen while you would normally be asleep, I am told that Coremetrics will be recording the presentation.  Assuming I am coherent at 2 AM my time and the recording comes of well, you should be able to download the webcast within the next few weeks at the Coremetrics web site.

I hope those of you in the European web analytics community will be able to join us next week.  I will try and leave plenty of time for questions and answers as well.

Following up on my post from Monday of this week announcing that Joseph Carrabis of NextStage Evolution will be joining “The Engagement Project” and bringing his mathematical expertise to the table, Mr. Carrabis has summarized what he’ll initially be doing for the chef in all of us.

According to Mr. Carrabis:

“Eric’s already posted that I’ll be working with him to make the formula more applicable to a wider variety of interfaces with greater general use features. I also know that I can always use help and have repeatedly and publicly stated that I don’t know web analytics.

So, first steps? A semantically exact statement of what we’re hoping to measure. I suggest this step because it’s much easier to know if your variables will result in the desired solution if you are exact in what the solution looks like and what you have to put into that solution.

Think of it this way; You want to make some chicken soup and you use your grandmother’s recipe. I want to make some chicken soup and I use my grandmother’s recipe. But your grandmother is Irish and mine is Italian. I’ll bet we’d use different spices, different vegetables, different noodles (if indeed we both did).

But I’d bet we both use chicken stock as a base. And is your chicken stock from the leftovers of a roast chicken? What spices did you use there? Or is your stock from bullion?

So the first step is to decide what we all mean by “chicken soup”. One of my mentors was a genius of an author who use to write “speculative fiction”. I would ask, “What is speculative fiction?” and he’d reply “It’s what I’m pointing at when I say it.” This is a great anecdote and an undefensible statement (except in cultural anthropology). If one person “owns” the definition of “speculative fiction”, “chicken soup” or “engagement” then that definition is only valid so long as there exists a market for that definition.

However, a definition that says something like “Basic Chicken Soup”, that is something I can start with to make “Italian Chicken Soup” and allows my Irish friend to extend it to “Irish Chicken Soup”? Now that’s a good definition.

I snuck the concept of “extendable” into the above. “Extendable” means the definition accommodates special cases (Italian, Irish, etc). Think of a recipe for Italian Chicken Soup that begins “Step 1: Make the Basic Chicken Soup. Step 2: Now add garlic, oregano, …” That “Step 2″ part means that the original definition isn’t limited, that it can be extended to incorporate specific features to make it unique to a given environment (Italian, Irish, …).

The concept of “extensible” has two parts; First, you can substitute one thing for another if they share some basic properties. For example, you can substitute a glass of wine for a glass of water in the recipe because they’re both liquids. You can’t substitute a lamb chop for a glass of water, though. Mathematically, this means that if we want to include “clickthroughs” we can use whatever product A calls clickthroughs, whatever product B calls clickthroughs, etc., so long as they all meet some definition of “clickthroughs” (I’ll let the WAA worry about things like that).

Second, “extensible” means new spices, new vegetables, new types of noodles, etc., can be used to make the chicken soup better. This means that you can add a new spice to your recipe in addition to the existing spices already in it. Extensible (in this sense) means you’re doing what you already do to make your style chicken soup and now you’ve discovered something more you can add to it to make even more “your style”. You’re not watering it down or adding more vegetables to make the soup go further. That’s scalability and the equation should be scalable without needing to define it as such.

The sum of these two concepts of “extensible” translates to “the equation is valid across all interfaces including those we haven’t thought of yet.” Mathematically extendability and extensibility form the axes of a very rich solution space.”

Joseph says “Basic Chicken Soup” and I say “a measure of the depth and degree of visitor engagement online” … clearly he and I both have our work cut out for us. If you’d like to join us in our quest for a better measure of visitor engagement online, please let me know.

In the last few months there has been a tremendous surge in interest in my framework for measuring engagement online. Lately, some of the largest and well-known companies in the world have approached me about working with them to bridge the gap between the metrics they have today and something similar to the composite metric I first described back in December 2006.

While I am tremendously flattered that I have somehow become the focal point for this conversation, I have been thinking lately about how the framework has been developed and how it might end up being used by the measurement industry in general. And while early tests using the framework I’ve described are very encouraging, the calculation in it’s current state was meant to move the discussion along and get more people to “think different” about how engagement could be calculated online.

Given that interest in the framework has clearly increased, one primary concern comes up again and again: the need to apply mathematical rigor to the framework and calculation so that A) the result is repeatable, reliable, and trustworthy and B) when naysayers inevitably emerge to criticize this small side project of mine, that I have a suitable response to their criticism, regardless of where and why it comes.

I believe that the need for “A” is obvious. The need to address “B” is perhaps less obvious, but I believe that I owe it to those of you who are investing your time, energy, and money into this framework. Especially as the stakes seem to increase exponentially with every presentation, every conversation, and every high-visibility blog post on the subject, I believe now is the time to approach the engagement framework not just as a hobby but as a serious project with committed resources.

To this end, I am extraordinarily happy to say that the single smartest person I know, Joseph Carrabis the Founder and Chief Research Officer of NextStage Evolution and NextStage Global, has offered to bring mathematical rigor and analytical precision to what I am officially dubbing “The Engagement Project.” Those of you not familiar with Joseph and his work are advised to A) meet him in person at the upcoming Emetrics Summit in San Francisco or B) read some of his recent work at iMedia Connection.

Joseph will be working to make the formula universally applicable and universally defensible. Suffice to say I can think of nobody better to bring mathematical and scientific rigor to the framework I have been evolving over the past year. Watch this blog and Joseph’s blog at BizMediaScience over the next week or so for a more complete analysis of the framework in it’s current state, something we’ve agreed is the first step towards creating a true function capable practically describing the degree and depth of engagement a visitor is displaying towards a web site over time.

At the end of the day, without regard to my framework, Joseph’s analysis, or any person or group’s particular position on the use of the word “engagement”, my goal is to solve one problem and one problem only:

If you’re interested in working with Joseph and me on The Engagement Project please feel free to contact me directly.

My post last week on measuring visitor engagement was pretty long by the time I outlined the calculation, so I put off publishing examples of how the metric could be used until now. I’m excited to see that this topic has generated so much interest, both in terms of comments and emails sent to me directly.

My goal for this post is to provide a few examples and explanations to show how the metric can be used to supplement our otherwise already-rich set of web analytics data. Since so many folks have been willing to explore the engagement metric, I have embedded a bunch of questions in this post in italics that I’d love your feedback on.

Distribution of engagement scores and segmentation. Here is the distribution of engagement scores for about six months at Web Analytics Demystified by percent of visitors. As you can see, these scores are left-skewed and tail off as the score increases, showing that nearly half (47.6%) of visitors to my site are “poorly engaged”. When I look at this distribution it makes perfect sense to me — what do you think?

I have created segments to group visitors by their engagement score: “Well engaged” visitors have engagement scores over 30%, “moderately engaged” visitors are those between 10% and 30%, and”poorly engaged” visitors score less than 10%. These segments can then be used to explore how the behavior of visitors in each engagement group differs by looking at my page and referring source dimensions (page, content group, referring domain, campaign, search phrase, etc.)

Identify relationships that might otherwise not be found. At the top of this report you can see the pronounced difference in visitor engagement (and traditional metrics) for “branded” and unbranded searches (”None”) bringing visitors to my site. Now, because branded searches are a component of the calculation (Brand Index), you definitely expect to see a difference between the two engagement scores. What is interesting is that while other metrics (duration, sessions per visitor, page views per session) show a slight difference, visitor engagement and conversion are all three times higher for branded searches. I think this difference observed in all the metrics is further evidence that brand-driven searches are bringing more engaged visitors — what do you think?

In the middle table you can see search phrases bringing visitor to my site, showing visitor engagement, page views per session, and sessions per visitor. Here three phrases stand out to me:

1. “web analytics book” and “web analytics process”, neither of which are particularly distinguished from other search phrases based on page views per session or sessions per visitor but both of which have visitor engagement scores over double my site-wide average of 8.8%. This is important to me because these are un-branded search terms that are critically important to my business.
2. “vendor discovery tool” which would appear to be pretty important based on traditional metrics but only stands out slightly using the visitor engagement score (at 13.6%) I spend a lot of time trying to figure out how to drive folks using the vendor discovery tool to take other actions (buy books, inquire about consulting) and this data suggests that there is an unrealized opportunity.
3. “performance indicators” which shows that the visitor engagement metric is useful to identify terms that you’d think are important to the site but aren’t attracting the right audience (average engagement score for these visitors is only 5.6%)

I think this level of information is actually pretty helpful for identifying search marketing opportunities — what do you think?

Engagement-derivative metrics like “Percent Highly Engaged Visitors” are useful. Here you can see a select group of referring domains showing the percent of highly and percent moderately engaged visitors they’re sending my way (with conversion to show that engagement and conversion are in fact different!) Avinash Kaushik is sending me a few (0.2%) highly engaged visitors (thanks!) but Ian Thomas is sending me a bunch (70.4%) of moderately engaged visitors, many of whom are purchasing books (1.2% conversion rate.)

By looking at traffic from Avinash’s site over time (bar graph) I can see peaks and valleys in overall engagement from folks coming from his site, which would be useful to back into those peaks to try and determine what other blogger’s readers might be reacting to when they’re exhibiting highly-engaged behavior on my site (see late August and early September.) Given that Clint proved that conversion is a poor measure of success when trying to evaluate traffic from other bloggers, I think visitor engagement is useful for examining the non-revenue value of referring sources — what do you think?

Those of you who are looking for correlation between engagement and conversion, have a look at the data for Mr. Jim Sterne’s wonderful site emetrics.org —  5.6% of the folks coming from Jim’s site are highly engaged, 66.2% moderately engaged, and man-oh-man does Jim help sell some copies of Web Analytics Demystified.  You’re the man, Jim!

Visitor engagement is globally useful. At least in Visual Sciences Visual Site you can apply engagement metrics and segments to pretty much any dimension tracked. Here I’m looking at the percentage of “highly” engaged visitors (50% or more) in my “well engaged” segment broken down by country. Now, this is certainly more interesting in light of the total volume of traffic coming from each geographic location, and as I think about localizing my books and planning future trips around the world this information becomes very helpful.

There is more, including some of the more granular visitor-level stuff I talked about in the first series of posts on the subject, but I want to be sensitive to protecting the identity of individual users on my site. If you’re interested in helping me collect some “ground truth” regarding the engagement calculation, write me and I’ll explain how you can help.

So what do you think? Do the screen-shots help you understand the calculation better? Or do they still make it look super-complicated and scary? Is there something specific you’d like to see me demonstrate with the calculation? Or do you think you could come up with these same insights using more traditional metrics?