Archive for 'Engagement'

I have had a ton of great feedback about the white paper Joseph Carrabis and I wrote on Web Analytics Demystified’s measure of Visitor Engagement.  Some folks have raised very good questions and I wanted to provide some of the answers to those questions here to better socialize the knowledge.

The first question came from Jonny Longden who asked:

“I was wondering if you could clear something up for me regarding your visitor engagement white paper? It is to do with the way the equation is stated:

Σ(Ci + Di + Ri + Li + Bi + Fi + Ii)

Apologies if this is me being unintelligent (I am not a mathematician), but the way I read this is that the result of the equation is the sum of the 7 different index values. However, in your examples on page 32 the VE appears to the be the average of those 7 values. Am I missing something?”

An excellent question and one that several other people asked.  When I wrote the equation I was looking for the simplest possible way to represent the relationship between the indices.  I overdid that and Jonny’s question points that out.  Technically, what I should have written is something like this:

VISITOR ( SESSION ( SUM(Ci + Di + Ri + Li + Bi + Fi + Ii) / 7 ))

Indicating that for every visitor in the set, the sum of indicies based on each visitor’s session needs to be divided by the total number of indices, which in this equation is seven.  The reason I recommend dividing by seven is that the resulting number will be between 0.00 and 1.00 every time, yielding a nice, clean number that can be translated to a percentage for communication’s sake.

My next question came from Nikolay Gradinarov who asked:

“Have you considered weighing the different indexes that are part of the Visitor Engagement calculation?”

Yes, I have considered weighting the different indices, as does nearly everyone who looks at the calculation.  The reason I don’t apply any weighting is that I don’t have any way to know what weighting to apply.  Put another way, since I don’t have another measure of engagement, I don’t have any basis for using weighting to correct components of the equation; thusly, at least to me, applying differential weighting to any of the indices seems contrived and likely to increase the complexity of explaining the calculation more than anything.

That said, those of you using the calculation are free to apply whatever weighting you like.  For example, if you didn’t have a good way to calculate the Feeback Index and wanted to exclude it from the calculation of Visitor Engagement, you would simply “zero weight” that index.  Or, if the HIPPO said that “duration is at least three times as important as anything else as a measure of engagement” then you could multiply the Duration Index value by three.

Keep in mind, relative to the last question, doing so will change the mathematics.  If you’re three weighting one index then you’ll either need to divide by 9 (i.e., six “1 weighted” indices plus a three weighted index) to get a value between 0.00 and 1.00 or understand that for some sets you’ll have a number greater than 1.00.

The final question I wanted to treat here comes from Elizabeth Robillard who asked, and I paraphrase since she sent quite the document, whether it was better to calculate the Recency Index using all of the sessions in the set or just the two most recent sessions.  Elizabeth’s point was that when she applied my Recency Calculation using all sessions and the two most recent sessions to a variety of made up situations she was confused by what the data was telling her regarding engagement.

The short answer is that Elizabeth (and any of you) can use whatever sessions you’d like when making the Recency Index calculation.  Again, if you choose to base your calculation on only the two most recent sessions then you’re functionally “zero weighting” the other sessions in the set, which is another assumption but one you’re free to make.

But the main point I would make here is that Elizabeth appears to be trying to examine a single component index and make a statement about the level of Visitor Engagement.  Tempting, I know, but not how the Visitor Engagement calculation is designed to be used.  The reason I have spent so much time evangelizing for/thinking about/understanding the mathematics behind/debating/etc. is that I believe that all of the component indices are required to understand the nuances of Visitor Engagement.

To better understand why I believe this, re-read the section on “Why a New Measure of Engagement” on pages 10 to 15.  Trying to determine the level of engagement of a visitor by looking only at the Recency Index is just like trying to make the same determination using nothing more then raw or average session duration data — I do not believe that a single measure or metric has the resolving power to determine the level of Attention that a visitor is paying to your web site (or whatever object you’re trying to measure.)

So you could have a very low recency between the two most recent sessions, yielding a Recency Index of 100% based on Elizabeth’s suggestion, but a 0% score for Click-Depth, Duration, Loyalty, Interaction, Brand, and Feedback … which would show a visitor who has been to your site twice recently but is otherwise paying no Attention.  Similarly, you could have someone who hasn’t been to the site in the last 30 days, yielding a Recency score of around 3% (1/30) but who had high scores for Click-Depth, Duration, Interaction, Brand, and Feedback and thusly appears to be paying Attention.

You can play these scenarios out all day long — trust me, I have done this again and again — but at the end of the day in my humble opinion no one index, metric, or measurement will provide you the same level of insight that these seven indices combined provides on a visitor-by-visitor basis.

Hopefully the answers to these questions are helpful to other’s of you reading the white paper.  Again, you can download the document freely from my web site and as always I welcome your feedback.

A lot of you have been following the thread in my blog about measures of engagement on the Internet. Over the past year we have certainly had a spirited discussion about the topic, and for the most part people’s interest in the subject has not apparently subsided. About six months ago I started working with Mr. Joseph Carrabis from NextStage Global on the engagement calculation and the byproduct of our work is now available as a somewhat lengthy white paper on the subject freely available to all.

You can download the white paper from the Research > Published Research section of this web site:

http://www.webanalyticsdemystified.com/link_list.asp?l=Research

The white paper includes a great deal of information about the calculation including background on it’s derivation, the calculation itself, it’s use in a business context, and the underlying mathematics.  I welcome your feedback on the paper and am more than happy to discuss the contents via phone or email.

The direct measure of a Visitor’s Engagement with a web site or set of properties is still a work in progress to be sure.  And despite some naysayers, I believe that all of us working with this or similar calculations are quite excited about the possibilities associated with moving on from more simple measures and beginning to combine metrics to create a more interesting (and potentially more valuable) view of visitor interaction on the Internet.

Let the debate begin again!

The same guys that want you all to believe web analytics is easy has now declared that “Visitor engagement formulas are largely another fad, just like parachute pants and the Hollywood diet. It’s a measure some consultants and vendors can pitch like snake oil.”

Omniture’s point that Visitor Engagement is a bad idea because it has subjective components fails to understand the work that folks like Jim Novo, Steve Jackson, Theo Papadakis, Joseph Carrabis and others have done; it makes me wonder if the author bothered to read anyone’s work on the subject.

Worse, it makes me question Omniture’s long-term commitment to Visual Site customers since Visual (= Omniture Discover OnPremise) is, at least for now, the industry’s leading solution for creating derived measures and experimenting with visitor-level data.  The point seems to be that simple measures of success, such as those provided by SiteCatalyst, are all that are required.

Hmmm …

We pretty much had this same debate a year ago when Avinash Kaushik disagreed with the use of calculated metrics to measure engagement, and I can see that Steve Jackson has already commented as such.  I wouldn’t normally have written about this except the author said one smart thing when he commented you shouldn’t “try to build a better mouse trap, when you’re not taking advantage of the one you’ve got today.”

Agreed.

If you’re thinking about trying to leverage any measure of visitor engagement, regardless of which measure you choose, you should definitely make sure your web analytics house is in order first.  Despite Omniture’s assertion, most people believe that web analytics is hard and requires a sometimes intense focus on people, process, and technology.  If you’re not staffed appropriately, if you haven’t defined your key performance indicators, if you haven’t established core web analytics business processes, and if you haven’t worked to optimize your web analytics implementation then trust me, Visitor Engagement is not for you.

A good analogy is the one provided in Tom Davenport’s book “Competing on Analytics” where he describes how baseball teams like the Oakland A’s and my friend Judah’s beloved Boston Red Sox, and football teams like the New England Patriots have used new and innovative metrics to evaluate the performance of players, concessions workers, and the entire fan experience.  Visitor Engagement is a new measure in web analytics, and thusly it will take a special type of analytical competitor to recognize the opportunity that this “uber measure” can potentially provide.  And just like some teams have shown that they are not ready to adopt new measures to evaluate their business, some companies are simply not ready to explore complex key performance indicators in an effort to “Compete on Web Analytics.”

If you’re like most companies doing web analytics today, it is likely that you will benefit more from focusing internally and learning more about how to leverage people, process, and technology more effectively, rather than look externally for new metrics of success.  You could get a good book on the subject of fundamental key performance indicators and spend a great deal of time implementing what you learn.

But if you’re interested in learning more about an innovative metric that describes the behavior and opportunity that exists with the 97% that don’t convert, a measure that you can apply to your advertising, content, B2B, marketing, or lead generation site that will compliment your otherwise robust key performance indicator suite, and a calculation that describes the level of Attention that visitors are paying to your site, your content, your testing and targeting, etc… well then I guess you’ll have to keep reading my blog (and Jim, and Steve, and Joseph, and a whole host of other people’s work who are committed to working these ideas out rather than just saying “balderdash!”)

If you’re not content to just keep reading and want to know more about my thoughts on Visitor Engagement, know this: I have been exceedingly clear that my measures of Visitor and Audience Engagement are new, and in their newness there is risk in the level of insight they may be able to provide you.  I am not promising you better skin, new hair, or more friends, despite the validation that the measurement of engagement recently received when NextStage was granted a patent for their work on the subject.  But, unlike some people, I have done my homework on the subject, and I continue to have conversations with some of the best companies in the world about how they can use new measures to improve their overall use of web analytic technology.

In the meantime, I guess I’ll put on my parachute pants, grab a glass of “Miracle juice”, and bust out the ol’ Snake Oil.

Since I am on the record as being supportive of the web analytics community around the world I wanted to make European readers aware of a webcast I will be doing next week.  Coremetrics has asked me to reprise the presentation I did on measuring visitor engagement that I did at their client summit last Fall.  The good news (for Europeans) is that the webcast is open to everyone and will be presented at 10 AM London time!

You can register for this free event at WebEx.

For those of you who can’t make the event because of holiday, or because like me the presentation will happen while you would normally be asleep, I am told that Coremetrics will be recording the presentation.  Assuming I am coherent at 2 AM my time and the recording comes of well, you should be able to download the webcast within the next few weeks at the Coremetrics web site.

I hope those of you in the European web analytics community will be able to join us next week.  I will try and leave plenty of time for questions and answers as well.

Following up on my post from Monday of this week announcing that Joseph Carrabis of NextStage Evolution will be joining “The Engagement Project” and bringing his mathematical expertise to the table, Mr. Carrabis has summarized what he’ll initially be doing for the chef in all of us.

According to Mr. Carrabis:

“Eric’s already posted that I’ll be working with him to make the formula more applicable to a wider variety of interfaces with greater general use features. I also know that I can always use help and have repeatedly and publicly stated that I don’t know web analytics.

So, first steps? A semantically exact statement of what we’re hoping to measure. I suggest this step because it’s much easier to know if your variables will result in the desired solution if you are exact in what the solution looks like and what you have to put into that solution.

Think of it this way; You want to make some chicken soup and you use your grandmother’s recipe. I want to make some chicken soup and I use my grandmother’s recipe. But your grandmother is Irish and mine is Italian. I’ll bet we’d use different spices, different vegetables, different noodles (if indeed we both did).

But I’d bet we both use chicken stock as a base. And is your chicken stock from the leftovers of a roast chicken? What spices did you use there? Or is your stock from bullion?

So the first step is to decide what we all mean by “chicken soup”. One of my mentors was a genius of an author who use to write “speculative fiction”. I would ask, “What is speculative fiction?” and he’d reply “It’s what I’m pointing at when I say it.” This is a great anecdote and an undefensible statement (except in cultural anthropology). If one person “owns” the definition of “speculative fiction”, “chicken soup” or “engagement” then that definition is only valid so long as there exists a market for that definition.

However, a definition that says something like “Basic Chicken Soup”, that is something I can start with to make “Italian Chicken Soup” and allows my Irish friend to extend it to “Irish Chicken Soup”? Now that’s a good definition.

I snuck the concept of “extendable” into the above. “Extendable” means the definition accommodates special cases (Italian, Irish, etc). Think of a recipe for Italian Chicken Soup that begins “Step 1: Make the Basic Chicken Soup. Step 2: Now add garlic, oregano, …” That “Step 2″ part means that the original definition isn’t limited, that it can be extended to incorporate specific features to make it unique to a given environment (Italian, Irish, …).

The concept of “extensible” has two parts; First, you can substitute one thing for another if they share some basic properties. For example, you can substitute a glass of wine for a glass of water in the recipe because they’re both liquids. You can’t substitute a lamb chop for a glass of water, though. Mathematically, this means that if we want to include “clickthroughs” we can use whatever product A calls clickthroughs, whatever product B calls clickthroughs, etc., so long as they all meet some definition of “clickthroughs” (I’ll let the WAA worry about things like that).

Second, “extensible” means new spices, new vegetables, new types of noodles, etc., can be used to make the chicken soup better. This means that you can add a new spice to your recipe in addition to the existing spices already in it. Extensible (in this sense) means you’re doing what you already do to make your style chicken soup and now you’ve discovered something more you can add to it to make even more “your style”. You’re not watering it down or adding more vegetables to make the soup go further. That’s scalability and the equation should be scalable without needing to define it as such.

The sum of these two concepts of “extensible” translates to “the equation is valid across all interfaces including those we haven’t thought of yet.” Mathematically extendability and extensibility form the axes of a very rich solution space.”

Joseph says “Basic Chicken Soup” and I say “a measure of the depth and degree of visitor engagement online” … clearly he and I both have our work cut out for us. If you’d like to join us in our quest for a better measure of visitor engagement online, please let me know.