Registration is now open for ACCELERATE 2014 in Atlanta, Georgia on September 18th. Reserve your spot today at Eventbrite — tickets are only $99 USD!


How do you calculate engagement? Part II

Published by Eric T. Peterson on December 18, 2006 All posts from Eric T. Peterson

Given that my last post on measuring engagement generated a fair amount of feedback, I wanted to follow-up with the post that in retrospect I should have published first, the nuts and bolts behind the engagement calculation.

Since there are numerous definitions of “engagement” that could be applied to the online channel, I choose to use the following definition:

Engagement is an estimate of the degree and depth of visitor interaction on the site against a clearly defined set of goals.

My definition sounds like conversion rate except engagement is a more flexible concept, one that can accommodate a variety of business needs such as those described by Bill Gassman in his comments to my last post and those of Craig Danuloff who is looking for a metric that accommodates a variety of visitor activities.

Based on my knowledge of my site visitors and their long-term usage patterns, my engagement goals are as follows:

  1. I would like that visitors would view and interact with certain content on my site;
  2. I would like visitors to subscribe to this weblog to stay connected;
  3. I would like visitors to maintain a low recency with my content, regardless of whether they’re reading blog posts or viewing pages on my site;
  4. When visitors are on my web site, I would like them to spend a reasonable amount of time interacting with my content;
  5. When visitors return to my site, I prefer they remember my domain name and return to my site directly, either via a bookmark or by directly entering my URL into their browser.

Now, many of you will likely argue with these criteria, and fairly so. It’s fine that you may have a different definition of engagement for your site; I think that Bill put it best when he commented:

“Each organization’s version of engagement will be unique. It will be derived from a number of root metrics, probably under a dozen. Common root metrics will be frequency, recency, length of visit, purchases and lifetime value. Some organizations may include visitor actions, such as subscribing, providing personal information, writing a comment, or participating in a blog.”

I’m using the criteria I listed above based on my knowledge of my site visitors, mined from a variety of channels including site activity, email, comments, personal conversations, etc., juxtaposed against my site’s business objectives (see below.) Given a sufficiently flexible analytics package you can build your engagement metric using any goals you like …

Regarding item #1 in the list above, wanting visitors to interact with certain content on my site, here are the activities I am tracking broken down by moderate- and high-value:

Moderate-Value Activities

High-Value Activities

Because it is very difficult to know a visitor’s intent when they visit a web site, these activities are designed to allow me to examine the visitor not in the context of their intent but rather in the context of my site’s specific objectives. I maintain Web Analytics Demystified for three primary reasons:

  1. To sell my books
  2. To maintain my visibility in the web analytics field
  3. To have a channel through which I can continue to contribute ideas to our community

You may argue that tracking a visitor’s interaction with specific contents is a poor measure of engagement given that visitors may be looking at an entirely different set of content and are intensely engaged … fair enough. But these lists represent the activities that visitors can perform on my web site that are in-line with my stated business objectives.

If highly-engaged visitors are interacting with some other content on my site, that would prompt me to reconsider that contents contribution to my engagement calculation and perhaps add it to one of the lists above. My belief is that any engagement estimate must take content consumption into account given that it is the content that drives visitor engagement in the first place.

This post is getting long so it’s clear I’ll need a “Part III” (and maybe a “Part IV”) but here is something tangible to chew on until I have time to post again. Based on my five business goals stated above, my engagement calculation is essentially this:

(Pct High-Value Content Consumption Sessions + Pct Moderate-Value Content Consumption Sessions + Blog Subscriber Reads per Session + Pct Recent Sessions + Pct “Long” Sessions + Pct Direct Sessions) / 6

I am calculating the percentage of sessions on a per-visitor basis and summing those percentages to generate an “engagement score” between 0.0 and 6.0. I convert this score to a percentage itself to make it easier to read and voila! I can apply my engagement metric to any dimension I am tracking in Visual Site.

By clearly defining my engagement goals and then systematically scoring visitors against that framework, I can build a metric that can be objectively applied regardless of whether visitors buy a book. I can apply my engagement estimate to any dimension I am tracking on my site, allowing me to discover patterns of visitor behavior that would not be obvious based on more traditional metrics such as conversion rate, session duration, or page view count.

Just so I don’t lose you, here is one of the visualizations I am using to better understand visitor engagement showing visitor engagement by percent of visitors by visitor city:

It’s hard to see with the scale I’m providing in this image but I can assure you that the long-tail is there. And sure, with my $50 book I’m unlikely to launch a geo-targeted marketing campaign in markets where visitors are, on average, twice as engaged as my site-wide population … but maybe you would!

Until next time, I welcome your comments and criticism.

Share this blog post ...
Twitter5LinkedIn0Facebook0Google+0Email

Categorized under Engagement, Key Performance Indicators, Web 2.0

  • Anonymous

    Two thoughts on your engagement goals:

    * #5: methods for coming to your site. Why? Does it matter? I see folk on several sites use, for example, Google as their bookmark holder. They Google for the obvious name. A simple appended “.com” would have worked for them too.
    The point being, are you predisposing how visitors choose to interact with your site? Funnelling before we even get here? ;-)
    My concern being, I almost always find it easier to get here via someone else’s blogroll. Despite having you bookmarked. Go figure. Maybe I’m just odd.

    * It may be the fault is within my understanding of what you’ve written. But it almost seems like you’ve determined over time how your visitors use your site, and set your goals to match that behavior.
    That may be fine for a MySpace or similar, but are you taking the easy way out with that target(s)? Friendly question, not harsh criticism.
    Would you be better off determining what your actual goals are and instead trying to capture that audience, which may be slightly different to your current one? Some overlap to be expected of course. :-)

    ???

    Cheers!
    - Steve

  • http://www.webanalyticsdemystified.com Eric

    Steve: Good point about how visitors come to the site. One thing I like about people coming to the site by typing the URL or bookmarking is that they’re remembering my brand name and URL when they do that. I suppose your point is valid though and I could add sessions where the visitor had searched for any variation on my brand name and treat those sessions as “direct” as well.

    Regarding your second point, it’s a good question. I think you can get to the same place by clearly defining your business objectives and engagement goals and then determining what pages or timeframes should be included in the engagement definition. But you’re right to question that since I’m not yet sure if you can actually measure engagement without a sense of your visitor’s interaction.

  • MikeK

    Eric — It seems like it might be important to take into account people who score highly in two or more components of your engagement metric. I’m thinking of it like a Venn diagram: the really engaged visitors will be in the intersection of several engagement circles. This would make your metric much more complicated to calculate, but it would identify the REALLY engaged visitors.

    Do you think worrying about this is needlessly complex?

  • http://www.webanalyticsdemystified.com Eric

    Mike: That’s absolutely what I’m doing but I like the idea of the Venn diagram to help explain what I’m doing. Fortunately, it turns out that is isn’t very complicated to make the calculation.

    As visitors engage in each of my defined goals they are given a score for each goal between 0 and 1. The sum of these individual goal scores then becomes the basis for the engagement calculation.

    I’ll provide more detail when I get to the third installment in the series, going so far as to actually show the calculation in Visual Sciences. Thanks for the feedback and keep your eyes peeled!

  • Anonymous

    Eric:

    I just read a post by Clint Ivy in response to your second post on engagement. Basically, he categorizes your list of engagement activities into various buckets –

    * Social Media Activities: 31.25%
    * Commerce Activities: 43.75%
    * CRM Activities: 12.5%
    * Other (Brand) Activities: 12.5%

    and observes that, quote “Only five of the activities are related to the social aspect of the site AND only one of those five is of high engagement value?”

    It highlights the difficulty in identifying what is regarded as “valuable” site content areas.

    I will argue that engagement is not necessarily limited to the visitors interaction with “useful” content on your site, but rather a broad indication of the visitors level of interest in anything your site offers. The varying degrees of interest is a direct measurement of the various levels of engagement.

    Hence, i think the core components of an engagement measure should be things like:

    * frequency of visit
    * recency of visit
    * duration of visit

    thanks!

    - Victor

  • http://www.webanalyticsdemystified.com Eric

    Victor: Excellent point. As I hope to show in my next post, the engagement calculation is a function recency, frequency, and duration, as well as the visitor’s attraction to site content that is important to my business goals.

    This way, the engagement calculation can represent both the user’s needs independent of my business objectives and my specific business objectives. Put another way, I think that by removing site content from engagement will result in a metric that does not map well to the reason I have a web site.

    This may be the essence of Clint’s criticism, that I have put on my “corporate hat” to define engagement. Clint wanted me to put on my “social media” hat. And you, I think, are asking me to not wear a hat at all and only look at truly objective measurements. But, given my firm position on the value of a key performance indicator (needs to drive action), I believe that a hat must be worn and that the engagement metric must relate back to a specific objective other than just visiting the site and clicking lots of links.

    Still, I think in my fourth installment I’ll do a comparison of your, Clint’s, and my engagement metric to see what the results look like. Hopefully that will give us a better lens through which to examine each of our hypotheses.

    Thanks for commenting!!!

  • http://blog.instantcognition.com Clint

    I think the question is – does engagement offer a unique and helpful point of view in terms of driving action (that’s the reason we measure right?)?

    In my post, I note that Eric, unlike most bloggers or others involved in social media that are asking for an engagement metric, has a site that has multiple purposes, where my site is just a blog. This is a question that so called ‘traditional’ web companies will be/are facing – what’s the value of social media and how will they measure it?

    Eric has cunningly turned engagement towards more traditional actions (buying books for example) but does engagement inform you better about propensity to buy than does traditional conversion rate analysis or scenario analysis – that’s yet to be answered.

    A more fundamental question might be “Do social media companies have a fundamentally new business model that requires new measurement models?”

  • Ian

    I just posted a really long comment to Clint’s blog but I see now I probably should have added it here. :O

    I think engagement does offer more beyond the vistor does X the more likely they are to do Y.

    IMHO – Engagement is about qualitatively measuring performance towards goals for how users interact with your business.

    Purchase interaction and Blog commenting interaction are both important if your business is about getting users to engage with you in those ways.

    I believe it is most powerful when used to trend against site changes (whether A/B or over time), marketing efforts and even simply time (with no changes) to see when you might need to adapt to changes in the marketplace.

    Yet with all of this I will also concede that most “Engagement” metrics seem very much like simply higher level (more complex) KPIs.

    The question I continue to ask myself as I develop these formulas for myself is what details are these complex formulas hiding. Which seems to be a common concern.

    For more on how I measure engagement see Clint’s blog. ;)

    -Ian

  • http://www.webanalyticsdemystified.com Eric

    Clint: You asked: “Do social media companies have a fundamentally new business model that requires new measurement models?” to which I would answer, in true-Sterne’ian fashion, “It depends.”

    If social media companies are trying to drive eyeballs who will ultimately click on banner ads (e.g., banner ads on YouTube, AdSense on Digg and LinkedIn) or make some kind of ancillary purchase (Business and Pro accounts at LinkedIn, memberships at Classmates.com) then I would say, “No.” But, if these companies are simply trying to build a substantial base of highly-engaged users, without regard to their likelihood to purchase or not, then I’d have to say “Yes.”

    Either way, I think that the engagement measurement framework I have proposed will work. Given that the calculation I have already described accommodates purchase activity just as easily as it does retention and duration, I think that these social media companies would do well to think about engagement the way I have outlined it. The framework accommodates (using YouTube as an example) the uploading of movies, the viewing of movies, commenting on movies, sharing of movies, and clicking on banner ads equally well.

    And I was thinking after our conversation yesterday, I’m not sure that I would ** want ** engagement to inform me better about a visitor’s propensity to purchase. For that I already have my visitor- and session-based conversion rates. With engagement I am looking for a KPI that describes ** much more ** than propensity to purchase. Does that make sense?

    Anyway, as usual, thanks for your feedback and the thoughts you have already blogged.

  • http://www.webanalyticsdemystified.com Eric

    Ian: I agree with you fully. You won’t be surprised when I show you how I use Visual Site to understand the component elements of my engagement calculation since it’s really not that difficult. But I believe that the beauty of a single engagement metric or score is that it provides a high-level view of how visitors from X are using the web site.

    The engagement metric I am describing can easily be trended, you can generate alerts against it, it is plug-and-play into dashboards and KPI reports, etc. And, more importantly, it provides a robust basis for comparison across multiple dimensions that does not require the type of oversimplification that other metrics rely upon.

    I appreciate your feedback and am charged up to expose the next level of detail in these calculations. Thanks!

  • Pingback: Web Analytics Demystified » Blog Archive » How to measure visitor engagement, redux

  • Pingback: Measuring User Engagement, with Examples from Yahoo

  • Pingback: Todays Research – Customer Engagement « Sinead Thesis

  • Pingback: Measuring User Engagement, with Examples from Yahoo

  • Pingback: Visitor Engagement Being Measured

  • Pingback: Engagement Score: the one metric you need (if you care about brand). | digital balance

 


Recent Blog Posts

SiteCatalyst Unannounced Features
Adam Greco, Senior Partner

Lately, Adobe has been sneaking in some cool new features into the SiteCatalyst product and doing it without much fanfare. While I am sure these are buried somewhere in release notes, I thought I’d call out two of them that I really like, so you know that they are there.

Continue reading this article ... ... more from Adam Greco

Hello. I’m a Radical Analytics Pragmatist
Tim Wilson, Partner

I was reading a post last week by one of the Big Names in web analytics…and it royally pissed me off. I started to comment and then thought, “Why pick a fight?” We’ve had more than enough of those for our little industry over the past few years. So I let it go.

Except I didn’t let it go.

Continue reading this article ... ... more from Tim Wilson

Competitor Pricing Analysis
Adam Greco, Senior Partner

One of my newest clients is in a highly competitive business in which they sell similar products as other retailers. These days, many online retailers have a hunch that they are being “Amazon-ed,” which they define as visitors finding products on their website and then going to see if they can get it cheaper/faster on Amazon.com. This client was attempting to use time spent on page as a way to tell if/when visitors were leaving their site to go price shopping.

Continue reading this article ... ... more from Adam Greco

How to Deliver Better Recommendations: Forecast the Impact!
Michele Kiss, Partner

One of the most valuable ways to be sure your recommendations are heard is to forecast the impact of your proposal. Consider what is more likely to be heard: "I think we should do X ..." vs "I think we should do X, and with a 2% increase in conversion, that would drive a $1MM increase in revenue ..."

Continue reading this article ... ... more from Michele Kiss

ACCELERATE 2014 “Advanced Analytics Education” Classes Posted
Eric T. Peterson, Senior Partner

I am delighted to share the news that our 2014 “Advanced Analytics Education” classes have been posted and are available for registration. We expanded our offering this year and will be offering four concurrent analytics and optimization training sessions from all of the Web Analytics Demystified Partners and Senior Partners on September 16th and 17th at the Cobb Gallaria in Atlanta, Georgia.

Continue reading this article ... ... more from Eric T. Peterson

Product Cart Addition Sequence
Adam Greco, Senior Partner

In working with a client recently, an interesting question arose around cart additions. This client wanted to know the order in which visitors were adding products to the shopping cart. Which products tended to be added first, second third, etc.? They also wanted to know which products were added after a specific product was added to the cart (i.e. if a visitor adds product A, what is the next product they tend to add?). Finally, they wondered which cart add product combinations most often lead to orders.

Continue reading this article ... ... more from Adam Greco

7 Tips For Delivering Better Analytics Recommendations
Michele Kiss, Partner

As an analyst, your value is not just in the data you deliver, but in the insight and recommendations you can provide. But what is an analyst to do when those recommendations seem to fall on deaf ears?

Continue reading this article ... ... more from Michele Kiss

Overcoming The Analyst Curse: DON’T Show Your Math!
Michele Kiss, Partner

If I could give one piece of advice to an aspiring analyst, it would be this: Stop showing your "math". A tendency towards "TMI deliverables" is common, especially in newer analysts. However, while analysts typically do this in an attempt to demonstrate credibility ("See? I used all the right data and methods!") they do so at the expense of actually being heard.

Continue reading this article ... ... more from Michele Kiss

Making Tables of Numbers Comprehensible
Tim Wilson, Partner

I'm always amazed (read: dismayed) when I see the results of an analysis presented with a key set of the results delivered as a raw table of numbers. It is impossible to instantly comprehend a data table that has more than 3 or 4 rows and 3 or 4 columns. And, "instant comprehension" should be the goal of any presentation of information — it's the hook that gets your audience's brain wrapped around the material and ready to ponder it more deeply.

Continue reading this article ... ... more from Tim Wilson

Automating the Cleanup of Facebook Insights Exports
Tim Wilson, Partner

This post (the download, really — it’s not much of a post) is about dealing with exports from Facebook Insights. If that's not something you do, skip it. Go back to Facebook and watch some cat videos. If you are in a situation where you get data about your Facebook page by exporting .csv or .xls files from the Facebook Insights web interface, then you probably sometimes think you need a 52" monitor to manage the horizontal scrolling.

Continue reading this article ... ... more from Tim Wilson

The Recent Forrester Wave on Web Analytics ... is Wrong
Eric T. Peterson, Senior Partner

Having worked as an industry analyst back in the day I still find myself interested in what the analyst community has to say about web analytics, especially when it comes to vendor evaluation. The evaluations are interesting because of the sheer amount of work that goes into them in an attempt to distill entire companies down into simple infographics, tables, and single paragraph summaries.

Continue reading this article ... ... more from Eric T. Peterson

Funnel Visualizations That Make Sense
Tim Wilson, Partner

Funnels, as a concept, make some sense (although someone once made a good argument that they make no sense, since, when the concept is applied by marketers, the funnel is really more a "very, very leaky funnel," which would be a worthless funnel — real-world funnels get all of a liquid from a wide opening through a smaller spout; but, let’s not quibble).

Continue reading this article ... ... more from Tim Wilson

Reenergizing Your Web Analytics Program & Implementation
Adam Greco, Senior Partner

Those of you who have read my blog posts (and book) over the years, know that I have lots of opinions when it comes to web analytics, web analytics implementations and especially those using Adobe Analytics. Whenever possible, I try to impart lessons I have learned during my web analytics career so you can improve things at your organization.

Continue reading this article ... ... more from Adam Greco

Registration for ACCELERATE 2014 is now open
Eric T. Peterson, Senior Partner

I am excited to announce that registration for ACCELERATE 2014 on September 18th in Atlanta, Georgia is now open. You can learn more about the event and our unique "Ten Tips in Twenty Minutes" format on our ACCELERATE mini-site, and we plan to have registration open for our Advanced Analytics Education pre-ACCELERATE training sessions in the coming weeks.

Continue reading this article ... ... more from Eric T. Peterson

Current Order Value
Adam Greco, Senior Partner

I recently had a client pose an interesting question related to their shopping cart. They wanted to know the distribution of money its visitors were bringing with them to each step of the shopping cart funnel.

Continue reading this article ... ... more from Adam Greco

A Guide to Segment Sharing in Adobe Analytics
Tim Wilson, Partner

Over the past year, I've run into situations multiple times where I wanted an Adobe Analytics segment to be available in multiple Adobe Analytics platforms. It turns out…that's not as easy as it sounds. I actually went multiple rounds with Client Care once trying to get it figured out. And, I’ve found "the answer" on more than one occasion, only to later realize that that answer was a bit misguided.

Continue reading this article ... ... more from Tim Wilson

Currencies & Exchange Rates
Adam Greco, Senior Partner

If your web analytics work covers websites or apps that span different countries, there are some important aspects of Adobe SiteCatalyst (Analytics) that you must know. In this post, I will share some of the things I have learned over the years related to currencies and exchange rates in SiteCatalyst.

Continue reading this article ... ... more from Adam Greco

Linking Authenticated Visitors Across Devices
Adam Greco, Senior Partner

In the last few years, people have become accustomed to using multiple digital devices simultaneously. While watching the recent winter Olympics, consumers might be on the Olympics website, while also using native mobile or tablet apps. As a result, some of my clients have asked me whether it is possible to link visits and paths across these devices so they can see cross-device paths and other behaviors.

Continue reading this article ... ... more from Adam Greco

The 80/20 Rule for Analytics Teams
Eric T. Peterson, Senior Partner

I had the pleasure last week of visiting with one of Web Analytics Demystified’s longest-standing and, at least from a digital analytical perspective, most successful clients. The team has grown tremendously over the years in terms of size and, more importantly, stature within the broader multi-channel business and has become one of the most productive and mature digital analytics groups that I personally am aware of across the industry.

Continue reading this article ... ... more from Eric T. Peterson

Ten Things You Should ALWAYS Do (or Not Do) in Excel
Tim Wilson, Partner

Last week I was surprised by the Twitter conversation a fairly innocuous vent-via-Twitter tweet started, with several people noting that they had no idea you could simple turn off the gridlines.

Continue reading this article ... ... more from Tim Wilson

Omni Man (and Team Demystified) Needs You!
Adam Greco, Senior Partner

As someone in the web analytics field, you probably hear how lucky you are due to the fact that there are always web analytics jobs available. When the rest of the country is looking for work and you get daily calls from recruiters, it isn’t a bad position to be in! At Web Analytics Demystified, we have more than doubled in the past year and still cannot keep up with the demand, so I am reaching out to you ...

Continue reading this article ... ... more from Adam Greco

A Useful Framework for Social Media "Engagements"
Tim Wilson, Partner

Whether you have a single toe dipped in the waters of social media analytics or are fully submerged and drowning, you’ve almost certainly grappled with "engagement." This post isn’t going to answer the question "Is engagement ROI?" ...

Continue reading this article ... ... more from Tim Wilson

It’s not about "Big Data", it’s about the "RIGHT data"
Michele Kiss, Partner

Unless you’ve been living under a rock, you have heard (and perhaps grown tired) of the buzzword "big data." But in attempts to chase the "next shiny thing", companies may focus too much on "big data" rather than the "right data."

Continue reading this article ... ... more from Michele Kiss

Eric T.
Peterson

John
Lovett

Adam
Greco

Brian
Hawkins

Kevin
Willeitner

Michele
Kiss

Josh
West

Tim
Wilson

Contact Us

You can contact Web Analytics Demystified day or night via email or by reaching out to one of our Partners directly.

» Contact Information

Web Analytics Demystified, Inc.
P.O. Box 13303
Portland, OR 97213
(503) 282-2601


Useful Links