Archive for 'Web Analytics People'

Long-time readers surely know that I hold June Dershewitz in high regard; not only do I consider her a friend, I respect June as one of the most talented web analytics practitioners and consultants I have ever met. More importantly, June is one of the most fair-minded and thoughtful people working in our industry today, which is why I’m so delighted that she has decided to run for Web Analytics Association board of directors.

To help spread the word about June’s candidacy she allowed me to interview her via email. My questions and her answers follow:

June, can you tell me what made you decide to run for the Web Analytics Association (WAA) Board of Directors?

The thought hadn’t even crossed my mind until last month, Eric, when you suggested that I put my name on the ballot. The more I considered it, and the more people I talked to about what I could possibly bring to the Board, the more I realized that it was a great idea. I’ve invested a lot of energy in developing the web analytics community in my own local area, and I know that I could bring the same energy up a level to help the web analytics community at large on behalf of the WAA.

In a nutshell, what are the top three reasons you believe yourself to be qualified for the board position?

1. I’ve been a hands-on web analyst my entire professional career, I love this work, and I am one of the strongest advocates you’ll find for our trade.
2. I take volunteer work seriously. The Board is made up of volunteers who’ve agreed to spend 15-20 hours per month on the cause. That’s a lot to ask, but I’m ready to make a serious commitment to the job.
3. I don’t play favorites. I want to make sure that we all benefit from the WAA’s efforts, and to that end I will strive to move the organization in a direction that’s in the best interest for all of us.

Have you given any thought to the kinds of things you would like to see the WAA accomplish in during your term if you are elected?

By all means we need to expand our member base at a rate that keeps up with the growth of our field, and at the same time we need to make sure that existing members continue to find value in their memberships. I’m all in favor of finding new ways to provide tangible, useful benefits to members. I also believe we need to form tighter bonds with related associations whose missions overlap with ours, especially as the scope of web analytics becomes broader. In terms of topics that are near and dear to me, I would like to see the formalization of local chapters and the development of a mentoring program.

I know you’re really involved in the web analytics community (being a founder of Web Analytics Wednesday!) Can you describe some of the other work you’ve done for our community in the past?

Web Analytics Wednesday has been a huge focus of my community involvement over the past couple of years. It’s evolved to the point where I’m not only heading up a monthly event series here in San Francisco, I’m also helping other organizers get started with their own events throughout the Bay Area and beyond. My work with WAW has really helped build an established local presence for the web analytics community, and I’m pleased to see similar developments in other cities where WAW has taken hold.

Within the WAA proper, I’ve been involved in the Education Committee, where I helped develop the document that will become the Web Analytics Body of Knowledge, and more recently I’ve become a member of Marshall Sponder’s Social Media Committee. I’m also contributing articles aimed at people who are new to the field of web analytics; my first one came out last month.

You’re running against some pretty heavy hitters in the field, folks like Avinash Kaushik and Jim Sterne to name a few. As you’ve looked at the slate of candidates, who would you like to see elected in this cycle: Who would you like to work with in 2008 and 2009 and why?

I would be honored to work with any previous Board member, especially those – like Avinash – who’ve voluntarily put themselves up for re-election a year early just so we’d have a balance of open positions this year and next.

As far as new people go, it would be great to get to work with Alex Langshur and Vicky Brock. After reading through all 17 candidate statements, I really like what those two have to say: Alex, because he’s committed to achievable goals aimed at bringing value to members, and Vicky, because she sees (as I do) web analytics evolving into business analytics.

Recently Lars Johansson of Satama proposed the idea of term limits for WAA board members, something that I don’t think is in place currently. What do you think? Should WAA board members be limited to one or two terms, or should people be free to serve for as long as they’re able to be re-elected?

The WAA has finally been around long enough for us to consider that issue. I’m definitely in favor of having a 2-term cap on Board membership. Our field is growing so quickly – there are great new people getting started all the time. If they have the energy and the inclination to run for the Board, I want to make sure they have every opportunity to get a spot – even without the name recognition that long-time Board members necessarily have.

Four years from now I’d love to see the Board made up of an entirely new set of people, full of enthusiasm and fresh ideas. By then we’ll be established enough as an association that our mission will be clear to whoever happens to be sitting in the driver’s seat.

You’re a web analytics blogger (one of my favorites!) so here’s an easy one: who’s blogs do you like to read and why do you like them?

Blogs are such a great way to keep up with our field and our community; I make a point to subscribe to everything I come across. Lately I’ve enjoyed Florian Pihs’s blog (I’m outing myself, he doesn’t know I’m a fan) because he’s covering web analytics in China and it’s a unique perspective that I could never hope to get on my own. Oh, and I like to follow bloggers whose sense of humor shines through in their writing, like Alex Cohen and Ian Thomas and (the occasional) Bob Page.

Some have accused the WAA of being somewhat close-minded and having a “not invented here” attitude, something that has the potential to negatively impact the community as a whole. Can you tell me if you encounter this how you might approach the problem?

I believe this attitude tends to propagate when the WAA’s activities are perceived as being shrouded in mystery. As members it’s really tough to figure out why certain policy decisions have been made when you don’t know what’s swirling around behind the scenes. I think it would be a lot better for everybody if more of what the WAA did, decision-wise, happened out in the public. I think we can do a better job being open with all members about what’s going on. Better communication – more honest, thorough communication – would keep negative sentiment in check.

What is your favorite thing about web analytics?

It’s a good intellectual challenge. There’s something about the natural shape of the data that lends itself really well to interesting, solvable story problems.

What is your least favorite thing about web analytics?

The occasional mistaken belief that what we’re doing is spying.

Ours is an increasingly international community and I firmly believe that some of the most exciting opportunities for growth in the industry are in Europe and Asia. Can you describe your experience working with international members of our community?

Over the course of my career I’ve been fortunate enough to work with a pretty global group. For 2 years I was employed by a first-generation web analytics vendor whose main office was in Britain; I spent some time working there, which was a great learning experience. Later, as a member of the central web analytics group at Oracle, I collaborated with an international team of marketers, analysts and developers. Now, as a consultant, I often find myself on the phone with clients many time zones away. Outside of work I’ve also enjoyed meeting international members of our community at conferences and through my blog.

Like you, I believe that our industry has a lot of growth potential beyond North America, and I want to make sure that the WAA does all it can to support international members by encouraging regional/local community, providing non-English language resources, and acknowledging the differences in the way we do business.

Fill in the blanks (here June’s responses are in bold print)

1. At Emetrics, after 10 PM, you’re going to find me in the middle of a great conversation.
2. On a long flight, I spend most of my time photographing my snack to post on Flickr as airplanefood.
3. The one thing most people don’t know about me is I have a herd of dairy goats named in my honor.
4. Everything I know about web analytics I learned from all the smart people I’ve gotten to work with and for over the past decade.

Anything else you think my readers should know about you as they prepare to vote this week?

If, after reading this, you’ve got any questions about where I stand or what my values are, you can write to me directly at june.dershewitz@gmail.com. I aim to represent every one of you, and I welcome your feedback, now and at any point in the future.

In the last few months there has been a tremendous surge in interest in my framework for measuring engagement online. Lately, some of the largest and well-known companies in the world have approached me about working with them to bridge the gap between the metrics they have today and something similar to the composite metric I first described back in December 2006.

While I am tremendously flattered that I have somehow become the focal point for this conversation, I have been thinking lately about how the framework has been developed and how it might end up being used by the measurement industry in general. And while early tests using the framework I’ve described are very encouraging, the calculation in it’s current state was meant to move the discussion along and get more people to “think different” about how engagement could be calculated online.

Given that interest in the framework has clearly increased, one primary concern comes up again and again: the need to apply mathematical rigor to the framework and calculation so that A) the result is repeatable, reliable, and trustworthy and B) when naysayers inevitably emerge to criticize this small side project of mine, that I have a suitable response to their criticism, regardless of where and why it comes.

I believe that the need for “A” is obvious. The need to address “B” is perhaps less obvious, but I believe that I owe it to those of you who are investing your time, energy, and money into this framework. Especially as the stakes seem to increase exponentially with every presentation, every conversation, and every high-visibility blog post on the subject, I believe now is the time to approach the engagement framework not just as a hobby but as a serious project with committed resources.

To this end, I am extraordinarily happy to say that the single smartest person I know, Joseph Carrabis the Founder and Chief Research Officer of NextStage Evolution and NextStage Global, has offered to bring mathematical rigor and analytical precision to what I am officially dubbing “The Engagement Project.” Those of you not familiar with Joseph and his work are advised to A) meet him in person at the upcoming Emetrics Summit in San Francisco or B) read some of his recent work at iMedia Connection.

Joseph will be working to make the formula universally applicable and universally defensible. Suffice to say I can think of nobody better to bring mathematical and scientific rigor to the framework I have been evolving over the past year. Watch this blog and Joseph’s blog at BizMediaScience over the next week or so for a more complete analysis of the framework in it’s current state, something we’ve agreed is the first step towards creating a true function capable practically describing the degree and depth of engagement a visitor is displaying towards a web site over time.

At the end of the day, without regard to my framework, Joseph’s analysis, or any person or group’s particular position on the use of the word “engagement”, my goal is to solve one problem and one problem only:

If you’re interested in working with Joseph and me on The Engagement Project please feel free to contact me directly.

For those of you who missed my presentation yesterday, “Web Analytics: A Day a Month”, you can now listen to the re-recorded webcast at WebEx thanks to Tableau and the American Marketing Association. I say “re-recorded” since once again I managed to bring a large enough crowd to the webcast to break WebEx. Web analytics is hot!

You can listen to the webcast without having to register (still requires name and email) until next week I think by going to:

amaevents.webex.com

Here are a few other things I should mention, as long as I’m writing:

• I’m going to be in Boston next week for Judah’s Web Analytics Wednesday event (rescheduled from last month due to me being a weather-wimp) and if you’re in Boston or nearby I’d love to catch up. Please join us in Cambridge!
• The next few weeks I will be in Chicago (Jan 25th), Seattle (Jan 30th), San Jose (Jan 31st) and New York (Feb 7th) giving the keynote address at OpinionLab’s client conferences. The nice folks at OpinionLab mentioned that they’re opening up the events to non-customers so if you’d like to hear me talk about how quantitative and qualitative data combined provide a much more actionable view of the online visitor, please join us!
• The nice folks at the Direct Marketing Association who gave away PDF copies of my book Web Analytics Demystified in exchange for participation in their web analytics survey (written up by the amazing W. David Rhee) are holding a webinar on the research findings on Januay 23rd. The event is not free but the research is pretty good and if you’re in the DMA you should consider joining the call.
• The nice folks at the Web Analytics Association are also holding a research call, tomorrow (Jan 17th) in fact, on the future of the web analytics industry. I think this event is free but it might only be free to WAA members (maybe if Richard or Andrea read this they can comment for all to see!) The call is tomorrow morning at 9 AM Pacific, noon Eastern and you can register to attend at the WAA web site.
• Anil Batra has apparently jumped on the “bounce rate” bandwagon and is having a “bounce rate survey” that he’d like you to participate in. I haven’t had a chance to take it yet but I really enjoyed Anil’s salary research so I’m sure he’ll do a great job with bounce rate too!
• I’ll be back in San Diego in late February at Aaron Kahlow’s Online Marketing Summit talking about Key Performance Indicators in a Web 2.0 World.  I really enjoyed OMS last year and am looking forward to getting back to Sea World Aaron’s event!
• I had nothing to do with that movie on web analytics, despite it being filmed here in the Rose City, and have no idea what Ian is talking about.  Ian should spend less time at the movies and more time reading what experienced practitioners are saying about Gatineau.  <grin>

If I’m forgetting anything please comment below.  I think you’ll really like the webcast — the feedback I got has been excellent so far (despite some people going gossipy about the title of my last post on the subject … cage match indeed!)

A number of you have commented that I have been oddly quiet on the subject of Omniture planning to acquire Visual Sciences and then the news that four senior-most managers at WebTrends were let go. It’s not that I don’t have an opinion — I can assure you that I do — but I wanted to take a little time to clarify my thoughts on these subjects before blogging about it.

On the Omniture/Visual Sciences deal, I sincerely do congratulate Josh James and the entire team at Omniture on building a company capable of completely taking out their biggest competitor. Over the years I have found myself having a somewhat topsy-turvy relationship with Mr. James and his organization: First I had to compete with them while at WebSideStory, winning some deals and losing others. Then I worked directly with them while at JupiterResearch, spending time both in their offices and also on their behalf through online seminars and client events. Finally I spent a little over a year competing with them again, this time at Visual Sciences, again winning some deals and losing others.

Regardless of where I worked, it was impossible to not develop a healthy respect for Omniture and their success. It pained me to watch deals like HP, AOL, CBS Sportsline, USAtoday, Overstock.com and others go their way, despite hard work from a talented group of individuals, and I absolutely hated going up against their particular version of salesmanship. But as an analyst it was encouraging to see Josh and John Pestana build a company that understood the underlying technology but also how that technology could make their customers more successful.

Their customers responded to this, and still do. It is not uncommon to meet Omniture customers who have “drank the kool aid” and for whom their customer status is very much a badge of honor. Hopefully Mr. James et al. will deliver the same Omniture experience for the 1,500-odd companies they’re purchasing from Visual Sciences/WebSideStory, because that is where I see the inherent risk in this deal.

All week last week people with a lot of money under their management asked me “what is the upside and what is the risk in this acquisition?” I’m not a financial analyst (disclosure: I don’t have any holdings in OMTR or VSCN) so all I could comment on was what I hoped the combined company would do and not do. And while nobody from Omniture has asked me — not that it would be particularly appropriate anyway — here are a few thoughts on what the new company needs to do to make this acquisition successful:

1. Suck it up and start the migration from HBX to SiteCatalyst immediately. I haven’t read all of the various transcripts on this deal, but nobody I am talking to expects HBX to survive the balance of 2008 if the deal is approved (which I sincerely believe it will be.) Omniture should smooth the transition path by splitting data collection at the gateways now and simultaneously loading whatever HBX-collected data into the SiteCatalyst data collectors, thusly giving HBX customers the easiest possible transition from one technology to the other. And while if I were on HBX I would be aggressively thinking about migrating to the SiteCatalyst code base, this transition is far from a slam-dunk at the customer-level. Splitting the data will give marketing something to show I.T. if they complain about needing to replace the JavaScript (again), and getting started on data collection now will potentially ease some of the pain associated with not being able to migrate years of HBX data that some customers might not want to lose (if that is the final assessment.)
2. Admit that Visual Workstation is the right interface for serious analysts. Again, I have not read the transcripts, but comments I have read are unclear about whether Discover 2 or Visual Workstation will live past the acquisition point. And while I have spent much time looking at Discover 2, I can assure you that of the two, Visual Workstation is the technology to keep (disclosure: I recently entered into a licensing agreement with Visual Sciences to use Visual Workstation at Web Analytics Demystified.) No disrespect to Omniture’s fine product team, but Visual Workstation is unparalleled for sheer analyst-class power, and I’m fairly sure that without modification Visual Workstaion can leverage whatever format Omniture stores visitor-level data to get up and running quickly. This may cause problems from a pure SaaS-perspective, and I could be wrong, but I suspect that most analysts wouldn’t actually mind having to run the software locally in exchange for having the robust data manipulation capabilities that Workstation provides.
3. As painful as it will be, resolve the internal stuff quickly. A huge potential pitfall in this deal is that it has tremendous potential to create confusion regarding who is managing what, when, where, and how all of these technologies are presented in sales and support situations. Not that this will be easy, any M&A transaction has the potential to be messy, but Josh and Jim MacIntyre won’t be doing anyone favors by sugarcoating what this deal is or being vague about who might be reassigned and who might be let go (keep in mind that these companies that were bitter enemies in the marketplace up until two weeks ago.) Any internal confusion about the transition will inevitably impact customers in the form of unclear deadlines, changing account managers, and other miscommunication that will only open the door for other vendors …

Which brings me to the other change in the web analytics market last week: WebTrends announcing that Greg Drew, Jason Palmer, Tore Steen, and Hamid Bahadori had all been asked to leave the company. I have to admit, I was more-or-less shocked by this announcement, specially given that I have been saying to folks since mid-July that I believe, at least from a software perspective, that WebTrends is finally getting back on track. I really do believe that WebTrends Score is one of the few true innovations we’ve seen in the web analytics marketplace recently, and learned WebTrends users far and wide have commented that they really like the stuff in the MarketingLab2 release.

I should also say that I personally really like Greg Drew and Jason Palmer. Now, I say that not having worked with or for them in any role other than that of an industry analyst, and anecdotally some of the recent flight from the company can be tied back to their leadership. But Greg has always struck me as one of the nicest guys in the entire industry and someone who was willing to do what it took to get the job done.

Regardless of Greg’s personal disposition, I again find myself nearly flabbergasted that the folks at Francisco Partners who are calling the shots would give up Greg and Jason’s experience in the field and knowledge about web analytics in general. I mean, it’s not like Eli Shapira is going to come back and run the company, or that it will be easy to find someone else to run the ship as experienced with web analytics as Josh James from Omniture, Joe Davis at Coremetrics, or Dennis Mortensen at IndexTools. Especially on the heels of the Omniture/Visual Sciences announcement, this whole thing sounds so fishy it’s almost unbelievable, but I have to believe that these four guys will be harder to replace than people think.

Case-in-point: when Jeff Lunsford showed up at WebSideStory sans web analytics experience, some of us were worried. But Jeff was a natural born-leader, and given time it was clear that Jeff had what it took to get the job done. Unfortunately, in retrospect, it is no longer clear exactly what that job was aside from making a small number of people a huge sum of money, and Jeff has moved on to even bigger deals. I liked working for Jeff tremendously, but I’m not 100 percent sure he left WebSideStory in better shape than he found it.

I’ll admit, I don’t have the experience that these guys have … I’ve been running a company of two people for seven months. But just as I felt like WebTrends was well positioned (along with Coremetrics) to be a strong solution with a great customer base, a good set of features, that was incidentally “not Omniture”, I now find myself questioning how strong the organization will really be when run by folks largely new to web analytics. No disrespect to Tim, John, Leo or Bruce, but web analytics is hard, the competition is big and about to get bigger, and sophisticated web analytics buyers will easily differentiate between passion and experience.

Trust me, I want to be wrong about this. I would like nothing more than to have someone clarify what happened at WebTrends and detail how the company is going to accelerate growth against Omniture given their recent momentum. I think despite Omniture’s strength and Google Analytics widespread deployment that the “web analytics wars” are far from over. Like others, I worry that a two horse race isn’t very exciting to watch, and despite believing that “it’s not the technology, it’s how you use it” that it’s nice to see innovation from time to time. For this to happen, I believe we need a strong WebTrends, a strong Coremetrics, and at least a small handful of smaller innovators out there in the world (Nedstat, IndexTools, Clicktracks, etc.) nipping at everyone else’s heels.

What do you think? Am I crazy? Am I just missing the most obvious thing? Am I too close to the situation, having worked with or for all of the companies involved in the past few weeks insanity? Or do you share some of the same concerns I do? Either way, I’d love to hear what you have to say.

Nick Arnett from MCC Media (and one of the creators of Buzzmetrics) posted a very well though-out and moderately critical assessment of the visitor engagement calculation I wrote about earlier this week. Nick makes some great points and I thought it was worth addressing them while I prepare the follow-up post that shows off some of what the metric can do. My comments are preceded by ETP and Nick’s statements are in italics.

Definitely thought-provoking, Eric… I’m deep into this issue, although focused specifically on community sites.Overall, your approach doesn’t work for me on two main counts — it is too complicated (and thus unlikely to become any sort of standard) and doesn’t generate a metric that allows different sites to be compared. The latter is arguable, since standardized weightings could yield comparable numbers, but I think that’s excess complication also.

ETP: I’m sorry the calculation doesn’t work for you but I do appreciate your thoughts on the subject. Regarding it being too complicated, compared to what? Compared to “simple” metrics like bounce rate and average page views per session? Or compared to the technology you built to power Buzzmetrics? I guess I separate the complexity of making the calculation from one’s ability to actually explain the calculation.

ETP: Regarding using this metric to compare different sites … as I mentioned in the post, I don’t think there is “one” measure of visitor engagement and thusly trying to compare sites is probably a futile effort at best. I suppose you could remove the Brand, Feedback, Subscription and Interaction indices and come up with a standard set of threshold values for specific vertical markets, but I’m not sure that is really the best use of this calculation.

Is there any ground truth behind this? In case that isn’t clear, do you have any sort of primary market data for engagement that correlates with the output of your engagement metric?

ETP: Hmmm, here I’m not sure what you mean. What kind of primary market data is actually able to identify “engaged” visitors? Because I am able to see individuals interacting with my web site, I did talk to a handful of people based on their engagement scores when I was doing the original work on this metric, and some of their feedback was critical to tweaking the metric and inputs to its current state. But other than that I’d love to see the primary data you’re talking about if you’re able to share it!

As I’ve dug into the issues and our data (about five dozen communities ranging from very large to very small), I keep coming back to two main indicators of engagement — return rates and proactive behavior. If visitors don’t visit regularly and do something other than passive page viewing, I have a tough time including them in any measurement of community engagement.

ETP: Exactly why the Recency Index and Interaction Index are included in the calculation, but I disagree with your assessment that these are the only measures of engagement. I’m not sure exactly how I would determine that someone was only “passively” viewing pages, and again this metric is not designed to be a measure of “community engagement” but rather visitor engagement more broadly considered.

Some point-by-point thoughts…

Click-depth index — this is a place where ground truth really matters, I think. I’m not comfortable with the assumption that more clicks per session means greater engagement. Do we know enough about browsing behavior to know that this is true? And of course there’s the old problem of bad design resulting in more clicks… but when I consider that issue, I tend to think that if people show willingness to click through a bad design, maybe that means they really are engaged! Perhaps we should all include some known bad design… ;-)

ETP: I haven’t seen anything that says that more clicks means less engagement but I agree that confused people might generate more clicks. But I think it’s unlikely that confused and frustrated people would return, complete defined events, subscribe to blogs, etc. so despite your assertion that the metric is complex, multiple inputs are designed to mitigate those that may be confusing.

ETP: You do, however, make an excellent argument for not using something as simple as “click-depth” or “average depth of visit” as your sole measure of engagement.

I have pretty much the same questions about duration. Is there good, objective evidence that session duration correlates to engagement? There are visitors with long-duration visits who don’t visit regularly and don’t do anything proactive… I can’t see including them in any measurement of engagement.

ETP: It sorta depends on your definition of engagement, doesn’t it? But see my comment above about why a single measure like duration (as in Nielsen’s Time Spent ranking system) is perhaps inappropriate on its own to determine engagement.

Recency makes perfect sense to me — the fact that engaged visitors return often is practically a tautology. I would be very skeptical of calling anybody engaged if they aren’t returning regularly.

ETP: What about first time visitor? Are you saying you can’t be engaged on the first visit to a site? I agree that regular returns are a good indicator of engagement, but in my analysis the metric I’ve defined is able to resolve first time visitors into several engagement segments which I personally have found quite useful.

Your Brand Index is a great piece of data, but I don’t believe it works in a metric intended to compare sites. Language is too subtle and ambiguous to infer engagement from search terms. I spent years in the search engine and related markets, which gave me a great appreciation for the fact that what sometimes seems obvious about language isn’t. When people search on brand-related terms, it indicates *reach* to me, not engagement. I’m unwilling to assume anything more than brand awareness. People search on things they dislike, but that doesn’t mean they’re engaged with the subject they’re searching. And my data shows that visitors who show many other indications of engagement actually search *less* often.

ETP: Same comment about this metric not being specifically designed for comparing sites. I know that is the uber-goal for lots of folks in the world, it’s just not necessarily my goal or the best use for my engagement calculation.

ETP: Doesn’t “reach” plus “action” equal engagement? I haven’t spent years in search and related markets, but I struggle to believe that people searching on brands they dislike are not somehow engaged. Again, maybe this is a semantic issue arising from conflicting definitions of engagement.

ETP: Because the calculation is designed to be made over the lifetime of visitor sessions, searching less often is not a problem. I guess I more-or-less expect that the “direct” component of the Brand Index will be more important over time with truly engaged visitors (who wouldn’t be as likely to go back to Google and search on a branded term.)

Counting brand-related searches makes sense if we’re measuring *brand* engagement.
Counting direct (non-referred) visits makes sense if we’re measuring *site* engagement.

Counting both in the same metric doesn’t make sense to me. I don’t think we should even be talking here about ways to measure brand engagement… because I believe that’s well beyond the scope of site analytics. It requires massive monitoring systems along the lines of BuzzMetrics. (I’m the primary inventor of one of their systems.)

ETP: I’m not differentiating *brand* and *site* engagement since I’m trying to calculate an operational measure of ongoing *visitor* engagement. Brand is just a component, and the site is the measurement point. I think I understand your desire to differentiate the two given your background with Nielsen but I’m not trying to do the same thing.

One more problem with the Brand Index — people will argue all day long about what terms are appropriate to include… and there’s a strong incentive for site owners to err on the side of too many terms if their success is being measured by this metric. For example, you included “web site measurement hacks” in your list… but that could be a generic term. Is “Web Analytics Wednesday” really your brand? Or is it the WAA’s? I don’t want to argue which it is, just point out the kind of ambiguity that is inevitable.

ETP: Here I agree with you, coming up with a reasonable list is not easy, but web analytics is hard so at some point you have to make some tough decisions. “Web Site Measurement Hacks” is a book title and a branded term but could be a generic phrase. “Web Analytics Wednesday” is a branded Web Analytics Demystified term and has nothing to do with the Web Analytics Association. I don’t think there is that much ambiguity at the site level, at least in my experience.

Your Feedback Index is a specific instance of what I think of as the general principle of tracking proactive behaviors — what you seem to be getting at in your Interaction Index. In communities, visitors have many such opportunities — posting, editing, tagging, voting and so forth. I decided very early in this work to just give people one point for each such proactive action, despite the temptation to weight them (which would violate the need to keep things simple). These are the behaviors that make a community work; sites that aren’t based on user-generated content can exist without them.

ETP: Same comment about this calculation perhaps not being what you’re looking for vis-a-vis communities.

Your session focus really got me thinking. Does it make more sense to count the number of sessions in which visitors signal engagement or the number of actual such signals? I think it’s close to a toss-up, but so far, our ground truth suggests the latter — the number of proactive actions correlates better to our subjective estimates of engagement… but among our future tasks is to establish better ground truth. So far, I’m just using our community manager’s collective subjective scoring… but it correlates quite well to all but our smallest communities.

ETP: I agree, it’s probably a toss-up but if you think about the calculation all it does is count the number of signals. Long sessions are a signal, deep sessions are a signal, frequent sessions are signal, etc. I know you don’t like anything but recency and interaction but we can agree to disagree on this point. I’d love to hear about your “ground truthing” efforts and I’ll try and keep you appraised of mine.

The subscriber index doesn’t work for me because we want to be able to compare communities regardless of whether or not visitors are able to subscribe, join, become members or what-have-you. Some of our clients — e.g., a large professional sports organization — allow full participation without any need to sign up. Also, as I’ll explain below, I’ve found a strong negative correlation between highly active visitors and RSS subscribers.

ETP: Again, not designed for comparison (and at this point no wonder you don’t like my calculation!) I’d love to see the negative correlation data for RSS and yes, if you don’t have a subscription it doesn’t make sense to assign a negative penalty.

Finally, I guess I’ll toss out one of the ideas that I’m working with — segmenting visitors by proactivity.

In several ways, communities (and most web sites, I suspect) have a bimodal distribution of users. There’s typically a relatively large “Core” group that visits often, looks at lots of pages and does a lot of proactive stuff. There’s a middle ground, which I’m calling “Lingerers,” of people who fall into the 10th to 80th percentiles of such activities. Third and last, there’s a large contingent in the 0th percentile, people who might have one or two activities in a given time period, which I call the “Drive-bys.” In our communities, the Drive-bys are the largest group, but the Core usually is a bigger group than the Lingerers. What this says to me is that people tend to engage a lot or hardly at all — there isn’t much middle ground. I’ve been focusing on the Core’s relationship to the whole community for my engagement measurements. That’s what seems to correlate best to what little ground truth we have.

ETP: I am seeing a more normal distribution, especially as visitors return a third time, but it is definitely left-skewed towards lower levels of engagement. I’ll try and highlight this when I show some data that highlights the calculation in action. And since I’m not working on a community proper, I’ve found myself focusing on my middle group (“Moderately Engaged”) and trying to determine what I might be able to do to shift them up to “Highly Engaged”.

Overall, I’ve found that the Drive-bys and Lingers exhibit fairly similar behavior, but the Core is different. The Core visitors post more, search less and use RSS far less (so much for “subscribing” to RSS as a positive indicator of engagement!)

ETP: Your assessment of RSS being a poor indicator of engagement runs contrary to popular opinion (why would you subscribe to a RSS feed or email newsletter if you weren’t engaged??!) Perhaps this result is uncovering a flaw in your engagement calculation?

This post is getting long… so I’ll wrap it up (but ready to discuss further, of course) by repeating myself. I think any sort of engagement metric has to be backed up by demonstrating correlation to some kind of ground truth. Otherwise, it’s a mental exercise that runs the risk of having little relevance to the marketplace.

ETP: You keep coming back to the notion of “ground truth” but surely you recognize that this is A) extraordinarily difficult to come by and B) if we had it easily available we wouldn’t need a measure of engagement. I would love to see your “ground truth” data and talk about how you’re generating that, but unless I’m missing something it sounds a little impractical for widespread use. Still, I appreciate your feedback and very thoughtful comments and will endeavor to demonstrate the correlation between my calculation and “truly engaged” visitors.

---

Man, talk about a long post! What do you think? Is Nick more right than wrong? Are you focusing on communities and have the same concerns? Do you have similar concerns about your site? The conversation is almost as interesting as the metric and resulting analysis in my opinion so please, comment away!