The blogger Avinash Kaushik recently won an award for his post that largely echoed the content of my first book, Web Analytics Demystified. While I have never heard of the award, I am happy for Mr. Kaushik and excited because his repeated mention of my book’s title is driving incremental sales. Thanks Avinash!While I’m happy for Avinash, I have to wonder about his repeated insistence that “web analytics is easy.” I wrote as much in a comment I posted to his blog. The comment has not yet appeared (not sure why, maybe Avinash is on vacation) but I’m interested in your reaction to my position so I figured “hey, I have a blog …” and have published the comment below.

Basically I think that web analytics is hard — not easy or necessarily complex — but I think that this is very good news! “Hard” is something that all of us can overcome with determination and effort, just like the New York Giants did this past Sunday … all we have to do is recognize the level of effort and determination that is required and set expectations accordingly.

The verbatim content of the (missing) comment follows:

Avinash,

Congratulations on your award. I’ve never heard of the award but I don’t work nearly has hard on SEM/SEO as you do. Regardless I’m sure winning this award is quite an important accomplishment for you.

I also wanted to thank you. As the author of “Web Analytics Demystified” — the classic work you say your post pays homage to — it was interesting to hear your take on the subject. More interesting was how your post positively impacted sales of the book — direct sales were up nearly 40% from the month previous!

Perhaps your post reminded people that some of us have been around “demystifying” web analytics for a long time.

Either way, despite some people’s complaints that you were inconsiderate in your attempt to game my brand and copyright, I can assure you that I do appreciate your willingness to continue to bring awareness to my work, both as an author and as a consultant. And I sincerely hope you didn’t mind my tongue-in-cheek title for my AMA presentation …

Still, I wanted to ask you one question: Do you think the Giant’s thought that winning the Super Bowl was complex? Or do you think they thought it was hard?

I ask because you seem hung-up on my assertion that “web analytics is hard”, going out of your way to try and convince people that it is “easy but complex, not hard” and that somehow complex is preferred.

And while I’m not sure why you’re so violently opposed to my assertion, I do disagree with you — web analytics is hard, and that is fantastically good news! Web analytics is hard just like an underdog with a young quarterback winning the Super Bowl is hard. Nothing good is easy, but anyone, anywhere can do something difficult simply by being dedicated to the end product.

• Sacking Tom Brady is hard (hell, very hard) but the Giants defense did it again and again and again
• Converting on third down is hard, but Eli Manning and his receivers did it when it counted the most

There is nothing particularly complex about football, perhaps save contract negotiations, and there is nothing overly-complex about web analytics. In my experience the complexity that does come up in web analytics usually comes from “over-promise, under-deliver” and end-user needs for basic tools like segmentation and data integration that are lacking in some of the most popular applications, not from the practice of web analytics itself.

• Web analytics is hard, but anyone with enough motivation can overcome this obstacle and, in your own words, create amazing, magnificent, and wonderful success!
• Web analytics is hard, but simply by recognizing this and moving beyond the “web analytics is easy” hyperbole companies are able to create the right expectation about the effort required to be successful.
• Web analytics is hard, and that is excellent, excellent news. Because if web analytics were impossible there would be no need for Google Analytics or any of the other great applications available today, but if web analytics was easy, most of us would be massively under-performing.
• Web analytics is hard, but just like the Giants and every other Cinderella team in the history of sports, hard can be overcome through sheer determination, effort, and a well-formed plan for success: often the result is greatness.

I hope you’ll print this comment, and I know my thoughts will do nothing to dissuade you from your “web analytics is easy” mantra, but I had some time (ironically waiting to fly to New York where I hope to catch a Giant ticker-tape parade this week) and wanted to comment.

Again, congrats on the prestigious award! I’m sure that Stephan Spencer and Tamar Weinberg were quite bummed to have been nominated against you — I know I would be!

Sincerely,

…

Some have accused me of “over-messaging” on this point and pointing out the obvious. Perhaps, but as long as I keep seeing the relieved look on my client’s and audience’s faces when I tell this simple truth, and as long as people continue to come up to me and thank me for clarifying expectations about their use of measurement technology, I’m going to stay on message. “Web analytics is hard” seems to match well with people’s experience, and more importantly, nobody appears too perturbed about this statement (except for perhaps Mr. Kaushik and his business partners.)

What do you think? Is web analytics easy? Is it complex? Is it hard? Does it matter? Are you perturbed by my assertion? If so, why? What am I missing?! I’d love to receive your feedback …

[ I’m really happy to have my first “guest post” from blogger Robbin Steif from Lunametrics. Robbin really liked my “gradual building of context” post from awhile back and she and I have been discussing a related metric that she thinks builds nicely on my visitor engagement metric. Without further ado, Robbin Steif … ]

On the one hand, I thought that Eric’s recent post, The Gradual Building of Context, was just awesome. Although every site has to define visitor engagement for itself, every site is still capable of pulling together similar numbers (which is why I loved it.)

On the other hand, I disagreed with Eric’s final conclusion, “I need to reach out folks like Matt, Marshall, and Clint and see if there is some way I can get them to more passionately advocate for my books in their weblogs. Given that their visitors are more highly engaged than the “average visitor’, I have to believe their is an opportunity to sell more books.”

I took one look at the numbers in the last chart and thought, well, that doesn’t make sense. Sure, Matt’s visitors (or Clint’s or Marshall’s) are somewhat more engaged than the average visitor, but their “start the purchase cycle” numbers are pitiful. If Eric were to put effort into this, the place to put effort in is where both those metrics are strong.

Eric was good enough to send me the spreadsheet, and I pushed the numbers. (Well ok, technically he and I pushed the numbers at the same time over the phone …) On the phone, he called it “Robbin’s Metric” and I left it that way. It is the product of his Visitor Engagement and Percent Buy Path Sessions:

[ Ugh! Yes, I know that image is hard to read! I will correct ASAP!!! ]

By multiplying the two metrics and then ranking all the referring blogs by that metric, you see where Eric should put in extra effort. I agree witrh the first conclusion that Eric already came to in his blog, i.e. that he needs to work out some kind of deal with Anil. However, the two blogs where he should put time/effort would be Justin Cutroni’s and ROI Revolution’s. Interestingly, they are both Google Analytic blogs, so there is a decent chance that the reader is newer at analytics and probably could really benefit from Eric’s books. I didn’t highlight Steve Jackson’s blog, Xavier’s or Aurelie’s because they are already converting well (if there is such thing as converting well.)

Finaly thoughts: An engaged visitor to a site that is also content rich, like Eric’s, doesn’t necessarily make a good customer. In fact, Clint did a survey on his blog and saw that many of his visitors already own many, if not all, of Eric’s books. When visitors go to the beginning of the checkout, we can actually see interest in the purchase, as well as interest in the content — and as direct marketers know, you should always pursue the customer who already has a propensity to buy.

[ Thanks to Robbin for taking the time to take visitor engagement to the next level! What do you think? Is Robbin on the right track? Did I miss the mark? As always, your comments are greatly appreciated! ]

I got an email last night from Eric Enge who writes for Search Engine Watch. A conversation about web analytics that we had last month was recently posted. Eric asked me a variety of questions about Visual Sciences, making decisions based on data, uniquely identified users, content groups, some of the challenges associated with page tagging, and Avinash Kaushik’s 90/10 rule (which I disagree with due to the rule’s impracticality …)

If you have the time and inclination, give the interview a read, and thanks to Eric Enge for interviewing me.

A few weeks back, Gary Angel from SEMphonic published an oddly-titled post called “Why 100% Conversion is a Very Bad Thing” in which he calls into question the whole notion that a key performance indicator (KPI) is only good if a change in the indicator suggests a specific action that can be taken. Gary calls this kind of thinking “the myth of actionability” and says:

“The myth of actionability is conventional wisdom in web analytics – and it suggests that you shouldn’t report on anything unless changes in the measured value can be directly addressed by specific actions. In other words, if you can’t answer the question “What would I do if the value changed up/down?” then you shouldn’t report on the measure.This criteria is designed to eliminate lots of useless data from report sets and insure that what is in report sets has substantive value.

Unfortunately, I believe the criteria of actionability is unsound in almost every way: being both wrong-headed about the purpose of reporting and impossible to actually satisfy in the real-world.”

Obviously Gary is not one to pull punches. Unsound, wrong-headed, impossible … yowch!

Gary calls the myth of actionability “conventional wisdom” and I absolutely agree with him. Everywhere you go, when people are working out key performance indicators and building dashboards, the basis for inclusion or exclusion is usually “is there some action that a change in this metric will encourage us to take?”

Where does this kind of thinking arise? Well, let’s look at page 10 of The Big Book of Key Performance Indicators by Eric T. Peterson. In the section titled “What is a a Key Performance Indicator?” under the subsection on “Action”, in 2006 I explicitly stated:

“Key performance indicators should either drive action or provide a warm, comforting feeling to the reader; they should never be met with a blank stare. Ask yourself “If this number improves by 10 percent who should I congratulate?” and “If this number declines by 10 percent who should I scream at?” If you don’t have a good answer for both questions, likely the metric is interesting but not a key performance indicator.There is enough data in the world already. What most people need is data that helps them make decisions. If you’re only providing raw data, you’re part of the problem. If you’re providing clearly actionable data, you’re part of the solution. If you discover you’re already doing the latter (being part of the solution), give yourself a hug.”

Hmmm, it seems like I am one of the sources of “the myth of actionability” Gary is railing against. But it gets worse; while I was at JupiterResearch I published and presented a number of times on the subject of key performance indicators, and every time I talked about the subject, I stated unequivocally that the “core” of a good key performance indicator was it’s ability to drive action.

Remember, Gary said “Unsound, wrong-headed, impossible …”

Now, I don’t feel the need to defend myself, not because I disagree with Gary, but rather because I think Gary (and perhaps other folks) have taken the interpretation of “needs to drive action” to an unreasonable extreme. Let’s quickly have a look at the history of Eric T. Peterson’s guidance on key performance indicators:

• In 2004 in my first book Web Analytics Demystified, I wrote in Chapter 15: Bringing it All Together Using Key Performance Indicators that “the most common complaint about Web analytics data and the applications that provide said data is that there is simply “too much information”; too many graphs, too many charts, too many options, too many variables—too much for the average user to understand and make use of.“
• In 2005, in my second book, Web Site Measurement Hacks, I wrote in Hack #94: Use Key Performance Indicators that “the best KPIs are those that, when people look at them and realize that they’ve gone down from week to week, make people freak out and call meetings.” I also said, relevant to Gary’s complaint regarding the establishment of which indicators to use, “if you’re thinking about a number but cannot think of any action you would take if that number absolutely tanks, set that number aside.“
• In 2006, in my third book, The Big Book of Key Performance Indicators, I wrote “key performance indicators should either drive action or provide a warm, comforting feeling to the reader; they should never be met with a blank stare. Ask yourself “If this number improves by 10 percent who should I congratulate?” and “If this number declines by 10 percent who should I scream at?” If you don’t have a good answer for both questions, likely the metric is interesting but not a key performance indicator.“

If I’m wrong, at least I’m consistent huh?

When I first started pushing the idea that indicators needed to be tied to some type of reasonable action, my statements were a direct response to the dominant paradigm at the time: that all the information you needed to run your online business was contained in the hundreds of reports all web analytics applications generate, all you need to do is find the right data and take the appropriate action.

The problem I saw with this was, well, almost nobody was being successful with this strategy. Not only were most companies hamstrung and suffering from data overload leading to analysis paralysis, senior managers were asking for relevant data from the web analytics systems but not getting particularly satisfying responses from the people running the systems. Relatively boring metrics like “page views” and “visits” were being pushed up the food-chain, but except in rare cases, an increasing number of page views and visits were only loosely tied to increasing business success.

And so I proposed an Occam’s Razor for web analytics reporting, one that mandated that companies actually carefully consider the metrics bound for widespread distribution, and choose those metrics based on their ability to generate some action.

I never said, and I’m not sure anyone really says, the “actions” that would be taken were as granular and spuriously precise as “if this metric declines, reduce your PPC spending by 10% per 3% point decline observed.” Web analytics just doesn’t work that way folks, and here I agree with Gary when he writes:

No single measurement can ever suggest an action - cannot, in fact, even be interpreted directionally as either good or bad. Only in the context of a complete view of the business system (and the knowledge that all other things are equal or heading in some specific direction) can a judgement be made about the meaning of single measure. I think this make it clear that no one measure can ever really be “actionable” when taken in isolation. And if no one measure is actionable, then surely the criteria of actionability is fruitless.”

So let me clarify my position, as I am perhaps the high priest of the “cult of actionability”:

1. At design time, key performance indicators should be included or excluded from a hierarchical reporting strategy as outlined in The Big Book of Key Performance Indicators based on the likelihood that the indicator will spur some type of action in the organization when the indicator unexpectedly changes.
2. The action the organization would take, when unexpected change occurs, is never precise. The action is nearly always “conduct additional analysis” at which time the indicator’s definition provides at least the nominal basis for the starting point of the analysis.

At the end of the day, my view on key performance indicators is that they are intended to promote the visibility of web analytics throughout the organization, especially to the upper echelons where it is increasingly unlikely that traditional web analytics reports will be given the attention they deserve.

By creating a reasonable set of metrics and indicators, derived directly from the site’s business objectives and supporting click-stream activities, and then delivering said metrics throughout the organization with serious thought to definition, presentation, and potential for action, companies have been shown to significantly improve the level of attention given to web analytics data.

All of this helps to directly combat Gary’s observation that he too is “often disappointed in the report sets [SEMphonic] generate[s].” At the end of the day, regardless of which side of the fence you’re on, I believe we all agree that the central goal of web analytics is to help the business make better decisions. We do this by continually refining the web analytics business process and striving to better educate decision makers about the actions they can take to improve the web site. We repeat as necessary and hopefully go to bed happy.

Anyway, I’m a huge fan of Gary Angel so I hope we can continue this debate. What do you think? Am I crazy? Is Gary crazy? Or, like so much in our industry, is the reality something between the lines?

The nice folks at the American Marketing Association and Aquent are sponsoring a free webcast I will be giving on March 6th titled Web Analytics Demystified: Ten Simple Strategies for Using Web Analytics to Improve Your Online Marketing Efforts. Given recent conversations I’ve had with some of the best and the brightest about how companies actually use web analytics, this presentation will hopefully be very appropriate for the market today.

(Oh, well they say that the only bad press is no press at all, I kind of hope to not get the same treatment that Tom Davenport got from the Juice Analytics guys when he did a similar Aquent webcast …)

Register today for this free webcast!