Web Analytics Demystified

Archive for August, 2009

White Paper: Testing Secrets of Success

If you’ve been reading my blog for any amount of time you’ve inevitably heard me comment that I think “web analytics is hard” — not complex, not mysterious … just plain difficult. It’s hard to select vendors, hard to install code, hard to train users, hard to get the “right” reports, hard to get management’s attention, hard to make the case for change … the list goes on and on (and on and on and on!)

Hard, but not impossible.

In the past few years we have definitely started seeing an increase in the number of companies that “get it” — so much so that we’re able to program an entire conference built around the superstars of web analytics. More and more I am talking to, working with, and hearing about companies who have leveraged the “web analytics is hard” mindset to properly set expectations regarding their use of technology, their deployment of people, and their use of business process to really excel at web data analysis and turn this analysis into tangible improvement for both the business and their customers.

As I collect more information about these web analytical competitors one thing that nearly always emerges as a hallmark of their success is some type of structured testing program. Of course this makes perfect sense because without testing the analyst is never really sure about the impact of their recommendations, so much so that I have often said “if you’re not testing, you’re not really doing web analytics!”

But the increase in testing has raised an interesting corollary to my “web analytics is hard” manifesto … testing is hard too!

Fortunately we’re all a little bit older and a little bit wiser this time around and we recognize that testing requires more than just throwing code on the page and clicking the “Optimize” button. Testing is a process that requires people and technology … sound familiar?

I’m bringing this up for two reasons:

  1. At the X Change conference in San Francisco on September 9, 10, and 11 Matthew Wright from HP will be leading a conversation titled “Testing, Testing, Testing: Building Consensus and Evaluating Results” to discuss the nuances behind testing, things like getting stakeholder approval, planning, and clearly defining measures of success;
  2. This morning the nice folks at SiteSpect published a white paper I wrote that details ten “best practices” for testing that I think a lot of folks new to testing often forget. Titled Successful Web Site Testing Strategies: Ten Best Practices for Building a World-Class Testing and Optimization Program, this white paper is freely available (requires registration) and covers nuances like testing teams, stakeholder involvement, test plans, timelines, and, of course, measurement.

If you’re working to become an analytical competitor and join the ranks of the kinds of companies who get invited to lead a conversation at X Change I highly recommend either grabbing the testing white paper, coming to the X Change, or BOTH! Especially if you’re serious about testing I think you’ll find this free white paper useful when you work to set expectations, and trust me, testing is as much about expectations as it is execution!

Register to download the free white paper on successful multivariate testing strategies!

Interview: John Lovett from Forrester Research

Following up my interview with Bill Gassman a few weeks ago I realized that I would be remiss if I didn’t build on Forrester’s recent Web Analytics Wave report with an interview with John Lovett. John, like Bill, totally, totally understands the web analytics industry, and in that understanding is able to clarify the marketplace in a way few others can. Don’t believe me? Check out his response to possibly the worst article about web analytics, ever. Measured, polite, even complimentary … that’s John.

I am personally honored that John accepted my invitation to return to the X Change this year and both lead the huddle on “Industry Standards (or a lack thereof)” and co-lead a huddle on technology with Bill Gassman. If you haven’t met John personally, and if you are able to join us at the X Change, I strongly recommend you make a point of introducing yourself to him.

Finally, before my questions and John’s answers, I wanted to point out how incredibly deft Mr. Lovett really is: in response to a high-and-hard fastball question about “which vendor is really the best,” John knocked the ball clear out of the park with his answer: none of them. I’ll let you read the rest for yourself …


Your recent Wave report really emphasized a lot of conventional wisdom about the web analytics vendors but had some surprises for folks.  What surprised YOU the most about the Wave results?

Well Eric, I like to say that surprises are for birthdays and not for business. So in terms of actual surprises, there weren’t any big bombshells for me. I was however pleased that the vendors demonstrated innovation in a number of areas (like social media measurement) and that despite my attempts to develop extremely challenging criteria, the vendors continue to improve year over year.

One comment people have made to me is that they question the validity of comparing fee and free solutions in a single matrix due to the fundamental differences in their business model.  How would (or do) you respond to that challenge?

That’s preposterous! I respond by saying that it’s negligent not to compare free vs. fee based solutions. In today’s economic environment if you’re not watching expenses by understanding the cost to benefit ratio of your Web analytics solution, you are acting irresponsibly. Free tools have merit for many organizations as both primary and secondary tools, while fee based solutions are more appropriate for others based on their capabilities. Organizations must do their diligence to understand what they need in a Web analytics solution to decide what’s right for them, which is really the insight the Wave attempts to provide.

I asked Bill Gassman from Gartner a variation on this question recently, but do you now or see in the near future a situation where you as a Forrester analyst are advising your clients to actively consider these free solutions in addition to “traditional” web analytics solutions from Omniture, Coremetrics, and Unica?  As a follow-up, how do you see free tools impacting the market in the next 12 to 24 months?

I advocate that a single system for measurement is always the best way to go, yet recognize that this isn’t always feasible. Duality of Web analytics tools is a reality for myriad reasons. Thus, company’s need to manage their data dissemination practices to ensure comprehension and mitigate doubt. This is tricky, but certainly possible. I often help clients determine which solution is best suited to meet their needs and financial implications are always a part of that discussion.

With regard to how free tools will impact the market: we are just witnessing the beginning of the incoming tide on this one. By this I mean that “free” will continue to disrupt the market by placing pressure for improvement on all vendors. Just look at the recent Webtrends product upgrade announcement – the majority of press around it cited a “look out Google Analytics” slant. Why the comparison…they’re worried! Fee-based vendors have even more to fear now that Yahoo! Web Analytics opened up its partner program.

Another comment I hear about the Wave results, and forgive me this, is that they’re lame because they do nothing to differentiate the “market leaders” who appear as a tight cluster.  The evidence cited is that all four vendors issued press releases declaring their “market leadership” which appears technically correct based on the Wave but as the Highlander said, “There can be only one.”  First, how do you respond to this and second, who is the real market leader in web analytics?

Here’s the dirty little secret – the real market leader is the wildly talented Web analytics practitioner. It’s not the tools that differentiate it’s the craftsman. Any company that believes the Web analytic technology alone will make them incredibly successful is delusional or just plain out of touch. There is no get rich quick scheme here. Each of the leading vendors on the Wave offers a highly customized solution that can be tricked-out to meet nearly anyone’s individual needs. But this takes a great deal of work. For those organizations that are looking for the far-and-away winner in this technology category, guess what: the tools will only get you so far – you need talented people to really make it happen.

Rumors are that Omniture has a bunch of “800 lb gorillas” hanging in their offices right now.  Clearly they’re proud of their position, but last quarters results highlighted that there are clear risks to their business that are beginning to manifest.  What do you think are the greatest risks to Omniture’s business over the next 18 months?

Well, I don’t buy into rumors and sure don’t know where I left my crystal ball. But things are tough all over. As I stated earlier, free solutions are threatening all fee-based vendors and forcing them to work harder. I can tell you that measurement technologies are an imperative for executing on digital marketing endeavors. Solutions like Omniture’s, Webtrends’, Coremetrics’, Unica’s and everyone else’s will continue to play an important role in the evolution of organizations conducting business online. I believe that Web analytics is increasingly becoming an integrated service and expect to see things evolve to easier access to data through new and alternative means. The leading vendors, including Omniture, will play a role in this evolution.

What’s your taken on the current hype cycle around “open”?  Omniture bangs the Genesis drum, Coremetrics connects, and now WebTrends appears to have decided that “open” will be the foundation of their future success (or lack thereof) … but some people think that “open” is a check-box requirement, not a competitive differentiator.  What do you think?

Open is not a feature, it’s a philosophy. The ability to get data into and out of a Web analytics solution is the crux of the issue and leading vendors facilitate this through bi-directional API’s, other import and export functions and data dissemination capabilities. Webtrends is currently doing this as well as anyone, but “open” also means talking to your customers about development plans, listening to criticism and demonstrating a willingness to change. These qualities aren’t unique to Webtrends, they’re characteristics that all vendors should exhibit. Webtrends is just marketing around them and if that’s causing people to want open, then it appears to be working.

As a previous attendee to the X Change what do you like best about the conference and what would you like to see us change this year or next?

I appreciate the intimate conversational format of X Change. The huddles really facilitate deep thought, controversial leeway and provocative discussion. As someone who attends a number of conferences, it is refreshing to engage in dialogue with individuals who are passionate about what they do and to initiate a true collaborative thinking environment. As far as change goes, I really hope to be able to guide the huddles that I’m leading toward resolution. Within our industry, all too often we surface problems and issues without identifying solutions. I’ve taken your challenge to heart and hope to walk away with some tangible results from my huddles.


John will be joining Bill Gassman, Gary Angel, June Dershewitz, and over 100 expert users, consultants, and vendors at the 2009 X Change conference in San Francisco on September 9, 10, and 11. Registration is currently underway and we’d love to have you join us! For more information please visit:

http://www.xchangeconference.com

How much do you pay for web analytics?

I was just cruising through the just published WebTrends 9 update and thinking about how the web analytics vendor market is evolving. “9″ looks neat and I’m sure glad to see some really important metrics like bounce rate appear in the UI. Still, I always scratch my head when I see vendors make statements like “[the] data visualization tool in Webtrends Analytics 9 lets anyone – even analytics novices – quickly and easily understand changes in key metrics” and then put up a feature list like this one.

Still, it’s nice to see WebTrends making some moves so congratulations to Jascha, Casey and the entire Portland crew for getting the update out the door!

Anyway …

I said I had been thinking about the evolution of the web analytics vendor market. A lot of my thinking this past week has been colored, well, purple, thanks to the announcement of Yahoo’s Web Analytics Consulting Network (the YWACN or, as I think about it “the Yack’n!”.)  On July 30th Yahoo announced that they were making Yahoo! Web Analytics much easier to get through 48 partners around the globe.

Now, when you look at the partner list you might not recognize a lot of the names — I sure don’t — but a few should stand out. Specifically Stratigent, Semphonic, Sapient, and my own company Web Analytics Demystified. While I can’t speak directly for any of these companies, all are run by very smart people, and I have to wonder if they’re not thinking about YWA much the same way I have been.

I mean, if you think about it, Yahoo! has basically come to us and said “Go sell excellent implementations of YWA and provide awesome ongoing support” for an application that, according to Forrester Reseach, has 77% of the core functionality of Omniture SiteCatalyst. Or, put another way, “Find companies that are struggling to get value from their existing investment in {pick a vendor}, kick that vendor out, and then make money helping them be successful for less then they spend today.”

Sweet, thanks Yahoo!

Not to brag (since it was pretty obvious) but I did say this would happen back in April 2008 given the hard work Google Analytics (who is ironically NOT a YWA competitor) had done with their similarly badly acronym’d GAAC. Yahoo wisely avoids having to support customers directly, leverages some incredibly smart folks, and lets companies reduce their annual analytics spend without having to forgo core functionality like multiple custom variables and visitor-level segmentation.

Hell, we’re not even talking about real-time updates and demographic reporting and segmentation, which while the former often has more value ascribed than necessary the latter, if I can say so based on my own usage, is pretty fantastic and not available in any other web analytics application in the market today. I mean, who would have guessed that so many mature, responsible adults love to Twitalyze themselves!

Now I sincerely doubt that any of the YWACN members are going to suddenly stop supporting the big for-fee applications out there … I know I’m not! And I fully expect the adoption of YWA to be slow and methodical (mostly because of existing contracts, Yahoo’s terms of service, and the fact that Yahoo is somewhat limiting YWACN access to new accounts although I think their strategy is fair and makes perfect sense.) But at the end of the day Yahoo has made quite possible the single best move they could have if their goal was to provide an awesome service with excellent third-party support at the best possible price.

Now if you were paying attention you may have noticed I commented that Google Analytics and Yahoo Web Analytics are not competitive. Crazy, huh? But they’re not. Google Analytics (as it exists today) and Yahoo Web Analytics (as it exists today) serve two near completely distinct target markets.

Now I know I’ll get heat for saying this (again) but I just don’t think Google Analytics is appropriate for “free standing” use within the true Enterprise. I’ll point again to Bill Gassman’s recent note on the service (which I thought was excellent) and will obviously concede that it is well within Google’s power to make GA the “bestest, most Enterprisey” web analytics application the world has ever seen … but it isn’t today. More importantly when I go looking for companies mature in their use of web analytics who rely exclusively on Google Analytics and have chosen to do so explicitly, I simply don’t find them.

I could be wrong — if you’re an analytics samurai using nothing but GA please let me know –  but what I see a lot of is mature businesses using Google Analytics to back-fill some limitation in their for-fee vendor’s service. For example, up until today it was amazingly difficult to get WebTrends to calculate a bounce rate and some people think setting up visitor segments in SiteCatalyst is a lot of work. More importantly, while lots and lots of people complain about how difficult their analytics application is to use, the team at Google has done a freaking brilliant job with the GA user interface and in my humble opinion it sets the bar for ease-of-use in web analytics.

Yahoo Web Analytics in an Enterprise context, and hopefully Dennis will forgive me this since he’s tanned and rested after a week or two in the islands, is not really that easy to use, not that easy to set up, and not that easy to configure — remember it’s 77% of Omniture SiteCatalyst which nobody ever describes as “easy to implement and easy to use” (except for Adam Greco, but he’s clearly an exception!)

But here’s the secret: Yahoo Web Analytics is not supposed to be easy to use, it’s supposed to be really, really powerful! Yahoo Web Analytics is an Enterprise-class web analytics application out of the box designed to support businesses with custom data collection needs, custom reporting needs, custom segmentation needs, and the challenges typically found within any company of size.

More importantly, because of this functionality I believe that Yahoo Web Analytics will be a gateway to a much deeper relationship between the YWACN and their customers than the GAAC have found for the most part.

Yes, Yahoo’s APIs are tightly held and thusly YWA is not as “open” as WebTrends or as “integrated” as Omniture. Yes, Yahoo is keeping Rubix under wraps so it is not as flexible as Affinium NetInsight or Coremetrics Explore. Yes the interface is kinda clunky and the terms of use were written by lawyers … I get the complaints and hear the FUD loud and clear. But given the massive adoption of Google Analytics I think that coupling exceptional services and support with a free Enterprise-class application has a lot of potential to be the permanent game changer that I first described last year.

What do you think? Is purple the new green? Is “9″ too little, too late? Does Yahoo! have a chance to focus now that they have outsourced their search business? Or am I missing the point and despite two great free solutions will the world continue to pay for web analytics the same way we always have? I’m totally willing to be wrong about this one … but if you don’t believe me about how powerful Yahoo Web Analytics is either read this book or contact me directly and I’ll see about getting you your own YWA account.

As always your comments are welcome.

 
COPYRIGHT © 2010 WEB ANALYTICS DEMYSTIFIED, INC. ALL RIGHTS RESERVED. PRIVACY POLICY