Matt Belkin of Omniture: Web Analytics is Easy!
Matt Belkin of Omniture recently posted on a few of the pitfalls companies fall into when deploying web analytics. I was pretty surprised to see Matt, someone who was worked in this field nearly as long as I have, make the following statement:
“Analytics success is all about building a baseline for performance (your KPI trend), and trying new things to improve on this baseline. That’s it! That’s why I think it’s easy. I know other bloggers have argued that analytics is hard, but I’ve done this for a living and I can tell you that it’s not.”
Ironically enough I have been meeting many of Omniture’s largest customers recently, none of whom seem to think web analytics is easy. They universally have some difficulty associated with technology, people, or process—the triumvirate that is truly “web analytics”—and I suspect many of them had the same response I did when I read Matt’s statement above.
I quickly scribbled out the following response late last night but for some odd reason it has not been approved yet. I figured I’d post my comment here so that Matt and his customers would have a chance to read an opposing point of view.
“Hysterical! I talk to Omniture customers constantly who complain about how hard it is to do the most basic things like calculate bounce rate, integrate data using your Genesis platform, make sense of your reports, and even just get the data they need when they need it.
Perhaps the problem that you and people like Stephane Hammel are having with my statement is something called the echo chamber effect. You say something for so long, and your buddies all repeat it, that eventually you ignore the reality of the situation and begin to believe something that is clearly not true. Seth Godin accused me of doing this once (he was wrong, it turned out, people are deleting cookies … you’ve said so yourself!)
But you’re wrong, Matt. Web analytics is hard. Ask your customers, they’ll tell you.
It’s not just hard to improve your baselines, it’s hard to implement code properly, it’s hard to understand reports and definitions, it’s hard to find qualified staff to run these applications, it’s hard for HR to stomach the salaries we are asking for, it’s hard to train newbies, it’s hard to produce quality analysis based on only quantitative data, it’s hard to get management to listen, it’s hard to make management understand, it’s hard to select a good vendor when so many are failing, it’s hard to know if and when to migrate off of HBX, it’s hard to know which low hanging fruit to pick, …
You get the picture.
You make my point yourself in your post. If there are multiple versions of the truth, it’s hard to know who to trust. If there are multiple systems, it’s hard to know which system’s “click” is the right click to count. If you yourself have had to spend “countless hours trying to reconcile differences” in data, how is that “easy?”
In a way I’m happy you wrote this post because it reinforces everything I say when I travel the globe and meet with your biggest customers. They say “Our vendor says this is easy … there must be something we’re not getting.” I say, “Why would you expect your vendor to tell you that web analytics is hard? Would that make the sales process move forward more quickly? Would that make you more likely to buy their ever-expanding series of offerings? Would it make you think you won’t end up spending more money counting events, creating custom reports, or adding ad hoc segmentation tools?”
No. If you told the truth about web analytics, your prospects and customers would think twice about their investment. But that is exactly what companies need to do to be successful, really successful, with web analytics — take the science of audience measurement seriously!
When I say “web analytics is hard” I’m not saying that it is impossible, I’m not saying it’s not complex, I’m not saying that it is best left to the experts, and I’m not saying that companies should give up and go home. I’m saying that vendors, consultants, and customers should set their expectations regarding web analytics appropriately.
In my humble opinion, your customers need to know that web analytics is hard so they can:
- Plan to spend a reasonable amount of time determining their needs
- Allocate resources appropriately for implementation and deployment projects
- Set expectations with management about when results will begin to appear and what will need to be done with those results
- Make the case to management when they need additional resources, more software, or more time
- Have an appropriate relationship with their vendor, based on clear expectations
When people are told that “web analytics is easy” they take their investment for granted. They expect that a “standard implementation” or something that comes from a cut-and-paste template will serve their needs, that a 0.25 FTE will be enough to produce analysis, that results will be available in a matter of days, that the software they have will solve all their problems, and that they won’t need their vendor’s support from time to time.
In a way it’s ironic that you say “web analytics is easy” given Omniture’s obvious commitment to their customer’s satisfaction — Larry Freed of ForeSee Results taught me that satisfaction is a function of expectation; when you say “I’ve done it, it’s easy brah” then as soon as they realize the truth, you’ve failed to set their expectation correctly and thusly they’re unsatisfied.
With the increasing numbers of your customers experimenting with less costly tools, I would think that customer satisfaction would be your #1 priority.
I doubt you’ll publish this comment and I suspect you’ll be pissed off at me (again) for voicing an alternative viewpoint but consider this: I’m not saying anything bad about Omniture or any of the companies you guys are buying. I think Omniture is a great organization full of incredible talent. I think the market position you’ve carved out is enviable. I think you guys have tremendous potential to advance the market, driving adoption of Web Analytics 2.0, Web Analytics 3.0, and beyond.
“Web analytics is hard” isn’t about any vendor technology or any one person. “Web analytics is hard” is about your customers and their ability to use your technology and your guidance to their greatest advantage.
When you say “web analytics is easy” you’re oversimplifying what is involved in being successful with web analytics. When you say “I’ve done [web analytics] for a living and I can tell you it’s not [hard]” you’re not paying attention to what your customers are going through. When you say “from your perspective, it’s just not that hard” you’re demonstrating your intelligence but not your wisdom. In fact, your statement “Analytics success is all about building a baseline for performance (your KPI trend), and trying new things to improve on this baseline. That’s it! That’s why I think it’s easy” really says it all.
Suffice to say I was bummed to see this hyperbole and tired rhetoric in an otherwise insightful post.
Sincerely,”
I think it’s one thing when people evangelize for free products as an easy-to-learn entry point into the market, and another entirely when one of the market leading vendors makes such bold and (in my opinion) unfortunately misleading statements.
Our collective ability to be successful depends on having clear expectations, not false ones, and our satisfaction is a function of our expectations. I think Matt is setting the wrong expectation with his comments. What do you think?
Jeremy added the following ...
Eric,
I could not agree with you more. As an HBX client I spend much of my waking hours contemplating when and if our organization migrates to Ominiture’s Site Catalyst Tool. The statements made by Matt about analytics being easy makes me doubt the decision to pull the trigging and migrate to Site Catalyst. I believe Omniture is really setting themselves up to fail if they continue to evangalize how easy analytics is. I have seen Site Catalyst in action and let me tell you it is not easy!!!
Also, as the sole owner of analytic analysis and reporting for my organization I find Matt’s comments insulting. I believe those of us in the field should be very proud to work in an industry that is challenging and forever a moving target.
Keep up the good fight!
eric added the following ...
Karsten and Jeremy: Thanks for your feedback and encouragement! And thanks to everyone who has been sending similar comments in email.
I hope that we’re soon able to put this issue to rest and agree that on the surface some aspects of our practice are designed to be easy to accomplish, and that early on in our use of web analytics some things can appear to be pretty simple.
But the unfortunate reality (and I think Bryan Eisenberg made this comment in my blog awhile back) is that when we get really engaged and involved in trying to make people, process, and technology work together for the benefit of the business, “easy” is a word that is almost never used.
Jacques Warren added the following ...
Anyone out there who’s found a $500,000 opportunity on their web site in the last three months, thanks to web analytics, has earned the right to call it “easy”.
The rest should keep sweating…
eric added the following ...
Jacques: I might misunderstand you. I’ve been doing web analytics a long time too, and I have found countless million dollar opportunities at my clients. In fact, I think ** finding ** big revenue opportunities is quite easy … but making them manifest into real money, that’s the hard part.
Let me tell you a story: When I was a little younger (and had less gray hair) I did a very good analysis for an HBX client in the technology retail sector. They had a fairly obvious problem in their conversion funnel, and using web analytics I was able to easily quantify the value of the leak. I estimated that fixing the leak would net them roughly $1M ** per day **.
My CEO at the time and I traveled to their corporate offices to present this information. I gave what I thought was a very compelling case for making a change; I gave them reams of data, and I even gave them a nice little spreadsheet they could use to make the calculation on their own (thusly changing some of the assumptions.)
The senior executives we presented to said it was very good work. They were a little embarrassed that we were able to find such a large problem so easily. Even with conservative assumptions, this looked like roughly a $300M correction they could make, largely based on good old fashioned web analytics.
Web analytics is easy.
Unfortunately, it took this company well over a year to make the change we recommended. In the meantime, they continued to hemorrhage money through their conversion process. We all sat and watched the money flow to their direct competitors — every day the numbers said the same thing.
When they finally made the change, miracle of miracles the money appeared! But it turned out we were wrong about the $1M per day estimate — we were too conservative and the amount of incremental revenue was even greater.
It took them over a year to discover this. Over a year to make a change estimated to make them an incremental $1M per day. More than 365 days of wasted opportunity.
Web analytics is hard.
If you want to take a simplistic definition of web analytics, as many of those so passionately arguing against me clearly are, then yes: web analytics is easy. If web analytics is little more than ** finding ** problems (not saying you disagree with me Jacques) then yeah, that’s easy. If web analytics is little more than producing pretty graphs in AJAX then yeah, that’s easy. If web analytics is nothing more than getting some code on pages, yes, that is sometimes easy …
But that is not web analytics. Web analytics is a complete discipline (in my opinion) that spans well beyond data collection and amazing wonderfulness of graph production: web analytics is complex, and in that complexity there is a lot that can go profoundly wrong along the way.
Thanks for your comment and as usual I look forward to seeing you in San Francisco!
Jacques Warren added the following ...
Hi Eric,
Well, I was being sacarstic; you know I share your view about how hard Web analytics is. I have grown quite tired lately with all the discussions about tools, etc.; I think our field is currently running the danger of being perceived as underdelivering.
I think Web managers should really try hard to refocus their attention to finding opportunities right now.
I, too, have witnessed what you’re saying. One of the things that have puzzled me most in the last year is how slow, or ineffective, organisations are with acting on the discoveries our analyses make. It almost defy the purpose of being online. But I guess 500 years of print have created habits, in particular the idea that what is “printed” is permanent.
I would say that one of the biggest challenges of our field in 2008 is to create the necessary processes that will ensure those investments produce spectacular ROI.
I know that is at the heart of what you do, and you know I was quick to detect that WAD’s business proposition is presently amongst the most relevant ones.
eric added the following ...
Jacques: Oh man, you should have used the <GRIN> tag in your comment! ;-)
I look forward to seeing you in San Francisco and continuing this discussion in real-time. I see we are in heated agreement.
Everyone: I have been getting a ton of really great feedback from many of you via email, especially from Omniture clients and people who have worked directly for Matt Belkin (and apparently still do.) Suffice to say I appreciate the kind feedback and respect your desire to keep your names anonymous.
That said, and this is ** not ** directed at Matt or Omniture, I think if you have a complaint about how your vendor positions themselves in the marketplace, you should voice your complaint directly. I think the vendors actually want to hear these kinds of concerns — and they really need to know how their biggest customers perceive their statements and actions so they can work diligently to improve.
Anyway, I already knew about the “Eric Peterson dartboard” but it’s still funny. Thanks for all your comments!
Steve Jackson added the following ...
I just had one meeting this morning with a guy running Visual Site (Or Omniture Discover 2 or whatever it’s called now) and he said exactly the same thing.
It was sold so well as a problem solver and yet it’s actually created far more problems. It’s as you say a lack of process and people around the tool which is the problem, not the tool itself.
It’s unfortunate that some sales folks don’t make that clear (not all, but some).
Fernando Mladineo added the following ...
As one who is new to web analytics and trying to find ways to implement it into my hobbyist sites, I would agree that web analytics is difficult. Granted, my sites were not built to be profit generating, so it’s been a little tricky to create conversion goals.
Having worked with Site Catalyst a few times (Omniture sponsors competitions at my university–BYU), I would say there is a fairly steep learning curve. That said, it is a powerful tool that can provide some very insightful information. The trick is, you need to be willing to invest a significant amount of hours before seeing results.
eric added the following ...
Steve: I’m sorry to hear that! Obviously I think that Visual Site is a fantastic product, but I agree with you — until there is a more honest and direct conversation about what it takes to be successful with web analytics, companies investing millions in technology and resources run the risk of being disappointed.
Fernando: That is all I’m saying: you need to understand the effort necessary and be willing to spend the time. I’m not saying web analytics is impossible, undoable, or not worth the effort — not at all.
And I agree, all of these applications can provide tremendous knowledge and depth of insight about your visitors — provided you spend the time to learn what you’re doing.
Thanks to both of you for your thoughts!
Web Analytics is easy...and hard. | davehamel.com 3.0 added the following ...
[…] of the worlds leading experts, Avinash Kaushik and Eric Peterson have been having a long term discussion on whether analytics is easy or hard. Am I going to answer […]
John Wyllie added the following ...
Matt should be glad web analytics is not easy! If it were, I’m sure there would be very little need for his Best Practice Consultancy team at Omniture to exist - as all of Omniture’s clients would be sipping Margaritas on the beach after clocking-off from their “easy” web analytics role at lunchtime.
I think the fact that there is a Best Practices team at Omniture, plus so many Consultants in the market demonstrates just how complex this industry is.
Another Day, Another Donut « Bold Lentil added the following ...
[…] may or may not be reducible to a mean satisfaction score. Satisfaction also might be a function of expectation. And maybe satisfaction is the death of desire. Who […]
Soeren Sprogoe added the following ...
I’ve definately found that the more I work with Web Analytics, the harder it gets.
In the beginning everything was simple, because I didn’t know the complexity of things. Now I know, and I find both the difficulty, but certainly also the opportunties, of web analytics greater than ever!
Man, I wish I was back at the beginning again. What a simple, careless life… Full of opportunity, rainbows and butterflies :-D
michael choe added the following ...
i agree with matt belkin partially. i don’t think web analytics is as hard as avinash kaushik or eric peterson says. the hard part is defining your business objectives, ‘building a baseline for performance’. implementation of web analytic software may be tedious and time consuming but not necessarily difficult.
my 2 cents.
Web Analytics Demystified » Blog Archive » Free IndexTools: Analysis and Market Implications added the following ...
[…] would I say that? Simple, I have been running IndexTools on my web site for the past six months. IndexTools is not easy, but it is no more complicated than anything else out there (IMHO.) If you look around at comments […]
Web Analytics Demystified » Blog Archive » It is official: IndexTools is now free for everyone! added the following ...
[…] Yahoo! has a lot to consider before they roll IndexTools out to the masses. I mean, if you think web analytics is hard, you should try developing, maintaining, selling (or not selling), and supporting a web analytics […]
Karsten added the following ...
Hi Eric,
you are so right. I fully agree. “We” consultants have to deal with expectations set too high, and buyers of analytics software still search for the button with the questionmark telling them what they have to do to improve their website. “Now that I have it implemented, why is my conversion rate still so low?” … To set realistic expectations, vendors should ask their prospects how the HR situation is, and tell them that software alone does not do the job. But I doubt this will happen.
Web Analytics is hard, and … the longer I work in the field of web analytics, the harder it gets, as we are facing new technologies (like ajax, rss, …) and new devices (like iPhone and other mobile access) faster than we can adopt measurement technology, KPIs and analysis.
And that is also why I like your blog and your ongoing efforts to “tell the truth” …
Thanks, and keep it up,
Karsten