Registration is now open for ACCELERATE 2014 in Atlanta, Georgia on September 18th. Reserve your spot today at Eventbrite — tickets are only $99 USD!


The Recent Forrester Wave on Web Analytics … is Wrong

Published by Eric T. Peterson on May 16, 2014 All posts from Eric T. Peterson

Having worked as an industry analyst back in the day I still find myself interested in what the analyst community has to say about web analytics, especially when it comes to vendor evaluation. The evaluations are interesting because of the sheer amount of work that goes into them in an attempt to distill entire companies down into simple infographics, tables, and single paragraph summaries. Huge spreadsheets of data, long written answers, and multiple calls and product demos … all munged down into a single visualization designed to tell the large Enterprise which vendors to call and which to avoid.

In the early days of web analytics having access to these evaluations could be a huge time-saver. At the time there were dozens of vendors all embroiled in a battle for market-share, and so the vendor summary provided an “at a glance” view of the landscape that had the potential to save the Enterprise time and money. Plus, during the early growth period in web analytics, no one vendor had hegemony over the market and so any errors or inconsistencies in the results could easily be swept under the rug based on this being “an emerging market …”

Today, however, the web analytics market is functionally mature, and two vendors have emerged as “market leaders” based on their particular strengths and business models. I don’t even have to tell you who these vendors are; if you work in this industry or you are paying any level of attention to the technology landscape, you already know who they are … and who they are not … which brings me to the main topic I wanted to discuss:

The most recently published Forrester Wave on Web Analytics (Q2 2014) authored by James McCormick is wrong.

You can get a free copy of this report from Adobe, and I would encourage you to have a look yourself, but based on hundreds of implementations, vendor evaluations, RFP processes, and thousands of hours of work on our part, the Partners at Web Analytics Demystified and I can assure you that only one of the vendors dubbed a “leader” in this document is truly leading in the market today. Additionally, another vendor labeled a “strong performer” has consistently demonstrated more leadership and commitment to digital analytics than any of the vendors evaluated.

[At this point you may be asking yourself "why isn't he naming names?" ... which is a fair question. The old me was kind of a dick; the new me is trying to be less of a dick. I suspect that I am doing a poor job at that, but I am trying ...]

I would encourage you, if you are interested, to review the scoring for the Wave reported in Figure 3 on page 9 … and ask yourself “do these results and, more importantly, these weightings, make sense?” For example:

  • A zero weighting for “Market Presence” … despite the fact that two vendors have an increasing lock on the market in 2014, especially when you look at wins and losses in the last twelve months.
  • The “Product” and “Corporate” strategy … which to me seem arbitrary at best, reporting that Google’s product and corporate strategy is “average” while that of a company that is on their third CEO and umpteenth head of Marketing is second only to A) a true market leader who is tied with B) a behemoth who is buying great companies but struggling to retain key employes who truly understand the market.
  • “Application usability and administration” … reporting that again Google is behind a vendor who has not updated their core analytics application for an estimated ten years.
  • The inclusion in the report of not one but two vendors whose names have not come up in Enterprise web analytics circles for years …

Take a look when you have a chance and see what you think. Maybe I’m the one who is wrong, and perhaps after 100+ collective years in this industry it is my Partners and I who have completely lost our connection to the web analytics vendor landscape …

At Web Analytics Demystified we rather enjoy the mature technology market we are working in today. With our clients increasingly standardizing on one, the other, or both of the true market leaders, our ability to move beyond the technology to how the technology is used effectively and efficiently in the business context is made that much easier. When analytics is put to use properly … good things happen.

I welcome your comments and feedback.

Share this blog post ...
Twitter71LinkedIn104Facebook14Google+33Email

Categorized under General Web Analytics, Vendors, Web Analytics Demystified Business

  • Andrew Garibay

    I agree with your sentiments regarding Leaders and Strong Performers. I believe that in the right hands, either of the two are capable of pulling tremendous insights, especially with a bit of programming experience. I don’t have a tremendous amount of experience with AT Internet or SAS Institute’s offerings, but I would argue that the tool is less important than the analyst having a sound understanding of terms/definitions, setup (execution), the questions they are asking, and whether or not their analysis accurately addresses those questions.

  • http://www.webanalyticsdemystified.com Eric T. Peterson

    I’d like to see a show of hands … who does have tremendous experience with AT Internet (dubbed a Market Leader by Forrester) …

    I’ll just be here waiting for the legions to come to AT and SAS’s defense. Can someone bring me some water after four or five days? Thanks. ;-)

  • Ole Bahlmann

    I can’t speak for SAS but I have worked with AT Internet for the last 3 years and was a happy customer (I recently changed jobs so I’m not personally using it anymore). I would agree that their strength is more on people and consulting than on the current tool. But having seen and used their upcoming new products I’d guess those are already positively impacting this study.

  • http://www.webanalyticsdemystified.com Eric T. Peterson

    Ole, thanks, but then we either have A) Forrester up-weighting the least scalable aspect of vendor offerings (people and consulting) or B) giving credit for new products that have not yet been released into customer’s hands.

    Either way I still do not believe that AT Internet is a “market leader” in web and digital analytics …

  • campbellmacdonald

    Eric:

    Great article calling out what is clear to everyone else. You stopped short of asking “How does Forrester get results like these?” Look at the “relationship building” that each of these companies do with Forrester (attending their conferences, sponsoring conferences, buying speaking spots at conferences, buying reports like these to share with prospects, etc) all in the name of providing objective assessments of the industry.

  • http://www.webanalyticsdemystified.com Eric T. Peterson

    I stopped short because I don’t personally have any evidence that the “relationship building” has any impact on inclusion/placement in the Wave.

    I never worked at Forrester, but during my tenure at Jupiter Research it was clear that it was not a “pay to play” model and we treated all vendors equally in the web analytics constellation. I also made sure that I fully understood the market dynamics and spent a ton of time “ground truthing” the results the model gave back so that we could be confident that we had painted an accurate picture of the landscape at the time.

    Because, you know, sometimes the models … are wrong.

  • campbellmacdonald

    Fair enough. I think the broader point is: what is the methodology for inclusion/placement in the Wave? They have to look at “objective” measures like revenue, # of customers, etc. And then they need a feature check box. Whereas some software that is exploding in usage and easy to use (e.g. Optimizely) makes me question the approach.

    Models are wrong? Or is it just data that’s not well behaved?

  • campbellmacdonald

    Fair enough. I think the broader point is: what is the methodology for inclusion/placement in the Wave? They have to look at “objective” measures like revenue, # of customers, etc. And then they need a feature check box. Whereas some software that is exploding in usage and easy to use (e.g. Optimizely) makes me question the approach.

    Models are wrong? Or is it just data that’s not well behaved?

  • Stephane Hamel

    @erictpeterson:disqus – we’re on the same page! :)

    My point of view wasn’t as elaborated as yours (I recognize you are more elloquant that I am!), but see what I wrote at https://plus.google.com/106091905429925862702/posts/GfeZstmQb6y

    I think Forrester is loosing credibility and trust with this report…

    @campbellmacdonald:disqus : one of the criteria for inclusion is to have revenues of over $20M… which weeds out most of the other tools that are a little more “niche” but very interesting.

  • http://www.webanalyticsdemystified.com Eric T. Peterson

    Thanks Stefan.

  • Robert Jackson

    The relationship between vendors, research companies, brands and consultants is an absolute game of thrones. Forrester have for me, seriously damaged their credibility by downgrading Google. I remember a similar situation last year when a leading industry figure promoted a white paper for a Tag Management Solution which lauded features which just didn’t exist.

  • Sergio Maldonado

    This is fun :) Well put.

    Now, Does it really come as a surprise? How many years have we seen a couple of old timers ranking high on the Wave and not raised a hand? (we all here have deployed and maintained all of these tools extensively… we were all well aware of the differences!)

    I guess it gets more surreal over time, as there is no “wave” when you only have 2 players.

    In AT’s defense I will say they have owned the French market for many years. But Webtrekk is even stronger in Germany and there is not a trace of them on the report.

    Seriously, Forrester should pull the plug on this particular Wave.

  • Roman Rock

    I am not so much shocked by the fact that GA is a strong performer depending on the context that you are using to grade the relevant platforms. In my mind there is a leader in this market space and that is Adobe there rest trail them by a distance. Trying to review just the product on it is own is not how most of us would select an analytics platform in this day and age. You want tools at easily integrate with other platforms with low effort and cost, this is where platforms such as Webtrends would barely even be a contender for me. They charge for everything, integrating is complex and it offers almost no flexibility with limited data visualisation making it almost impossible to drive a data adoption. Compare this to Adobe where most integration as drag and drop, on the fly correlations and advance segmentation, simple visuals and cloud integration, how are they even close?

  • Pingback: Forrester의 Web Analytics 벤더 평가와 반론 | Be Advertising Scientist

  • Arpit Srivastava

    No doubt both the giants are making a steady progress I remember when we internally decided to go with the Adobe’s enterprise web analytics offering 2.5 years back, GA was no where in terms of custom dimensions, visitor segmentation, attribution but as on date they have moved on to pretty robust offering and I think lot of these have been derived from Adobe & the rate at which GA is building is perhaps higher then Adobe. But with feature additions & customization comes the complexity. And the end of day we need good team technically & analytically strong enough to utilize these tools optimally.

  • http://www.webanalyticsdemystified.com Eric T. Peterson

    Approve—
    Eric T. Peterson
    (858) 663-6079

  • vbhagat

    Eric – I agree with your sentiments and many of those voiced on this thread. Today, we published an alternative Digital Analytics Software Rankings based upon 230 authenticated reviews by real end-users. We take a market-segment view – small businesses, mid-sized companies, enterprises, and remove subjectivity by plotting vendors on two simple axis – likelihood to recommend based upon aggregate ratings by customers in that segment vs. estimated adoption (based upon Builtwith.com website usage data) in that segment. For adoption rates, we use data for penetration of top 10k websites for the enterprise, 100k for mid-mkt and 1m for SMB. We also present the pros/cons of each product based upon a distillation of what’s said in reviews. We believe that rankings can help narrow your list, but at the end of the day you need to find the right tools or combination of tools for your use case. Our findings contradict Forrester’s in several respects.

    Here’s our blog post on the vendor rankings: http://blog.trustradius.com/2014/09/trustradius-reveals-top-rated-digital.html

    Article published today by CMS Wire – In Web Analytics Ranking World, Another Choice Emerges
    http://www.cmswire.com/cms/customer-experience/in-web-analytics-ranking-world-another-choice-emerges-026383.php

 


Recent Blog Posts

Top 5 Metrics You’re Measuring Incorrectly ... or Not
Eric T. Peterson, Senior Partner

Last night as I was casually perusing the days digital analytics news — yes, yes I really do that — I came across a headline and article that got my attention. While the article’s title ("Top 5 Metrics You’re Measuring Incorrectly") is the sort I am used to seeing in our Buzzfeed-ified world of pithy “made you click” headlines, it was the article’s author that got my attention.

Continue reading this article ... ... more from Eric T. Peterson

Bulletproof Business Requirements
John Lovett, Senior Partner

As a digital analytics professional, you’ve probably been tasked with collecting business requirements for measuring a new website/app/feature/etc. This seems like a task that’s easy enough, but all too often people get wrapped around the axle and fail to capture what’s truly important from a business users’ perspective. The result is typically a great deal of wasted time, frustrated business users, and a deep-seated distrust for analytics data.

Continue reading this article ... ... more from John Lovett

Welcome to Team Demystified: Nancy Koons and Elizabeth Eckels!
Eric T. Peterson, Senior Partner

I am delighted to announce that our Team Demystified business unit is continuing to expand with the addition of Nancy Koons and Elizabeth “Smalls” Eckels. Our Team Demystified efforts are exceeding all expectation and are allowing Web Analytics Demystified to provide truly world-class services to our Enterprise-class clients at an entirely new scale.

Continue reading this article ... ... more from Eric T. Peterson

When to Use Variables vs SAINT in Adobe Analytics
Adam Greco, Senior Partner

In one of my recent Adobe SiteCatalyst (Analytics) "Top Gun" training classes, a student asked me the following question: When should you use a variable (i.e. eVar or sProp) vs. using SAINT Classifications? This is an interesting question that comes up often, so I thought I would share my thoughts on this and my rules of thumb on the topic.

Continue reading this article ... ... more from Adam Greco

5 Tips for #ACCELERATE Exceptionalism
Tim Wilson, Partner

Next month’s ACCELERATE conference in Atlanta on September 18th will be the fifth — FIFTH!!! — one. I wish I could say I’d attended every one, but, sadly, I missed Boston due to a recent job change at the time. I was there in San Francisco in 2010, I made a day trip to Chicago in 2011, and I personally scheduled fantastic weather for Columbus in 2013.

Continue reading this article ... ... more from Tim Wilson

I’ve Become Aware that Awareness Is a #measure Bugaboo
Tim Wilson, Partner

A Big Question that social and digital media marketers grapple with constantly, whether they realize it or not: Is "awareness" a valid objective for marketing activity?

I’ve gotten into more than a few heated debates that, at their core, center around this question. Some of those debates have been with myself (those are the ones where I most need a skilled moderator!).

Continue reading this article ... ... more from Tim Wilson

Advanced Conversion Syntax Merchandising
Adam Greco, Senior Partner

As I have mentioned in the past, one of the Adobe SiteCatalyst (Analytics) topics I loathe talking about is Product Merchandising. Product Merchandising is complicated and often leaves people scratching their heads in my "Top Gun" training classes. However, many people have mentioned to me that my previous post on Product Merchandising eVars helped them a lot so I am going to continue sharing information on this topic.

Continue reading this article ... ... more from Adam Greco

Team Demystified Update from Wendy Greco
Eric T. Peterson, Senior Partner

When Eric Peterson asked me to lead Team Demystified a year ago, I couldn’t say no! Having seen how hard all of the Web Analytics Demystified partners work and that they are still not able to keep up with the demand of clients for their services, it made sense for Web Analytics Demystified to find another way to scale their services. Since the Demystified team knows all of the best people in our industry and has tons of great clients, it is not surprising that our new Team Demystified venture has taken off as quickly as it has.

Continue reading this article ... ... more from Eric T. Peterson

SiteCatalyst Unannounced Features
Adam Greco, Senior Partner

Lately, Adobe has been sneaking in some cool new features into the SiteCatalyst product and doing it without much fanfare. While I am sure these are buried somewhere in release notes, I thought I’d call out two of them that I really like, so you know that they are there.

Continue reading this article ... ... more from Adam Greco

Hello. I’m a Radical Analytics Pragmatist
Tim Wilson, Partner

I was reading a post last week by one of the Big Names in web analytics…and it royally pissed me off. I started to comment and then thought, “Why pick a fight?” We’ve had more than enough of those for our little industry over the past few years. So I let it go.

Except I didn’t let it go.

Continue reading this article ... ... more from Tim Wilson

Competitor Pricing Analysis
Adam Greco, Senior Partner

One of my newest clients is in a highly competitive business in which they sell similar products as other retailers. These days, many online retailers have a hunch that they are being “Amazon-ed,” which they define as visitors finding products on their website and then going to see if they can get it cheaper/faster on Amazon.com. This client was attempting to use time spent on page as a way to tell if/when visitors were leaving their site to go price shopping.

Continue reading this article ... ... more from Adam Greco

How to Deliver Better Recommendations: Forecast the Impact!
Michele Kiss, Partner

One of the most valuable ways to be sure your recommendations are heard is to forecast the impact of your proposal. Consider what is more likely to be heard: "I think we should do X ..." vs "I think we should do X, and with a 2% increase in conversion, that would drive a $1MM increase in revenue ..."

Continue reading this article ... ... more from Michele Kiss

ACCELERATE 2014 “Advanced Analytics Education” Classes Posted
Eric T. Peterson, Senior Partner

I am delighted to share the news that our 2014 “Advanced Analytics Education” classes have been posted and are available for registration. We expanded our offering this year and will be offering four concurrent analytics and optimization training sessions from all of the Web Analytics Demystified Partners and Senior Partners on September 16th and 17th at the Cobb Galaria in Atlanta, Georgia.

Continue reading this article ... ... more from Eric T. Peterson

Product Cart Addition Sequence
Adam Greco, Senior Partner

In working with a client recently, an interesting question arose around cart additions. This client wanted to know the order in which visitors were adding products to the shopping cart. Which products tended to be added first, second third, etc.? They also wanted to know which products were added after a specific product was added to the cart (i.e. if a visitor adds product A, what is the next product they tend to add?). Finally, they wondered which cart add product combinations most often lead to orders.

Continue reading this article ... ... more from Adam Greco

7 Tips For Delivering Better Analytics Recommendations
Michele Kiss, Partner

As an analyst, your value is not just in the data you deliver, but in the insight and recommendations you can provide. But what is an analyst to do when those recommendations seem to fall on deaf ears?

Continue reading this article ... ... more from Michele Kiss

Overcoming The Analyst Curse: DON’T Show Your Math!
Michele Kiss, Partner

If I could give one piece of advice to an aspiring analyst, it would be this: Stop showing your "math". A tendency towards "TMI deliverables" is common, especially in newer analysts. However, while analysts typically do this in an attempt to demonstrate credibility ("See? I used all the right data and methods!") they do so at the expense of actually being heard.

Continue reading this article ... ... more from Michele Kiss

Making Tables of Numbers Comprehensible
Tim Wilson, Partner

I'm always amazed (read: dismayed) when I see the results of an analysis presented with a key set of the results delivered as a raw table of numbers. It is impossible to instantly comprehend a data table that has more than 3 or 4 rows and 3 or 4 columns. And, "instant comprehension" should be the goal of any presentation of information — it's the hook that gets your audience's brain wrapped around the material and ready to ponder it more deeply.

Continue reading this article ... ... more from Tim Wilson

Automating the Cleanup of Facebook Insights Exports
Tim Wilson, Partner

This post (the download, really — it’s not much of a post) is about dealing with exports from Facebook Insights. If that's not something you do, skip it. Go back to Facebook and watch some cat videos. If you are in a situation where you get data about your Facebook page by exporting .csv or .xls files from the Facebook Insights web interface, then you probably sometimes think you need a 52" monitor to manage the horizontal scrolling.

Continue reading this article ... ... more from Tim Wilson

The Recent Forrester Wave on Web Analytics ... is Wrong
Eric T. Peterson, Senior Partner

Having worked as an industry analyst back in the day I still find myself interested in what the analyst community has to say about web analytics, especially when it comes to vendor evaluation. The evaluations are interesting because of the sheer amount of work that goes into them in an attempt to distill entire companies down into simple infographics, tables, and single paragraph summaries.

Continue reading this article ... ... more from Eric T. Peterson

Funnel Visualizations That Make Sense
Tim Wilson, Partner

Funnels, as a concept, make some sense (although someone once made a good argument that they make no sense, since, when the concept is applied by marketers, the funnel is really more a "very, very leaky funnel," which would be a worthless funnel — real-world funnels get all of a liquid from a wide opening through a smaller spout; but, let’s not quibble).

Continue reading this article ... ... more from Tim Wilson

Reenergizing Your Web Analytics Program & Implementation
Adam Greco, Senior Partner

Those of you who have read my blog posts (and book) over the years, know that I have lots of opinions when it comes to web analytics, web analytics implementations and especially those using Adobe Analytics. Whenever possible, I try to impart lessons I have learned during my web analytics career so you can improve things at your organization.

Continue reading this article ... ... more from Adam Greco

Registration for ACCELERATE 2014 is now open
Eric T. Peterson, Senior Partner

I am excited to announce that registration for ACCELERATE 2014 on September 18th in Atlanta, Georgia is now open. You can learn more about the event and our unique "Ten Tips in Twenty Minutes" format on our ACCELERATE mini-site, and we plan to have registration open for our Advanced Analytics Education pre-ACCELERATE training sessions in the coming weeks.

Continue reading this article ... ... more from Eric T. Peterson

Current Order Value
Adam Greco, Senior Partner

I recently had a client pose an interesting question related to their shopping cart. They wanted to know the distribution of money its visitors were bringing with them to each step of the shopping cart funnel.

Continue reading this article ... ... more from Adam Greco

A Guide to Segment Sharing in Adobe Analytics
Tim Wilson, Partner

Over the past year, I've run into situations multiple times where I wanted an Adobe Analytics segment to be available in multiple Adobe Analytics platforms. It turns out…that's not as easy as it sounds. I actually went multiple rounds with Client Care once trying to get it figured out. And, I’ve found "the answer" on more than one occasion, only to later realize that that answer was a bit misguided.

Continue reading this article ... ... more from Tim Wilson

Currencies & Exchange Rates
Adam Greco, Senior Partner

If your web analytics work covers websites or apps that span different countries, there are some important aspects of Adobe SiteCatalyst (Analytics) that you must know. In this post, I will share some of the things I have learned over the years related to currencies and exchange rates in SiteCatalyst.

Continue reading this article ... ... more from Adam Greco

Linking Authenticated Visitors Across Devices
Adam Greco, Senior Partner

In the last few years, people have become accustomed to using multiple digital devices simultaneously. While watching the recent winter Olympics, consumers might be on the Olympics website, while also using native mobile or tablet apps. As a result, some of my clients have asked me whether it is possible to link visits and paths across these devices so they can see cross-device paths and other behaviors.

Continue reading this article ... ... more from Adam Greco

The 80/20 Rule for Analytics Teams
Eric T. Peterson, Senior Partner

I had the pleasure last week of visiting with one of Web Analytics Demystified’s longest-standing and, at least from a digital analytical perspective, most successful clients. The team has grown tremendously over the years in terms of size and, more importantly, stature within the broader multi-channel business and has become one of the most productive and mature digital analytics groups that I personally am aware of across the industry.

Continue reading this article ... ... more from Eric T. Peterson

Ten Things You Should ALWAYS Do (or Not Do) in Excel
Tim Wilson, Partner

Last week I was surprised by the Twitter conversation a fairly innocuous vent-via-Twitter tweet started, with several people noting that they had no idea you could simple turn off the gridlines.

Continue reading this article ... ... more from Tim Wilson

Omni Man (and Team Demystified) Needs You!
Adam Greco, Senior Partner

As someone in the web analytics field, you probably hear how lucky you are due to the fact that there are always web analytics jobs available. When the rest of the country is looking for work and you get daily calls from recruiters, it isn’t a bad position to be in! At Web Analytics Demystified, we have more than doubled in the past year and still cannot keep up with the demand, so I am reaching out to you ...

Continue reading this article ... ... more from Adam Greco

A Useful Framework for Social Media "Engagements"
Tim Wilson, Partner

Whether you have a single toe dipped in the waters of social media analytics or are fully submerged and drowning, you’ve almost certainly grappled with "engagement." This post isn’t going to answer the question "Is engagement ROI?" ...

Continue reading this article ... ... more from Tim Wilson

It’s not about "Big Data", it’s about the "RIGHT data"
Michele Kiss, Partner

Unless you’ve been living under a rock, you have heard (and perhaps grown tired) of the buzzword "big data." But in attempts to chase the "next shiny thing", companies may focus too much on "big data" rather than the "right data."

Continue reading this article ... ... more from Michele Kiss

Eric T.
Peterson

John
Lovett

Adam
Greco

Brian
Hawkins

Kevin
Willeitner

Michele
Kiss

Josh
West

Tim
Wilson

Contact Us

You can contact Web Analytics Demystified day or night via email or by reaching out to one of our Partners directly.

» Contact Information

Web Analytics Demystified, Inc.
P.O. Box 13303
Portland, OR 97213
(503) 282-2601


Useful Links