Registration is now open for ACCELERATE 2014 in Atlanta, Georgia on September 18th. Reserve your spot today at Eventbrite — tickets are only $99 USD!

Guest Post: Kevin Hillstrom

Published by Eric T. Peterson on July 19, 2010 All posts from Eric T. Peterson

Kevin Hillstrom is one smart dude. President of MineThatData, author of Online Marketing Simulations, and prolific contributor to the Twitter #measure channel. Kevin spends a huge amount of time in Twitter challenging web analysts to think and work harder on behalf of their “clients,” 140 characters at a time.

A few weeks ago I asked Kevin “what five practices learned in the offline data analytics world would you like to see web analytics professionals adopt?” The following contributed blog post has Kevin’s answers which are, unsurprisingly, awesome. Near the end Kevin says “The Web Analyst has the keys to the future of the business, so it is a manner of getting the Web Analyst to figure out how to use keys to unlock the future potential of a business.”

Brilliant. We are the future of business … so what future will we be helping to create?

Kevin Hillstrom, President, MineThatData

In 1998, I became the Circulation Director at Eddie Bauer. Back in those days, Eddie Bauer printed money, generating more than a hundred million dollars of pre-tax profit on an annual basis.

One of the ways that Eddie Bauer generated profit was through the use of discounts and promotions. If a customer failed to purchase over a six month period of time, Eddie Bauer applied a “20% off your order” offer. The customer had to use a special promotion code, in order to receive discounted merchandise.

We analyzed each promotion code, using “A/B” test panels. Customers were randomly selected from the population, and then assigned to one of two test panels. The first test panel received the promotion, the second test panel did not receive the promotion. We subtracted the difference between the promotion segment and the control segment, and ran a profit and loss statement against the difference.

In almost all cases, the segment receiving the promotion generated more profit than the control segment. In other words, it became a “best practice” to offer customers promotions and incentives at Eddie Bauer. Over the course of a five year period of time, the marketing calendar became saturated with promotions. In fact, it became hard to find an open window where we could add promotions!

Being a huge fan of “A/B” testing, I decided to try something different. I asked my circulation team to choose two customer groups at random from our housefile. One group would receive promotions for the next six months, if the customer was eligible to receive the promotion. The other group would not receive a single promotion for the next six months. At the end of the six month test period, we would determine which strategy yielded the most profit.

At the end of six months, we observed a surprising outcome. The test group that received no promotions spent the exact same amount of money that the group receiving all promotions spent. After calculating the profitability of each test group, it was obvious that Eddie Bauer was making a significant mistake. It appeared that we would lose, at most, five percent of total annual sales, if we backed off of our promotional strategy. Eddie Bauer would be significantly more profitable by minimizing the existing promotional strategy.

In 1999, we backed off of almost all of our housefile promotions. At the end of 1999, the website/catalog division enjoyed the most profitable year in the history of the business.

This experience shaped all of my subsequent analytical work.

Just because we have the tools to measure our activities in real-time doesn’t mean we are truly optimizing business results. In the Eddie Bauer example, we had the analytical tools to measure every single promotion we offered the customer, and we used existing best practices and “A/B” testing strategies. All of it, however, was wrong, costing us $26,000,000 of profit on an annual basis. Simply put, we were measuring “conversion rate”. What actually happened was that we “shifted conversions” out of non-promotional windows, into promotional windows! Had we measured non-promotional windows, we would have noticed that demand decreased.

So, by measuring customer behavior across a six month period of time, we made a significant change to business strategy, one that dramatically increased annual profit.

What does this have to do with Web Analytics?

The overwhelming majority of Web Analytics activity is focused on improving “conversion rate”. Our software tools are calibrated for easy analysis of events. Did a visitor do what we wanted the visitor to do? Did a promotion work? Did a search visitor from a long-tail keyword buy merchandise when they visited the website? All of these questions are easily answered by the Web Analytics expert, the expert simply analyzes an event to determine if the event yielded a favorable outcome.

Offline analytics experts (often called “Business Intelligence” professionals or “SAS Programmers” if they use SAS software to analyze data) frequently analyze business problems from a different perspective. They use whatever data is available, incomplete or comprehensive, to determine if the individual actions taken by a business over time cause a customer to become more loyal.

With that in mind, here are five offline practices I wish online analytics experts would adopt.

Practice #1 = Extend the Conversion Window: Instead of analyzing whether a customer converted within a single visit or session, it makes sense to extend the conversion window and learn whether the customer converted across a period of time. For instance, when I ran Database Marketing at Nordstrom, we learned that our best customers had a 5% conversion rate, when measured on the basis of individual visits, but our best customers nearly achieved a 100% conversion rate when combining website visits and store visits during a month. By extending the conversion window, we realized that we didn’t have website problems, instead, we had loyal customers who used our website as a tool in a multi-channel process.

Practice #2 = Measure Long-Term Value: Offline analytics practitioners want to know if a series of actions results in long-term profit. In other words, individual conversions are relatively meaningless if, over the course of a year, individual conversions do not yield incremental profit. This is essentially the “Eddie Bauer” example I mentioned at the start of this paper, we learned that individual conversions (customers purchasing via a promo code) yielded increased profit during the promotional period, but generated a loss when measured across a six month timeframe. A generation of Web Analytics experts were trained, largely because of software limitations, to analyze short-term business results, and have not developed the discipline to do what is right for a business across a six month or one year timeframe. Fortunately, Web Analytics practitioners are exceptionally bright, and are easily able to adapt to longer conversion windows.

Practice #3 = Comfort with Incomplete Data: I recently analyzed data for a retailer that was able to tie 70% of store transactions to a name/address. During my presentation, an Executive mentioned that my results must be inaccurate, because I was leaving 30% of the transactions out of my analysis. When I asked the Executive if it would be better to make decisions on incomplete data, or to simply not make any decisions at all until all data is complete and accurate, the Executive acknowledged that inferences from incomplete data are better than inaction caused by data uncertainty. Offline analysts have been dealing with incomplete multi-channel data for decades, and have become good at communicating the benefits and limitations of incomplete data to business leaders. The same opportunity exists for Web Analytics practitioners. Don’t hide from incomplete data! Instead, make confident decisions based on the data that is available, simply communicating what one can and cannot infer from incomplete data.

Practice #4 = Demonstrate What Happens to a Business Five Years From Now Based on Today’s Actions: Believe it or not, this is how I make a living. I use conditional probabilities to show what happens if customers evolve a certain way. Pretend a business had 100 customers in 2009, and 44 of the 100 customers purchase again during 2010. This business must find 56 new customers in 2010 to replace the customers lost during 2010. I can demonstrate what the business will look like in 2015, based on how well the business can retain existing customers or acquire new customers. This type of analysis is the exact opposite of “conversion rate analysis”, because we are looking at the long-term retention/acquisition dynamics that impact every single business. I find that CEOs and CFOs love this type of analysis, because for the first time, they have a window into the future, they actually get to see where the business is heading if things remain as they are today. Better yet, the CEO/CFO can go through “scenario planning” to identify ways to mitigate problems or to capitalize on favorable business trends. The Web Analytics practitioner has the data to do this type of analysis, it is simply a matter of tagging customers or shaping queries in a way that allows the analyst to make inferences that impact long-term customer value.

Practice #5 = Communicate Better: This probably applies to all analysts, not just Web Analytics experts. Executives are frequently called “HiPPOs” by the Web Analytics community, a term that refers to “Highest Paid Person’s Opinion”. The term can be used in a negative manner, suggesting that the Executive is choosing to not make decisions based on data but rather on opinion or gut feel or instinct or internal politics. I was a member of the Executive team at Nordstrom for more than six years, and I can honestly say that I made far more decisions based on opinion than I made based on sound data and analytics … and I am an analyst by trade!! Too often, the analytics community tells an incomplete story. Once, I witnessed an analytically minded individual who made a compelling argument, demonstrating that e-mail marketing had a better return on investment than catalog marketing. This analyst used the argument to suggest that the company shut down the catalog marketing division. On the surface, the argument made sense. Upon digging into the data a bit more, we learned that 75% of all e-mail addresses were acquired when a catalog shopper was placing an online order, so if we discontinued catalog marketing, we would cut off the source of future e-mail addresses. This is a case where the analyst failed to communicate in an appropriate manner, causing the Executive to not heed the advice of the analyst. Too often, analysts fail to put data and customer findings into a larger context. Total company profit, long-term customer profitability, total company staffing strategies and politics, multi-channel customer dynamics, and Executive goals and objectives all need to be taken into account by the analyst when communicating a data-driven story. When this is done well, the analyst becomes a surrogate member of the Executive team. When this is not done well, the analyst sometimes perceives the Executive to be a “HiPPO”.

These are the five areas I’d like to see Web Analytics experts evolve into. The Web Analyst has the keys to the future of the business, so it is a manner of getting the Web Analyst to figure out how to use keys to unlock the future potential of a business. Based on what I have witnessed during the past forty months of multi-channel consulting, I am very confident that Web Analytics practitioners can combine offline techniques with online analytics. The combination of offline techniques and online analytics yields a highly-valued analyst that Executives depend upon to make good business decisions!

Share this blog post ...

Categorized under #measure channel, General Web Analytics, Interviews, Web Analytics People

  • Peter O’Neill

    Overall it is difficult to disagree with many of the comments Kevin makes or points he raises. Being comfortable with incomplete data and communicating effectively are both skills that I agree any analyst should have.

    But, while I agree in theory with the other points, I am not sure how possible it is with web analytics data, although it makes perfect sense with database or catalog marketing. The limitation in my mind is the ability to track individual people, linking marketing to them (easier with catalogs or emails), their visits to the site, actions on the site and any subsequent purchases.

    If a website cannot uniquely identify visitors, how can they understand if they convert over a period of time? While you can track customers who repurchase over time, how can you identify those visits without purchase (unless they have logged in)? I am ignoring here using cookies to identify individuals due to multiple device use and cookie deletion.

    So yes, I agree with the principle behind what Kevin has said but I do not currently believe a lot of it is possible with web analytics data. Instead we should be looking to develop alternative methods and modeling techniques that also focus on long term performance as per Practice #4 that take advantage of the data that is available within web analytics.


  • Jan A. Cornish

    The Web Analytics Book Club, chaired by David Rogers just completed grappling with Jim Novo’s book Drilling Down. This book delves into a lot of the issues that Kevin Hillstrom’s post identifies.

  • David Barber

    Kevin IS one smart dude. I’ve worked with him in his past lives. The practices he has outlined are both possible, and very effective.


Recent Blog Posts

Hello. I’m a Radical Analytics Pragmatist
Tim Wilson, Partner

I was reading a post last week by one of the Big Names in web analytics…and it royally pissed me off. I started to comment and then thought, “Why pick a fight?” We’ve had more than enough of those for our little industry over the past few years. So I let it go.

Except I didn’t let it go.

Continue reading this article ... ... more from Tim Wilson

Competitor Pricing Analysis
Adam Greco, Senior Partner

One of my newest clients is in a highly competitive business in which they sell similar products as other retailers. These days, many online retailers have a hunch that they are being “Amazon-ed,” which they define as visitors finding products on their website and then going to see if they can get it cheaper/faster on This client was attempting to use time spent on page as a way to tell if/when visitors were leaving their site to go price shopping.

Continue reading this article ... ... more from Adam Greco

How to Deliver Better Recommendations: Forecast the Impact!
Michele Kiss, Partner

One of the most valuable ways to be sure your recommendations are heard is to forecast the impact of your proposal. Consider what is more likely to be heard: "I think we should do X ..." vs "I think we should do X, and with a 2% increase in conversion, that would drive a $1MM increase in revenue ..."

Continue reading this article ... ... more from Michele Kiss

ACCELERATE 2014 “Advanced Analytics Education” Classes Posted
Eric T. Peterson, Senior Partner

I am delighted to share the news that our 2014 “Advanced Analytics Education” classes have been posted and are available for registration. We expanded our offering this year and will be offering four concurrent analytics and optimization training sessions from all of the Web Analytics Demystified Partners and Senior Partners on September 16th and 17th at the Cobb Gallaria in Atlanta, Georgia.

Continue reading this article ... ... more from Eric T. Peterson

Product Cart Addition Sequence
Adam Greco, Senior Partner

In working with a client recently, an interesting question arose around cart additions. This client wanted to know the order in which visitors were adding products to the shopping cart. Which products tended to be added first, second third, etc.? They also wanted to know which products were added after a specific product was added to the cart (i.e. if a visitor adds product A, what is the next product they tend to add?). Finally, they wondered which cart add product combinations most often lead to orders.

Continue reading this article ... ... more from Adam Greco

7 Tips For Delivering Better Analytics Recommendations
Michele Kiss, Partner

As an analyst, your value is not just in the data you deliver, but in the insight and recommendations you can provide. But what is an analyst to do when those recommendations seem to fall on deaf ears?

Continue reading this article ... ... more from Michele Kiss

Overcoming The Analyst Curse: DON’T Show Your Math!
Michele Kiss, Partner

If I could give one piece of advice to an aspiring analyst, it would be this: Stop showing your "math". A tendency towards "TMI deliverables" is common, especially in newer analysts. However, while analysts typically do this in an attempt to demonstrate credibility ("See? I used all the right data and methods!") they do so at the expense of actually being heard.

Continue reading this article ... ... more from Michele Kiss

Making Tables of Numbers Comprehensible
Tim Wilson, Partner

I'm always amazed (read: dismayed) when I see the results of an analysis presented with a key set of the results delivered as a raw table of numbers. It is impossible to instantly comprehend a data table that has more than 3 or 4 rows and 3 or 4 columns. And, "instant comprehension" should be the goal of any presentation of information — it's the hook that gets your audience's brain wrapped around the material and ready to ponder it more deeply.

Continue reading this article ... ... more from Tim Wilson

Automating the Cleanup of Facebook Insights Exports
Tim Wilson, Partner

This post (the download, really — it’s not much of a post) is about dealing with exports from Facebook Insights. If that's not something you do, skip it. Go back to Facebook and watch some cat videos. If you are in a situation where you get data about your Facebook page by exporting .csv or .xls files from the Facebook Insights web interface, then you probably sometimes think you need a 52" monitor to manage the horizontal scrolling.

Continue reading this article ... ... more from Tim Wilson

The Recent Forrester Wave on Web Analytics ... is Wrong
Eric T. Peterson, Senior Partner

Having worked as an industry analyst back in the day I still find myself interested in what the analyst community has to say about web analytics, especially when it comes to vendor evaluation. The evaluations are interesting because of the sheer amount of work that goes into them in an attempt to distill entire companies down into simple infographics, tables, and single paragraph summaries.

Continue reading this article ... ... more from Eric T. Peterson

Funnel Visualizations That Make Sense
Tim Wilson, Partner

Funnels, as a concept, make some sense (although someone once made a good argument that they make no sense, since, when the concept is applied by marketers, the funnel is really more a "very, very leaky funnel," which would be a worthless funnel — real-world funnels get all of a liquid from a wide opening through a smaller spout; but, let’s not quibble).

Continue reading this article ... ... more from Tim Wilson

Reenergizing Your Web Analytics Program & Implementation
Adam Greco, Senior Partner

Those of you who have read my blog posts (and book) over the years, know that I have lots of opinions when it comes to web analytics, web analytics implementations and especially those using Adobe Analytics. Whenever possible, I try to impart lessons I have learned during my web analytics career so you can improve things at your organization.

Continue reading this article ... ... more from Adam Greco

Registration for ACCELERATE 2014 is now open
Eric T. Peterson, Senior Partner

I am excited to announce that registration for ACCELERATE 2014 on September 18th in Atlanta, Georgia is now open. You can learn more about the event and our unique "Ten Tips in Twenty Minutes" format on our ACCELERATE mini-site, and we plan to have registration open for our Advanced Analytics Education pre-ACCELERATE training sessions in the coming weeks.

Continue reading this article ... ... more from Eric T. Peterson

Current Order Value
Adam Greco, Senior Partner

I recently had a client pose an interesting question related to their shopping cart. They wanted to know the distribution of money its visitors were bringing with them to each step of the shopping cart funnel.

Continue reading this article ... ... more from Adam Greco

A Guide to Segment Sharing in Adobe Analytics
Tim Wilson, Partner

Over the past year, I've run into situations multiple times where I wanted an Adobe Analytics segment to be available in multiple Adobe Analytics platforms. It turns out…that's not as easy as it sounds. I actually went multiple rounds with Client Care once trying to get it figured out. And, I’ve found "the answer" on more than one occasion, only to later realize that that answer was a bit misguided.

Continue reading this article ... ... more from Tim Wilson

Currencies & Exchange Rates
Adam Greco, Senior Partner

If your web analytics work covers websites or apps that span different countries, there are some important aspects of Adobe SiteCatalyst (Analytics) that you must know. In this post, I will share some of the things I have learned over the years related to currencies and exchange rates in SiteCatalyst.

Continue reading this article ... ... more from Adam Greco

Linking Authenticated Visitors Across Devices
Adam Greco, Senior Partner

In the last few years, people have become accustomed to using multiple digital devices simultaneously. While watching the recent winter Olympics, consumers might be on the Olympics website, while also using native mobile or tablet apps. As a result, some of my clients have asked me whether it is possible to link visits and paths across these devices so they can see cross-device paths and other behaviors.

Continue reading this article ... ... more from Adam Greco

The 80/20 Rule for Analytics Teams
Eric T. Peterson, Senior Partner

I had the pleasure last week of visiting with one of Web Analytics Demystified’s longest-standing and, at least from a digital analytical perspective, most successful clients. The team has grown tremendously over the years in terms of size and, more importantly, stature within the broader multi-channel business and has become one of the most productive and mature digital analytics groups that I personally am aware of across the industry.

Continue reading this article ... ... more from Eric T. Peterson

Ten Things You Should ALWAYS Do (or Not Do) in Excel
Tim Wilson, Partner

Last week I was surprised by the Twitter conversation a fairly innocuous vent-via-Twitter tweet started, with several people noting that they had no idea you could simple turn off the gridlines.

Continue reading this article ... ... more from Tim Wilson

Omni Man (and Team Demystified) Needs You!
Adam Greco, Senior Partner

As someone in the web analytics field, you probably hear how lucky you are due to the fact that there are always web analytics jobs available. When the rest of the country is looking for work and you get daily calls from recruiters, it isn’t a bad position to be in! At Web Analytics Demystified, we have more than doubled in the past year and still cannot keep up with the demand, so I am reaching out to you ...

Continue reading this article ... ... more from Adam Greco

A Useful Framework for Social Media "Engagements"
Tim Wilson, Partner

Whether you have a single toe dipped in the waters of social media analytics or are fully submerged and drowning, you’ve almost certainly grappled with "engagement." This post isn’t going to answer the question "Is engagement ROI?" ...

Continue reading this article ... ... more from Tim Wilson

It’s not about "Big Data", it’s about the "RIGHT data"
Michele Kiss, Partner

Unless you’ve been living under a rock, you have heard (and perhaps grown tired) of the buzzword "big data." But in attempts to chase the "next shiny thing", companies may focus too much on "big data" rather than the "right data."

Continue reading this article ... ... more from Michele Kiss

Eric T.








Contact Us

You can contact Web Analytics Demystified day or night via email or by reaching out to one of our Partners directly.

» Contact Information

Web Analytics Demystified, Inc.
P.O. Box 13303
Portland, OR 97213
(503) 282-2601

Useful Links