Archive for July, 2010

I am very sorry to say that our European partner Aurelie Pols has decided to leave Web Analytics Demystified and pursue other goals in her life. While I am very sad to announce this, I have certainly enjoyed working with Aurelie over the past year and on behalf of myself, my family, and our partner John Lovett we wish Aurelie, Rene, and little Luca all the best.

Kevin Hillstrom is one smart dude. President of MineThatData, author of Online Marketing Simulations, and prolific contributor to the Twitter #measure channel. Kevin spends a huge amount of time in Twitter challenging web analysts to think and work harder on behalf of their “clients,” 140 characters at a time.

A few weeks ago I asked Kevin “what five practices learned in the offline data analytics world would you like to see web analytics professionals adopt?” The following contributed blog post has Kevin’s answers which are, unsurprisingly, awesome. Near the end Kevin says “The Web Analyst has the keys to the future of the business, so it is a manner of getting the Web Analyst to figure out how to use keys to unlock the future potential of a business.”

Brilliant. We are the future of business … so what future will we be helping to create?

Kevin Hillstrom, President, MineThatData

In 1998, I became the Circulation Director at Eddie Bauer. Back in those days, Eddie Bauer printed money, generating more than a hundred million dollars of pre-tax profit on an annual basis.

One of the ways that Eddie Bauer generated profit was through the use of discounts and promotions. If a customer failed to purchase over a six month period of time, Eddie Bauer applied a “20% off your order” offer. The customer had to use a special promotion code, in order to receive discounted merchandise.

We analyzed each promotion code, using “A/B” test panels. Customers were randomly selected from the population, and then assigned to one of two test panels. The first test panel received the promotion, the second test panel did not receive the promotion. We subtracted the difference between the promotion segment and the control segment, and ran a profit and loss statement against the difference.

In almost all cases, the segment receiving the promotion generated more profit than the control segment. In other words, it became a “best practice” to offer customers promotions and incentives at Eddie Bauer. Over the course of a five year period of time, the marketing calendar became saturated with promotions. In fact, it became hard to find an open window where we could add promotions!

Being a huge fan of “A/B” testing, I decided to try something different. I asked my circulation team to choose two customer groups at random from our housefile. One group would receive promotions for the next six months, if the customer was eligible to receive the promotion. The other group would not receive a single promotion for the next six months. At the end of the six month test period, we would determine which strategy yielded the most profit.

At the end of six months, we observed a surprising outcome. The test group that received no promotions spent the exact same amount of money that the group receiving all promotions spent. After calculating the profitability of each test group, it was obvious that Eddie Bauer was making a significant mistake. It appeared that we would lose, at most, five percent of total annual sales, if we backed off of our promotional strategy. Eddie Bauer would be significantly more profitable by minimizing the existing promotional strategy.

In 1999, we backed off of almost all of our housefile promotions. At the end of 1999, the website/catalog division enjoyed the most profitable year in the history of the business.

This experience shaped all of my subsequent analytical work.

Just because we have the tools to measure our activities in real-time doesn’t mean we are truly optimizing business results. In the Eddie Bauer example, we had the analytical tools to measure every single promotion we offered the customer, and we used existing best practices and “A/B” testing strategies. All of it, however, was wrong, costing us $26,000,000 of profit on an annual basis. Simply put, we were measuring “conversion rate”. What actually happened was that we “shifted conversions” out of non-promotional windows, into promotional windows! Had we measured non-promotional windows, we would have noticed that demand decreased.

So, by measuring customer behavior across a six month period of time, we made a significant change to business strategy, one that dramatically increased annual profit.

What does this have to do with Web Analytics?

The overwhelming majority of Web Analytics activity is focused on improving “conversion rate”. Our software tools are calibrated for easy analysis of events. Did a visitor do what we wanted the visitor to do? Did a promotion work? Did a search visitor from a long-tail keyword buy merchandise when they visited the website? All of these questions are easily answered by the Web Analytics expert, the expert simply analyzes an event to determine if the event yielded a favorable outcome.

Offline analytics experts (often called “Business Intelligence” professionals or “SAS Programmers” if they use SAS software to analyze data) frequently analyze business problems from a different perspective. They use whatever data is available, incomplete or comprehensive, to determine if the individual actions taken by a business over time cause a customer to become more loyal.

With that in mind, here are five offline practices I wish online analytics experts would adopt.

Practice #1 = Extend the Conversion Window: Instead of analyzing whether a customer converted within a single visit or session, it makes sense to extend the conversion window and learn whether the customer converted across a period of time. For instance, when I ran Database Marketing at Nordstrom, we learned that our best customers had a 5% conversion rate, when measured on the basis of individual visits, but our best customers nearly achieved a 100% conversion rate when combining website visits and store visits during a month. By extending the conversion window, we realized that we didn’t have website problems, instead, we had loyal customers who used our website as a tool in a multi-channel process.

Practice #2 = Measure Long-Term Value: Offline analytics practitioners want to know if a series of actions results in long-term profit. In other words, individual conversions are relatively meaningless if, over the course of a year, individual conversions do not yield incremental profit. This is essentially the “Eddie Bauer” example I mentioned at the start of this paper, we learned that individual conversions (customers purchasing via a promo code) yielded increased profit during the promotional period, but generated a loss when measured across a six month timeframe. A generation of Web Analytics experts were trained, largely because of software limitations, to analyze short-term business results, and have not developed the discipline to do what is right for a business across a six month or one year timeframe. Fortunately, Web Analytics practitioners are exceptionally bright, and are easily able to adapt to longer conversion windows.

Practice #3 = Comfort with Incomplete Data: I recently analyzed data for a retailer that was able to tie 70% of store transactions to a name/address. During my presentation, an Executive mentioned that my results must be inaccurate, because I was leaving 30% of the transactions out of my analysis. When I asked the Executive if it would be better to make decisions on incomplete data, or to simply not make any decisions at all until all data is complete and accurate, the Executive acknowledged that inferences from incomplete data are better than inaction caused by data uncertainty. Offline analysts have been dealing with incomplete multi-channel data for decades, and have become good at communicating the benefits and limitations of incomplete data to business leaders. The same opportunity exists for Web Analytics practitioners. Don’t hide from incomplete data! Instead, make confident decisions based on the data that is available, simply communicating what one can and cannot infer from incomplete data.

Practice #4 = Demonstrate What Happens to a Business Five Years From Now Based on Today’s Actions: Believe it or not, this is how I make a living. I use conditional probabilities to show what happens if customers evolve a certain way. Pretend a business had 100 customers in 2009, and 44 of the 100 customers purchase again during 2010. This business must find 56 new customers in 2010 to replace the customers lost during 2010. I can demonstrate what the business will look like in 2015, based on how well the business can retain existing customers or acquire new customers. This type of analysis is the exact opposite of “conversion rate analysis”, because we are looking at the long-term retention/acquisition dynamics that impact every single business. I find that CEOs and CFOs love this type of analysis, because for the first time, they have a window into the future, they actually get to see where the business is heading if things remain as they are today. Better yet, the CEO/CFO can go through “scenario planning” to identify ways to mitigate problems or to capitalize on favorable business trends. The Web Analytics practitioner has the data to do this type of analysis, it is simply a matter of tagging customers or shaping queries in a way that allows the analyst to make inferences that impact long-term customer value.

Practice #5 = Communicate Better: This probably applies to all analysts, not just Web Analytics experts. Executives are frequently called “HiPPOs” by the Web Analytics community, a term that refers to “Highest Paid Person’s Opinion”. The term can be used in a negative manner, suggesting that the Executive is choosing to not make decisions based on data but rather on opinion or gut feel or instinct or internal politics. I was a member of the Executive team at Nordstrom for more than six years, and I can honestly say that I made far more decisions based on opinion than I made based on sound data and analytics … and I am an analyst by trade!! Too often, the analytics community tells an incomplete story. Once, I witnessed an analytically minded individual who made a compelling argument, demonstrating that e-mail marketing had a better return on investment than catalog marketing. This analyst used the argument to suggest that the company shut down the catalog marketing division. On the surface, the argument made sense. Upon digging into the data a bit more, we learned that 75% of all e-mail addresses were acquired when a catalog shopper was placing an online order, so if we discontinued catalog marketing, we would cut off the source of future e-mail addresses. This is a case where the analyst failed to communicate in an appropriate manner, causing the Executive to not heed the advice of the analyst. Too often, analysts fail to put data and customer findings into a larger context. Total company profit, long-term customer profitability, total company staffing strategies and politics, multi-channel customer dynamics, and Executive goals and objectives all need to be taken into account by the analyst when communicating a data-driven story. When this is done well, the analyst becomes a surrogate member of the Executive team. When this is not done well, the analyst sometimes perceives the Executive to be a “HiPPO”.

These are the five areas I’d like to see Web Analytics experts evolve into. The Web Analyst has the keys to the future of the business, so it is a manner of getting the Web Analyst to figure out how to use keys to unlock the future potential of a business. Based on what I have witnessed during the past forty months of multi-channel consulting, I am very confident that Web Analytics practitioners can combine offline techniques with online analytics. The combination of offline techniques and online analytics yields a highly-valued analyst that Executives depend upon to make good business decisions!

I’m excited to announce that most of the 2010 X Change huddle topics and leaders have now been announced on our web site. If you’ve heard about the X Change and have been wondering what we will be talking about, please go have a look at the 2010 topics! We are more or less talking about everything … mobile, social, tagging, analysis, big data, testing, … you name it and we will be talking about it in Monterey September 21st and 22nd.

Serious. We have Kim Weller from ESPN talking about Digital Convergence, Kelly Olin from Nike talking about Measuring Global Brands, Dylan Lewis from Intuit talking about Testing, Lynn Lanphier from Best Buy talking about Analytics for Retailers, and 16 more amazing minds talking about the pressing topics of our day.

Are you ready to join us yet?

Just in case you’re not, please take a look at some of the amazing practitioners we have leading this year’s conversations. I consider it an honor to be co-producing a conference with so many brilliant web analytics practice leaders coming to join us and make the event happen. Folks like Shari Cleary from MTV Networks, Blandon Casanave from NBC Universal, Bob Page from eBay, and Adam Greco (yes, THAT Adam Greco) from Salesforce.com!

I know, amazing!

We will be adding a few more topics and conversation leaders this week so bookmark those pages and keep checking in. And by all means, if you have any questions about whether X Change is right for you, what the event is like, and what you can expect to take back to your boss after the conference, please don’t hesitate to contact me or one of my partners.

Don’t forget to read about this year’s exciting keynote with our “three VPs” as well. Shari, Joe Megibow from Expedia.com, and Steve Bernstein from Paypal will be talking about the career path from analyst to Vice President and the types of challenges they face heading analytics organizations as part of their companies senior leadership teams.

We are waaaay ahead on registrations this year compared to previous years and so a sell-out is more or less assured at this point. Don’t get left out — register right away and ensure your seat at the table at X Change 2010!

(This is a guest post from Jason Thompson, one of the great Analysis Exchange mentors that have been working to help us create an entirely new way to train web analysts while also providing free analytics to nonprofit organizations around the world.  Jason blogs at http://emptymind.org and can be found banging around Twitter @usujason.

We are offering a complimentary pass to this year’s X Change conference in Monterey, California to one mentor and one student who distinguish themselves in the program.)

There is a concept in Zen Buddhism called Shoshin, meaning “beginner’s mind”. This concept refers to being open and eager or as Shunryu Suzuki puts it, “In the beginner’s mind there are many possibilities, in the expert’s mind there are few.”

When I first steeped foot inside The Analysis Exchange, I did so as a mentor or in my mind “the expert.” Sure, I had a warm, fuzzy feeling deep down inside about giving back to the community, sharing freely of my knowledge, and showing my altruistic side but really I was there to teach, after all, I was the expert.

For those of you who may not be familiar with The Analysis Exchange, let me take a step back. The Analysis Exchange is a unique community of non-profit organizations, web analytics beginners, and industry experts, each willing to give of their time in order to reap their own rewards. For organizations, they gain access to free resources that help analyze data, train future analysts, and establish measurement road maps. The students, well, they get to attend school for free and learn on the job while they are mentored by the industry expert — not to mention it’s a great bullet-point on their resume. The mentors have the opportunity to share their skills, help shape the future of the industry, and yes, get a nice stroke to their ego.

It was not long into my first project that I was reminded of why Shoshin is so important. I was greeted by a student and an organization who were open to any possibilities and best of all were eager and excited about what web analytics had in store for them. Their child-like exuberance rekindled a flame inside me that had slowly faded away as the years of segmenting data past by.

The team quickly bonded and in 3 weeks we delivered an executive presentation highlighting low hanging fruit that the organization could quickly change and realize huge results, needless to say, this made our project manager look like a rockstar. It didn’t take much, a barebones Google Analytics implementation and a student full of bright ideas.

As extra credit, we delivered an implementation guide that the organization could use to beef up their data collection and an analytics road map to help successfully guide them down a path of measurement maturity.

I came to The Analysis Exchange as the expert but by the end, I had become the student. Those with the beginner’s mind had much to teach and I am grateful for the gift of a rekindled passion that they gave me.

(I am humbled by Jason’s description of his experience. Will you join Jason and make a difference in the world by mentoring a web analytics student and helping a great organization?)

Anyone who has read my blog for long knows that I am passionate about two things in web analytics: process and people. Process is the glue that holds all the hard work we do as analysts together and allows our effort to translate into tangible business value. But without a doubt it is the people who are absolutely critical to any businesses ability to compete and succeed on web analytics.

Unfortunately people, especially really good ones, are incredibly hard to find. So much so that my partners and I have invested heavily in creating an entirely new way for novice and veteran analytics practitioners alike to gain valuable “hand’s on” experience using data to answer business questions, The Analysis Exchange.

While the Analysis Exchange has exceeded every single short-term milestone we have established for the effort, it has long been clear to my partners and I that training alone is not enough to satisfy the immediate needs of businesses working to take advantage of their existing investment in web Analytics. Companies need analytical talent now, not a year from now, not in six months, right now.

Why the urgency? Myriad reasons. The money has been spent on technology, the clock is ticking, the promises have been made, offline revenues are in decline and the company’s digital channels are the hope and future and difference between profitability and not.

The web analytics promise is real — companies that have become adept at generating analytically-driven insights and then translating those insights into sound business decisions have staked a clear competitive advantage. The giants of our industry — brilliant people like Joe Megibow, Dylan Lewis, Shari Cleary, and Lynn Lanphier <plug>all of whom are coming to the X Change conference in September, are you?</plug> — have not only determined the value of people but have also figured out how to convince management of that value.

Have you? Most companies have not.

Most companies persist in their belief that web and digital analytics is something that they can do “part time” and still have the successes that Intuit, Expedia, MTV, Best Buy, and others gain by hiring brilliant people, giving them clear direction, and recognizing the value of the analytical output they produce. Despite being well-intentioned, far too many managers still believe that software alone will provide insights and make recommendations.

But I digress.

Because we at Web Analytics Demystified believe in people and process so strongly, and because we are pretty confident in our consulting as it relates to process, we have decided to put our money where our mouths are and start helping companies fill their open positions for “web analyst, senior.” Today we are extremely proud to announce our first-of-it’s-kind partnership with the web analytics community’s leading recruiting firm, IQ Workforce.

Working directly with Workforce CEO Corry Prohens and his team, Web Analytics Demystified has crafted a “one-two” punch to help speed the process of finding, vetting, and hiring the kind of deep talent and teams required to take complete advantage of any investment in digital measurement technology. The Demystified partners and IQ Workforce will help you determine exactly which roles you need to fill, what strengths the ideal candidate will have, and how hired resources will fit into the organization that both creates business value and a satisfying experience for the analyst (which has a surprisingly positive impact on retention!)

In essence Web Analytics Demystified with our 30+ years of experience in web analytics will sit on your hiring panel and help you find and hire the critical difference between “web analytics as a cost center” and “web analytics as a profit center.”

Did we mention we will do it for a fixed price and in a way that allows most companies to circumvent HR’s aversion to “outside help?”

If you’re looking for an analytics guru for your organization, give us a call. We are more than happy to explain how this partnership creates a dramatic advantage for most companies, and would love to talk with you about our business and our partners at IQ Workforce. In the meantime please have a look at our press release on the announcement and more details about the offering:

• Read the press release about this partnership
• Read about our Staffing and Resource Allocation offering

Thanks to Corry and his team for making this idea a reality. On behalf of IQ Workforce and the Demystified Partners we look forward to helping you with your staffing needs.