Via Juice Analytics, I was up early this morning to catch up on my reading and I found myself flipping through Matthew May’s Change This presentation on elegant solutions. The slide deck is a great read and I’m definitely going to check out The Elegant Solution: Toyota’s Formula for Mastering Innovation. One thing really stood out for me as particularly relevant to the plight of companies working to be successful with web analytics.

“Make Kaizen Mandatory” is his key practice driving innovation #9. May says:

Kaizen has three steps: First, create a standard. Second, follow it. Third, find a better way. Repeat endlessly. Trying to improve and innovate without a standard as reference is like a journey with no starting point. It’s like hitting golf balls in the fog.

Over the years many companies have asked me “How can we be more successful with web analytics?” to which I invariably respond “Determine where you are today, establish goals for improvement, and iterate until you achieve your goals.” Generating and distributing the “right” reports is the key–creating a common understanding of where the business is today, and where the business needs to go tomorrow.

Sounds reasonable, doesn’t it? Innovation through iteration. Don’t “swing for the fences” or get too clever if you can avoid it; rather, implement a functional process that allows you to move from point “A” to point “B” systematically. Often times folks are surprised that the simple solutions are also the right solutions.

I don’t know golf, but hitting golf balls in the fog sounds like a waste of money. Check out Mr. May’s ChangeThis presentation.

I recently had the pleasure of chatting with Megan Burns at Forrester Research. Megan covers, among other things, web analytics. Despite her being with Forrester for just about a year now, she is one of the leading analysts thinking about how companies actually deploy and use web analytics technology.

Megan hit the ground running, filling some pretty big shoes, and has published nearly a half-dozen reports directly relevant to the web analytics market. I don’t normally interview people in my weblog but when Forrester analysts talk, people listen. The transcript of our conversation follows:

Eric T. Peterson: First question: You’ve been in the job at Forrester Research for just over a year now. What would you say the biggest thing you’ve learned about the web analytics industry is so far?

Megan Burns: There are probably two things that stand out. First, the fact that measuring a Web sites is tougher than many people think.

There’s so much data to chose from, and all of it’s imperfect. Plus there are multiple ways to solve most measurement problems. Deciding which approach to take isn’t always straightforward and for many people measurement is just one part of their job, so even though they’d like to dedicate time to thinking about the best way to leverage all the data they could be collecting, they have to make some tough priority calls.

But the second thing I’ve learned is that many people believe very passionately in the power of data, and they’re committed to figuring out the tough problems.

Eric T. Peterson: So on one side of the coin you have the complexity of measurement as a function of expertise, approach, and time, and on the other side of the coin a strong desire to make it work.

Megan Burns: Absolutely.

Eric T. Peterson: So the folks you talk to who are being successful with web analytics, is there something that sets them apart? Something quantifiable?

Megan Burns: I’m not sure if it’s quantifiable, but there’s an understanding that metrics are a means to an end not an end unto themselves. They constantly think about what they’re trying to do, and how data can act as a tool to help them do it.

Eric T. Peterson: In your experience, does success with analytics improve with company size or does it appear to be tied to motivation?

Megan Burns: I haven’t seen a correlation to company size, but I haven’t looked at that relationship specifically.

I don’t think company size is a factor, though. I think it has more to do with attitude and approach to the problem. Often it’s about teaching people in the company about data and what it can do for them so that they change the way they make decisions. Any time you’re dealing with people and change it takes a good dose of both patience and time.

Eric T. Peterson: You and I have talked in the past about the importance of “process” to web analytics. You commented once that “people think process is a four letter word” which made me laugh and wince at the same time.

First, can you describe your position on the need for process in web analytics? And second, do you have any advice to help companies get past their fear of the “P” word?

Megan Burns: Sure. I think process is important in any discipline as a way to help people make sure the right things get done by the right people at the right time. We’re all trying to do so much these days, it helps to have a process that reminds us what needs to get done. It also sets clear lines within the organization as to what each person or group is responsible for and who they are dependent on. Web analytics is no different.

People who design sites need to understand that others in the business have to be able to measure the impact and success of those sites. They need to factor measurement requirements in to the process. But they have many other people asking them to build in other requirements, so it helps to have a checklist to make sure you’ve thought about all the different types of requirements you need to capture before you build something. That checklist is part of the process.

But it’s important to remember that the “Web analytics process” is really a sub-process of the larger eBusiness process. That data is needed by certain people in the firm at certain points in their decision making cycle. If the two aren’t integrated properly, things break down. People don’t get the data they need when they need it.

Changing people’s perceptions of process can be tough, depending on their experience with it. But I think the most important thing to remember is that process != bureaucracy.

When the only thing people are trying to do is check off boxes on the process so they’re “in compliance”, you’ve totally missed the point. It needs to be detailed enough that it’s useful and insures key steps don’t get missed, but it shouldn’t impose unnecessary restrictions or red tape. That’s a very fine line — one that’s not easy to get right.

Eric T. Peterson: So tell me the truth and don’t hold back … in my presentation at Emetrics where I advised our community to go so far as to draw business process diagrams for how web analytics integrates into the bigger picture … good idea or a superfulous waste of time?

Megan Burns: Somewhere in between. I think people responsible for analytics should start by looking at the larger site design/interactive marketing process (which probably isn’t written down anywhere, by the way ) and see where and when the data needs arise. Then look at where they need to be involved (i.e. requirments, development) in order to get what they need to meet those requirements. To me, measurement is a section in the business and technical requirements documents that must always be filled out. Even if it just says “No Impact” or “No new requirements”. But at least that way you know someone thought about what new data might be needed, or what code might have to change to maintain existing measurement.

Metrics need to be considered in the project planning process, like any other feature of the site. How long will it take to define requirements for this? To implement and test them? The process is no different, but it’s not something customers use so often it gets missed.

Or skipped intentionally to save time. But then it takes twice as long to add in after the fact, so you didn’t really save any time.

Eric T. Peterson: Excellent points, all.

You have a background in software development process, don’t you?

Megan Burns: Yes, I do.

Eric T. Peterson: Okay, new direction here: You’ve written a ton on web analytics since joining Forrester Research. I especially enjoyed your work on the ROI of dedicated headcount for analysis. What was the overall response to that report?

Megan Burns: The response was extremely positive. So many people I talk to tell me that resources are their biggest obstacle to using and interpreting the Web analytics data they’re collecting. The report helped them explain to senior management what analysts do and how they add value to the organization.

In a quantitative way, that is.

Eric T. Peterson: When companies ask you where to find experienced web analytics talent, what kind of advice do you give?

Megan Burns: That’s a tough question, because experienced Web analytics talent is so hard to find these days. My advice is usually to engage professional services consultants from either their vendor or an independent consulting firm to act as mentors for existing staff.

If they really want to hire, I suggest networking, networking, networking.

This is such an active community with the Yahoo group, blogs, Web Analytics Wednesdays … there are plenty of ways to meet others who might be able to lead them to a qualified candidate.

Eric T. Peterson: We’re just about out of time and I want to thank you for being so generous in allowing me to interview you.

Megan Burns: Quite welcome. Glad we could finally arrange it.

Eric T. Peterson: Last few questions … what book or books are currently on your nightstand?

Megan Burns: There are so many books … Information Dashboard Design, by Stephen Few is one.

Eric T. Peterson: What music would we find on your iPod?

Megan Burns: My iPod’s full of all sorts of music. Everything from rock to oldies to show tunes.

Eric T. Peterson: Who are some of your favorite bloggers?

Megan Burns: I wish I had time to read as many blogs as I’d like to.

I try to keep up regularly with Charlene Li and the other Forrester blogs, you, Avinash, and Om Malik. A few others, too. It also depends on what I’m working on.

Eric T. Peterson: Megan thanks very much for taking the time to chat with me today. I hope to see you in San Francisco in May and look forward to your upcoming research on Rich Internet Applications.

Megan Burns: You’re quite welcome. Take care, and I’ll see you in CA (if not sooner).

The folks at Register.com are looking for someone pretty senior to help them get oriented around web analytics. I talked to Steven Riccobono in Human Resources at Register.com and he told me that the company was looking for a strong presenter familiar with both online marketing, web analytics, and statistical analysis of data to help guide the next phase of Register.com’s growth. The company is under new leadership (their new CEO recently transformed Hoovers.com), is profitable, and is aggressively working to improve the customer experience on the site.

I asked about salary and Steven said that the right person will probably make between $90K and $100K and that the position is bonus eligible. He described their employee benefits package and it sounded pretty good. Still, most people I know take jobs like this because they can be part of a team that will make a difference in how the business is run, not for 401k matching.

Register.com has never had a dedicated web analytics resource before so this sounds like a pretty good opportunity to bring experience and skills to a company that says they’re looking for a “bottom-line difference maker, someone who can really drive success.” If you’re out there and looking for a new opportunity in Manhattan, have solid experience working in the most commonly used web analytics packages, understand the necessary statistics to differentiate signal from noise, and can present with the best of them you should give Steven a call. Tell him “Eric sent you.”

Check out the position posting here.

While I have not had time to write Part V of my series on measuring visitor engagement, I wanted to take a few minutes to address some comments folks have made about the metric recently. It’s very encouraging to see folks like Gary Angel and Daniel Markus pushing the conversation about measuring engagement along as I can think of few more qualified to critique this work.

Gary Angel, who had very nice things to say about the metric, commented on how in some areas the metric is biased, specifically towards search engines and specific types of content. Gary is concerned that the Brand Index will unfairly bias towards search engines (given that one component is searches for brand-specific terms like “eric t. peterson” and “web analytics demystified”.) I examined this effect and it turns out that “branded searches” make up only a small part of the index for my site but Gary makes an excellent point, unnecessary bias should be removed from the index whenever possible. As such, in my current calculation I have removed this weighting from the Brand Index, redefining said index to only be direct sessions (non-search, non-referred.)

Score one for Gary.

Gary also commented that:

“… if I’m using my metric to measure the “engagement” produced by visitors who used a specific part of a site (like the blog or the press releases), it’s vitally important that my metric not include a strong built in bias toward one of the areas (like blogging). Some analysts might argue that this represents a flaw in the metric Eric proposes. I don’t think so. Every metric carries with it some biases – and no metric is appropriate to every situation.”

This is a good point, one that had been made by a handful of other folks who critiqued the metric early on. The problem I have with removing the Blog Index (ratio of blog reading sessions to all sessions) is the evidence that my weblog is a prime driver of engagement with my site and overall web analytics brand: Over the last 12 months, weblog subscribers are nearly 400 percent more likely to have returned to the site recently than non-readers; those visitors not subscribed to my blog (e.g., in Bloglines or Google Reader) but who are still reading blog content are 300 percent more likely to have returned recently.

Score one for Eric.

One thing worth noting, the way I am using Visual Site to measure weblog readership and subscription, this activity does not show up as traditional “page views” unless the reader A) reads the post on my web site or B) clicks through to the web site (at which time the post appears as a session “referrer”) — Visual Site is able to track external RSS and XML-based content using a non-page view event (something I call “reads”.) Not all web analytics systems afford their operators this flexibility so I thought it would be worth bringing up. This is part of the reason that the Blog Index needs to be a separate index, not part of the Click Depth Index as some have questioned.

But enough about Gary … Daniel Markus posted what I surmise to be a nice post about my visitor engagement metric at Marketing Facts late last week in which he called my calculation “the mother of all Web Analytics KPIs.” The post is entirely in Dutch and my Dutch is horrible so I wrote to Daniel and asked for a rough translation . While there were many good comments about the metric, they raised two concerns:

1. The calculation is complicated and difficult to understand.
2. There was some question of the utility of this metric, essentially calling into question the overall “actionability” (not a word) of visitor engagement.

Regarding the complexity of the calculation, as Gary has so eloquently stated any number of times, no indicator or metric is any use without understanding its components, its definition, and its inherent biases. Clearly the onus is on the web analyst to explain the metric and it’s definition to any audience they present engagement data, especially given the complete lack of formality around measuring “engagement” (at least until you started reading my posts on the subject.)

Given the complexity of the calculation, the latter concern is valid but one that misses the point of the metric. There are any number of loose definitions of “engagement” floating around in our community — duration, page views, average page views per session, sessions per visitor, etc. But none of these more easily understood (note: not easily interpreted) metrics, in my mind, captures the essence of an engaged visitor.

Visitor engagement has to be examined over diverse criteria, simple assessments simply do not work. To wit:

• To say that session duration is a good measure of engagement is fine, unless the visitor never returns to the site.
• To say that a high number of page views is a good measure of engagement is fine, unless the visitor runs up those page views in a very short period of time and was unlikely able to actually read content.
• To say that recency of visit is a good measure of engagement is fine, unless the visitor has only looked at your home page and left.
• To say that direct visits are a good measure of engagement is fine, unless those direct visits lead to short sessions of few pages viewed and the visitors never return.

I believe that the complexity of the calculation is where visitor engagement derives its value. For practitioners who are lucky enough to have access to a platform that can actually make this calculation and who are willing to take the time to explain to their audience what the metric measures and what its limitations and biases are, the metric can yield insights that would be unlikely to fall out of “traditional” web analytics.

I will leave you with an example of how I am deriving small insights from my measurement of visitor engagement.

Marshall Sponder is the WebMetricsGuru blogger and all-in-all a pretty nice guy. He and I had a little tiff awhile back over Avinash’s web analytics blogger index (something Avinash has stopped doing for some reason …) when I was less than complimentary about the volume of web analytics posts that he produced relative to his blogging in general. Examining traffic metrics from Marshall’s blog I would interpret the value of having a good relationship with him based on a set of commonly understood data:

Almost no volume and no books sold. Come on Marshall, let’s see a nice recommendation for Web Analytics Demystified already! ;-)

But wait, what if I have a closer look at the measured engagement of the visitors he’s been sending to my site:

While my “average” visitor to the site is only 24.2 percent engaged, visitors from Marshall’s posts are nearly 40 percent engaged with my site and, more importantly, of these visitors almost 10 percent are “highly engaged” (50 percent engagement or better.)

Marshall may not be selling books yet, but I have the nagging feeling if he tried even just a little, he could probably drive pretty good numbers given the engagement of the audience he referrers.

Now just imagine that you were running a million or billion dollar business, looking for new opportunities on the Internet. You have hundreds-if-not-thousands of sites sending you visitor traffic all day, every day. Maybe some of these people make purchases, but maybe you have nothing for them to purchase … how do you decide who to spend more time with and who to ignore?

Me, I’m going to write nice things about Marshall Sponder and if the folks from e-consultancy call me and want to do another interview, I’m taking that call right away! How’s that for a KPI defining an action?

I wish I had more time right now to address all the valuable points that Gary Angel of SEMphonic makes in his critique of my visitor engagement metric but I’m heading out early in the AM tomorrow. Suffice to say, Gary took the time to really drill-down into my work on measuring engagement and his analysis is great. Personally, when someone of Gary’s caliber says, “… while I’d quibble with one or two of his choices, I think it’s one of the best metrics for this that I’ve ever seen in web analytics” I get a big smile on my face.

Check out Gary’s great post on “Eric Peterson’s Engagement Metric“, and be sure to read it in the context of his last post on “the myth of actionability” … something sure to raise some hackles (mine are up, but I have not had a chance to respond!)