Decision Maker-derived vs. Sales-derived Win Loss Analysis

by Ken Allred, August 11, 2009

Which Methodology is Best for Product Managers?

Recently, respected product managers have been discussing the value of win loss reports as they relate to product strategy and technology roadmaps. In an interesting blog post on Product Management Insights, Michael Shrivathsan argues Why Win/Loss Reports Shouldnt Drive Your Product Roadmap. In his article, Shrivathsan outlines why doing win loss reports with your sales team might be unreliable, including the fact that there might be hidden bias in the sales teams perceptions. He cites a humorous tweet from Mike Boudreaux, a fellow Product Management twitterer, to illustrate his point: Typical win/loss analysis from sales force: majority of losses due to product and price. Majority of wins due to relationship.

As I read this article, I could understand Shrivathsans argument, but only if you define and develop your win loss program in a narrow and short-sighted way. By operationalizing your win loss in the way described, or by abandoning win loss analysis altogether, youll be missing out on perhaps the most important intelligence you need to validate your product strategies and roadmaps.

To begin, an important distinction needs to be made concerning the win loss analysis methodology being discussed in Shrivathsans post. From his comments, we can infer that the win loss analysis methodology being utilized is to have the responsible sales representatives identify the reasons they won or lost a deal and report this information back to product management and other interested parties. I define this as a sales-derived win loss analysis methodology.

This is a common conception of win loss analysis. Based on our experience at Primary Intelligence, we know that approximately 1/3 of companies indicate that they perform win loss analysis; however, we have found that only about 50% of those companies that say they have a win loss program implement what we define as a decision maker-derived win loss analysis methodology as opposed to a sales-derived one.

The distinction between these two very different types of win loss programs is very important. In fact, to fully illustrate this point I went to our own win loss program and grabbed a recent real-world example. This is an analysis of a recent competitive win, but the principal holds true for losses as well (if there is interest, I can show a real-world loss example at a later date).

Lets compare the results between a sales-derived win review and a decision maker-derived win review.

Selection Reasons

When our sales representative was asked why we won the opportunity, his response was:

The reason we won the deal is because I was tenacious and kept working on the deal and building relationships until I got in front of the right person. I then built my relationship with that person so that we had a great relationship. It was good that we had technology, but the primary reason they selected us was because of the relationship I was able to build with the decision maker.

Does this sound familiar? This response sounds similar to the kind that Shrivathsan laments in his blog post. However, lets compare our sales representatives response to the decision makers response on why they selected Primary Intelligence:

There were a number of reasons. One is their extensive experience in this business. [PIs sales rep] and [PIs account consultant]s personalities were a compelling factor, as were the quality of the work and the technology. We really liked the dashboard. Were just now starting to tap into the power of the technology. Primary Intelligence also gave us the ability of integrating the win loss data into Salesforce.com, our CRM system. The others could not do that. Those would be the primary selection reasons.

We can see from the decision makers comments that our sales reps relationship with the decision maker (along with our account consultant) absolutely had an impact on the decision. However, if thats all the information we had, we would be missing valuable intelligence on how this customer actually arrived at their purchase decision.

More importantly, our product management team could be led to believe that the integration with our customers SFA tools we have been investing in may not be a high priority for our customers and we could begin to question our product strategy.

Competitor Weaknesses

An important element of every win loss program is gaining a better understanding of your competitors products. Lets take a look at the competitive intelligence gaps we had in the different win loss analysis methodologies.

Our sales representative was asked why our primary competitor lost the deal. His response was:

They werent pleased with the depth of the information [our competitor] showed in their samples. The decision maker was really looking for a strategic partner and they didnt feel they could get that from [our competitor].

And the decision makers response to why they didnt select our competitor:

Primarily it was the quality of their work. I found the quality of the work was not very good and the questions were not followed up on. The deliverable was fairly weak and looked very unprofessional.

Our sales representative did a pretty good job describing the situation, but the decision makers response is a lot clearer when it comes to trying to understand our competitors weaknesses. The decision maker identifies three distinct weaknesses, while our sales rep identified only two weaknessesone that was confirmed by the decision maker and one that wasnt.

Valued Solution Features

Now lets dig a little deeper and focus on the solution features to see if we can identify additional gaps between sales-derived and decision maker-derived win loss programs. We asked our sales representative what features of the evaluated solutions were valued the most by the decision maker:

Quality of interviewing, because we do a much better job than [PIs competitor] and really uncover the things that affect the outcome of purchase decisions.

Quality of interviewing is definitely an important feature of our solution, but was it really the feature that the decision maker valued most? In our win loss interview, we asked the customer to rate the quality of several product criteria for both PIs solution and the solutions offered by the competition:

We can see here that while our sales rep was partially correct in that we did perform well in the quality of interviewing decision criterion, the decision maker indicated that the quality of deliverable and tools and dashboards were what they valued most. The customer commented:

Primary Intelligences deliverable is very balanced between quantitative and qualitative information. The executive summary thats produced saves me a lot of time and effort and I get a lot of value from the analytics we get like the competitive advantage scores and predictive analytics. And the value of delivering everything that is produced through the dashboard is really important.

If our current product strategy is to invest in specific technology features and we were to see several sales-derived win reviews like this one, we might begin to question our current plans. Seeing enough reviews like this might even cause us to halt development and redirect those resources to other projects. You can see how it would be easy to make some very costly mistakes if you are ONLY reviewing sales-derived win loss data.

Conclusion

The following table clearly illustrates the large differences between the two methodologies we have been discussing:

If you, or your organization, currently defines win loss analysis as a debrief from the responsible sales representative, I hope these examples can help shine some light on the big, risky gaps that are inherent in performing post-decision analysis via the sales representative.

To be perfectly fair, Shrivathsan does make a brief mention of including decision makers in win loss analysis, saying, theyre often much more open & honest to a product manager with whom they have no relationship, than a sales rep who worked with them for a period of time. As he alludes to, and as we have found in our own experience, these same principals weve been discussing will apply if you are having your sales representatives interview decision makers, so its important to have a third party of some sort perform the analysis (Ill be posting on this subject in more depth at a later point).

I have to agree with Shrivathsans assessment that you shouldnt allow sales rep-derived win loss reports to drive or affect your product roadmaps?but only if you are performing sales-derived win loss analysis. Even in our organization, where we specialize in helping companies implement win loss programs, if our product managers relied on win loss reports from sales reps alone to drive product strategy, we could quickly get ourselves into trouble. Fortunately, we have learned to incorporate a superior source of informationcustomers and prospectsthat provide us with data we need to make a roadmap we can trust and that will ultimately allow us to serve our customers needs more effectively.

In my next post I will write about how you can get a tremendous amount of value from sales-derived win loss reports and how to appropriately use them.

Note: If youve never had the opportunity to review a decision maker-derived win loss report I would like to change that now and extend an offer to do a couple win loss reviews for freeno expectation or obligation on your part. Contact me if youre interested in taking me up on this offer.

About the Author: Ken Allred, Founder and CEO of Primary Intelligence, is a thought leader in SaaS-based sales intelligence, analytics and sales enablement solutions. He is committed to the optimization of sales, marketing and product management teams through the implementation of advanced Sales 2.0 intelligence solutions.