Surveys and Confidence and Errors! Oh, my!

by Ken Allred, March 26, 2010

How many data points do you really need?

One of the common questions I am asked when discussing our win loss analysis offerings with companies is, “How many deals do I need to analyze before I can start using the data?” In the past, my response has always been that you will get value from the very first win, loss, or no decision that you analyze. However, because statistics teaches us about minimum confidence levels, confidence intervals, and margins of error, people often struggle with the idea that they can start leveraging the data from their win loss program from the very first data point.

Get away from the “survey” mindset

By just listening to the terminology that many executives use, it is obvious that many initially base their vision of a win loss program on their prior knowledge of market research or political polls. For instance, the term “survey” is often used to describe what is actually an in-depth “interview”. Unfortunately, this market research-based mindset often diminishes the usefulness of a win loss program and gives the false impression that the program will need hundreds (or even thousands) of data points to be accurate or useful.

Certainly, if you were to use a win loss program to try to determine the general buying habits of your buyers (or the populace as a whole), you would want to interview a large number of random customers in order to achieve a representative sample and a high level of statistical confidence. However, working with hundreds of companies over the past ten years has shown that the most successful win loss analysis programs do not try to cast such a wide net. Instead, they act more like a laser pointer that illuminates very specific information about very specific issues and questions. In other words, to get the most out of your win loss program, it should not be seen as a survey tool (which simply gathers information), but rather a decision-making tool and an early warning tool.

If you think that you can’t get value from your win loss program, or if you believe you can’t begin making decisions based on your win loss data until you achieve a statistically significant number of data points, you risk missing as much as 75 percent of the potential ROI you could achieve with your win loss analysis program. The key here is in understanding how and when to use the intelligence you are gathering through your win loss analysis.

The decisions executives and managers make

To determine how many interviews are needed for a win loss program to be effective in helping you with your decisions, it is important to understand the kinds of decisions you make in your organization and how win loss analysis can help you make better decisions. Professor Robert A. Harris, in his VirtualSalt blog, outlines three major types of decisions that individuals and organizations make: strategic, tactical, and operational.

Strategic decisions are those that determine the overall direction of an enterprise, product, or department over the long term. They also tend to direct and control an organization’s resources. Strategic decisions involve a high degree of uncertainty due to the scope and time frames usually attached to these types of decisions. Real-world examples of win loss analysis data helping clients make better strategic decisions include:

1. Analyzing competitive losses indicates that buyers want a vendor with strong off-shore capabilities in two specific countries
2. Competitive losses point to buyers having issues with the key contractual terms the client requires of new customers
3. Buyers in both wins and losses describe a specific set of needs that aren’t currently being addressed in the market and warrant the development of a new product

Tactical decisions are used to help support or achieve strategic decisions. In other words, tactical decisions are made to help bring about the results that are envisioned in the organization’s overall strategy. As such, they tend to be more immediate than strategic decisions, and can be made at both the managerial and executive level, whereas strategic decisions are almost exclusively determined by executives. Real-world examples of win loss program data that led to clients making tactical decisions include:

1. Competitive wins indicate that a key factor in buyers’ decisions was the presence of superior sales engineer support
2. Buyers in competitive wins talked about specific sales messaging and sales collateral that helped them make their decision, while buyers in lost opportunities did not reference these things
3. Buyers in losses cited over-confidence of their sales representatives and disparaging the competition as key reasons for non-selection

Operational decisions are everyday choices used to support tactical and strategic decisions. They help to ensure that daily operations go smoothly and that the tactical decisions are implemented. As the name indicates, these are usually made by managers, although they can also be made by executives in some cases. Examples of win loss program intelligence helping clients make operational decisions include:

1. A buyer in a lost opportunity indicates that their sales representative did a very poor job in presenting the solution and did not seem to really understand how their product would solve the buyer’s problems
2. A buyer in a won opportunity indicates that they were having trouble with communication and it was having a negative impact on their decision satisfaction
3. A buyer in what the client thought was a lost opportunity indicates that they are still evaluating solutions and that they would like to hear from the client (this situation happens more often than you think)
4. A buyer in a lost opportunity indicates that they didn’t select the client primarily due to a product feature the buyer thought was missing, but the client’s product actually had the feature the buyer wanted

The number of data points needed to support accurate decision-making increases as you move from operational (which is often immediate and one data point can be sufficient) to strategic, which may require a large amount of information to get a clear picture of the entire situation. For instance, deciding which paper supplier to use for your direct mail campaign (an operational decision) would probably not require you to interview clients on their preferences, but if you wanted to change the focus of your company from services to software (a strategic decision), it would be in your best interest to interview a representative number of your current clients to determine how they would react to such a change.

Win loss as a decision-making and decision support tool

From my experience over the last ten years, I have found that win loss analysis provides the most benefit at the tactical and operational level, although, certainly, it does provide tremendous insight at the strategic level as well. It just takes a little more time before you can start to lean on your win loss program when making those long-term strategic decisions.

A few months ago, I asked myself the question, “On average, how many potential operational decisions are presented to our clients in a typical win, loss, or no decision review?” In other words, how many opportunities for potential change (or no change) will a client be able to identify in reading a profile of a won or lost deal? I began a subjective, ad-hoc analysis to figure this out and help our clients understand the potential value of each completed opportunity analysis. Based on my analysis to date, for each win, loss, or no decision interview performed, you can expect to identify an average of ten operational or tactical decisions that can be acted upon.

When you consider that each analysis is going to help you make an average of ten operational or tactical decisions, it becomes clear very quickly that you can realize great value from the very first interview of your win loss program. Just how much value does each of these potential decisions represent? Let’s look at an example: one of the most common operational decisions identified in win loss reviews are sales weaknesses for an individual sales representative that can be immediately addressed through focused training and/or coaching. If this training can help that sales representative avoid making the same mistakes in future opportunities, their individual win rate will improve. The impact of this one decision to help a sales representative with an area of weakness identified by a buyer could be in the millions of dollars, depending on the size of your sales deals.

If done correctly, every win loss interview performed will net key, actionable intelligence in such areas as:

• What business needs are your buyers trying to solve?
• What could you do to better serve your buyers’ needs?
• What is the character and reputation of your company in the eyes of your buyers?
• How well do your sales teams communicate your value proposition?
• What are the strengths and weaknesses of your products?
• What are the strengths and weaknesses of your competitors’ products?
• How are your costs perceived in relation to those of the competition?

Last year, when I was reviewing some interviews we had done with our own buyers, I saw an issue emerge that definitely led to a better tactical decision. One of our respondents commented that their communication with Primary Intelligence was almost entirely with the salesperson and that they did not have sufficient contact with the Account Consultant who would be handling their program.  Since one of our strategic goals is to be a trusted consultant to our clients rather than just a provider of data, I realized that this was a significant issue. After getting the same feedback in two other interviews, I determined that this was not an isolated case and made the tactical decision to involve our Account Consultants and other subject matter experts early in the sales process. The results have been good, with our clients coming out of the sales process more informed and more committed to their win loss programs than we have seen in the past. This tactical decision was made based on three data points, but would have been entirely missed if we had not been performing win loss reviews on our own opportunities.

So, how many data points do you need?

The case I just described did not require many interviews or data points at all for the win loss analysis to have a positive impact—in fact, it only took one to begin the decision process. This example illustrates what we commonly see working with hundreds of clients. You can and will receive value from your win loss program from the very first opportunity that you analyze. And as you add more data points, you will receive further validation and discover important, new insights.

The point is this: since win loss analysis is a dynamic process, it can and should focus on those issues that are most important to you and your buyers. This means that, unlike a static survey, fewer data points are needed in a win loss analysis to reach the insights you need to make better tactical and operational decisions that will help you achieve your current strategic goals and overall success.

It would be a serious error to believe that you need to wait until you have a “sufficient number” of data points before you can realize value in your win loss program. Don’t make this mistake; begin analyzing and using your win loss data from your very first buyer interview and you will begin seeing returns much more quickly.

About the Author: Ken Allred, Founder and CEO of Primary Intelligence, is a thought leader in SaaS-based sales intelligence, analytics and sales enablement solutions. He is committed to the optimization of sales, marketing and product management teams through the implementation of advanced Sales 2.0 intelligence solutions.