PI Blog: Sales and Competitive Intelligence
Advice for sales, marketing & product management success
Analytics
More on Product Management Metrics
by Ken Allred, September 8, 2009
This post continues our ongoing discussion about product management metrics. To catch up on the discussion thus far, youll want to review Saeeds post on product managements mandate and my post on two examples of key product management metrics. The lively conversation about product management metrics got me thinking about good metrics and bad metrics and how to tell the difference.
There are four key aspects we should use to evaluate a potential metric:
- Do product managers have influence on the factor being measured (Do they have enough control over the factor to significantly affect it)?
- Is the metric a predictor of success?
- Is the metric actionable?
- Can you tie compensation to the metric?
The four criteria above can help you evaluate any metric you may be considering and give you an idea of their potential effectiveness. Unfortunately, there is such a thing as a bad metric, and there is a very real risk to your strategic objectives if you measure the wrong thing. A bad metric will cause you to focus on the wrong thingsyou may be successful in that metric, but you will ultimately miss the mark. However, a good metric that meets the criteria above can be a powerful motivator and an incredible tool.
There has been a lot of debate that if a person doesn’t have complete control of the thing being measured they shouldn’t be held accountable for itor it shouldn’t be a metric used to monitor their success. While I agree that the more control a person has over the thing being measured the better, my experience has taught me that if the person can exert significant influence on the thing being measured, even if they don’t have complete control, it can still be a fantastic metric if the other three factors can also be met (actionable, predictor and compensation tied to it).
After the influence test, the next important test of a metric is to ask yourself if performing well in this metric will lead to success 100 percent of the timeis the metric a predictor of success? If you can perform well in a given metric, but still fail at your strategic objective, then you need a better metric.
The third key test of a metric is to ask yourself if you can determine specific actions to take based on the metricis the metric itself actionable? Can you look at a metric at any given point in time and see specific actions you can take to improve in that metric? If you can’t, the metric isn’t actionable and you need a better metric.
And the last test, and one of my favorites, is whether or not you can tie compensation, or a portion of compensation to the metric. This isn’t absolutely a requirementthe other three tests are the most important when it comes to identifying good metricsbut if you can tie compensation to the metric, “you’ll be cooking with gas” as a buddy of mine likes to say.
In my experience running Primary Intelligence, we have implemented, monitored and then discarded so many different metrics for every role in the organization that it would be difficult to list them all. The one thing I’ve learned from this exercise is that internal metrics (activity-based), while interesting, will never measure up to external metrics (results-based)the metrics that directly measure, without ambiguity, our progress towards our strategic objectives. In the case of product management, we have already defined the strategic objective as “optimizing the business at a product, product line, or product portfolio level over the product lifecycle.”
In my previous post, I recommended two potential metrics we could use to measure our effectiveness as product managers:
- Product performance versus customer problems
- Product performance versus competitors’ product performance
I’m still inclined to use these two metrics because I believe they meet the four tests described above, they’re results-based metrics, and they have significant impact on the three drivers of revenue:
| Revenue Drivers | Product Performance |
|---|---|
| A prospect’s likelihood of purchasing our product | The probability that a customer buys our product directly correlates with how well they perceive our product will solve their problems |
| A customer’s likelihood of renewing, or purchasing more of our product | The probability that a customer renews with us directly correlates with how well our product actually solves their problems |
| A customer’s likelihood of recommending our product to a friend | The probability that a customer recommends us directly correlates with how well our product solves their problems |
I also believe that these two metrics are relatively easy to monitor using product management activities that are already (or should be) part of our process: talking to customers and evaluators.
This is the approach that I am using to set these metrics up for our own organization:
- Identify the key problems/business needs that our product solves for our customers
- Identify the product features that solve, or help solve, a specific customer problem (repeat for each key problem)
- Ask the customer to rate our performance in those features (talking to customers)
- Ask the customer to rate our performance and our competitors’ performance in those features (talking to evaluators)
- Track these metrics over time (probably quarterly)
The first step is probably the most important, as we have to make sure that we’re solving the right problems for our customers the problems they’re willing to pay for. For each key problem we want to solve for our customers, we need to identify the major features, or feature categories that help solve this problem for our customer.
For example, one of the key problem categories that we solve for our customers is their need for actionable, real-time competitive intelligence. Now that I’ve identified this problem, I have to examine our product for the key features that help solve this problem. The partial list that I came up with looked like this:
- Real-time competitor SWOT analysis
- Role-based CI dashboards
- Reporting capabilities
- Competitor pricing analysis
Once I have the key features identified, I am ready to measure the performance of our product in solving this specific problem for our customers. I do this through two types of interviewing:
- Talking to customers through customer satisfaction interviews or impromptu customer interviews
- Talking to evaluators in recent competitive wins and lossesour win loss analysis program
Performing this analysis can lead me to create a flow chart based on our performance scores that looks something like the following:
Additionally, I can create a similar flow chart to compare our performance versus each of our primary competitors’ product performance that would look something like the following:
Let’s ask the tough questions about these two metrics now:
Do we as product managers have enough control or influence over these two areas we will be measuring? I think we do. Do others affect these metrics? Absolutelybut I don’t think that should be used as an argument against these metrics because our mandate as product managers is to build products that solve problems customers are willing to pay for. Sales, marketing and support all play important parts in this, but product managers really are the foundation. If we have the foundation right, we can help fix sales, marketing and support problems that may be negatively affecting our metrics.
Will these metrics predict our success in optimizing our products over the product life-cycle? I think they will. The better we are at solving problems customers will pay for, the higher these metrics will be and the more likely we will be to meet our strategic product management objectives.
Can I look at these metrics and immediately identify specific actions to take to improve them? I think we can. The great thing about these metrics is they immediately identify both risks and opportunities that we can act on.
Can we tie compensation to these metrics? As the CEO, I can tell you that these are exactly the type of metrics I would want to tie compensation to.
The key to implementing these metrics is making sure that I am carefully aligning my product managers to focus on the most important thing they can do to impact our businessanalyzing and improving how well we are solving our customers’ problems.
About the Author: Ken Allred, Founder and CEO of Primary Intelligence, is a thought leader in SaaS-based sales intelligence, analytics and sales enablement solutions. He is committed to the optimization of sales, marketing and product management teams through the implementation of advanced Sales 2.0 intelligence solutions.
Two Key Product Management Metrics
by Ken Allred, September 4, 2009
Saeed, in his blog post at On Product Management, posited the question, why is it difficult to measure the value and contribution of product management? To help us focus on the right metrics, he defined Product Management’s mandate as:
“Product Management’s mandate is to optimize the business at a product, product line, or product portfolio level over the product lifecycle.”
This is a great question and his definition of the Product Management’s mandate really got me thinking.
Webster’s definition of “optimize” is to make perfect, effective, or as functional as possible.
That means that the product manager’s mandate is to make the product as perfect (or effective) as they possibly can. If we then define a “perfect product” as completely solving our customers’ problem, I think we can start to think of creative ways to measure how well were accomplishing this mandate.
So, what are some of the ways we could measure how well we are solving our customers’ problems?
Product features or internal performance benchmarking? While I do think that measuring the internal aspects of product management is important, I would propose that measuring the actual results of product management is much more vital.
How about how much revenue the product is producing? Product revenue is certainly a result of our efforts in product management and certainly a good thing to monitor, but it probably isnt the best way to measure product management performance, as there are so many factors that are beyond the control of a product manager: sales process, sales channel, sales effectiveness, marketing strategy, marketing budget, etc. All of these factors will have a significant impact (negative and positive) on product revenue.
Instead of looking at these, I would propose two key metrics to measure your effectiveness in achieving the product optimization goal:
- Your product performance versus customer problems
- Your product performance versus competitors’ product performance
The first metric allows us to measure how well our solution is solving our customers problems. It will also allow us to identify gaps in our features and identify areas that need improvement all in an effort to “more perfectly” solve the customers’ problem.
The second metric is important because it allows us to see how well we’re doing as product managers in making sure our products are superior to the competitions’ products. This is only important insomuch that we define being superior to the competition as being able to solve customers’ problems better than the competitions’ solutions.
Measuring the first metric without measuring the second is a lot like a sprinter running a race and never checking in to see where the other runners are during the racethey just keep their eye on the finish line. You can still win races this way, but its a lot easier for your competition to sneak up and overtake you if you’re not monitoring their progress. You can be sure they’re keeping their eye on you.
If I’m measuring and monitoring these two areasthe ongoing results of my product management effortschances are the other things like product revenue, market share, sales enablement and bottom-line results are going to be meeting or exceeding expectations.
Im really looking forward to Saeeds follow-up post to see what kind of metrics he comes up with. What do you think? Are these two metrics that product managers should be monitoring? Or, are there others that are more important for determining the perfection of your solution?
About the Author: Ken Allred, Founder and CEO of Primary Intelligence, is a thought leader in SaaS-based sales intelligence, analytics and sales enablement solutions. He is committed to the optimization of sales, marketing and product management teams through the implementation of advanced Sales 2.0 intelligence solutions.
- Mark Larson (December 30th, 2009 at 10:38 pm)
please provide a link to the post you quote and that will be fine. Follow our CEO on Twitter at ... - malk616 (December 25th, 2009 at 3:49 pm)
I want to quote your post in my blog. It can? And you et an account on Twitter?
7 Steps to Identifying & Validating Market Problems
by Ken Allred, September 1, 2009
This week, we kick off our series of blog posts focused on showing how win loss analysis can help you become an expert on your market and gain a deeper understanding of the problems inherent in your target markets. Pragmatic Marketing teaches that the Market Problems activity is the cornerstone of their framework, defining this activity as discovering problems in the market by interviewing customers, recent evaluators, and untapped potential customers, as well as validating the problems identified to show their pervasiveness and impact on your market.
New product failure rates are high. Studies indicate that we can see new product failure rates of nearly 90 percent. A study done in the food industry showed that the failure rate for new products was between 70 and 80 percent, but that the U.S. Top 20 food companies were enjoying a success rate of 76 percent for their new product introductions, while the other 20,000 food companies had only an 11.6 percent success rate for their new products. The biggest difference between the top 20 and the bottom 20,000? The lack of market research.
Studies also show that a third or more of a companys revenue is likely derived from products that did not exist five years ago. There is no question that if a company is to grow and survive, they have to be able to create and launch new products, or at the very least enhance and upgrade their existing products. Take a look at Motorola, a company that led the cell phone industry. At its peak in 2000, Motorola had a market cap that was 8 times larger than their market cap today. Motorola has been struggling for the last couple of years; in fact, their last new cell phone product launch was in 2004. The market has changed tremendously over the last five years, and given the current state of the company, its pretty safe to say that they havent been listening to their markets.
I just realized I used the word product nine times in the preceding two paragraphs to describe the landscape that is product development, management, and marketing. The truth is, it isnt about products. People dont buy products; they buy solutions to their problems. In The Business Battlecard, author Paul ODea quotes Harvard Marketing Professor Theodore Levitt to illustrate the importance of understanding your customers pain:
People dont want to buy a quarter-inch drill. They want a quarter-inch hole.
If we want to create value for our customers, ensure our products success, and secure our companys future, we have to understand our customers needs and problems better than our competitors. There is really only one way to do thatwe have to talk to our market, and more specifically, the decision makers in our target markets.
Based on the new product failure rates that I quoted earlier, we can assume that most product development and management is done using activities performed inside the companyi.e. founders and executives drive product development based on their experience and gut, while occasionally checking in with customers. This is typically the place where everyone has to start when we are creating a new product, but at some point (the earlier the better), the process needs to change from an internally-driven development process to a customer-driven, external process.
In order to create a customer-driven product development process, we must be consistently listening to our market to identify and validate the key customer problems that our solution will solve and measure how well our solution is solving those problems. Pragmatic Marketing defines three groups within our target markets that we need to be actively listening to:
- Existing Customers: this group includes customers that recognized they had a problem and purchased your solution to solve that problem
- Evaluators: this group includes both your customers (competitive wins) and those customers that have selected your competition (competitive losses)
- Potentials: the group of potential customers that havent purchased your type of product, but have tried to solve the same problems through internal efforts
Over the last decade, we here at Primary Intelligence have spent countless hours figuring out how to best listen to the second group identified by Pragmatic Marketingthe Evaluators. Based on that experience, we have found the following seven-step process is an effective methodology for improving your understanding of your markets problems:
- Make sure you are asking questions that will reveal the problems the customer is trying to solve.
- Before you begin gathering and evaluating our win loss data, write down your target markets three to five most important or pressing problems.
- Examine demographic information for the recent wins and losses you are going to interview in order to make sure they are in your target markets (you dont want to identify market problems for markets that you dont want to be in).
- Perform the win loss analysis and gather the data.
- Build a repository of the problems identified and validated.
- Analyze the data to identify problems you may have missed. You also want to validate that the problems you wrote down are real for your markets and not problems that you hope they have.
- Remember that identifying market problems and validating those problems is a program and not a project. You want to build this process into your weekly/monthly activities to make sure that you are able to stay on top of your markets
STEP ONE: Ask the Right Questions
Make sure that you have your questions prepared before you perform the interview with the decision maker. It will be tempting to just pick up the phone and wing it, but being prepared with the questions you want to get answered before-hand will dramatically improve the probability that you will accomplish your objectives.
Based on our experience, we have found that you need to ask the following questions to both identify and validate market problems:
- What were your business needs that prompted this evaluation?
- What were the primary reasons you selected the winning vendor over the other vendors you evaluated?
- What were the primary reasons you did not select the other vendors you evaluated?
- What could the vendors you didnt select have done differently to win your business?
- What could the winning vendor have done differently to better meet your business needs?
- What did you value most about the vendors solutions you evaluated?
- What were the biggest solution weaknesses you identified based on your business requirements?
- What important features were missing from the solution?
There are also several quantitative questions that will help you validate market problems and quantify how well your solution is solving those problems, but Ill need to save that for a follow-up discussion to try to keep the length of this post down.
The first question (what were your business needs that prompted this evaluation?) is your money question when it comes to identifying and understanding your target markets problems. In our experience, the other questions will also identify and validate your customers problems. They wont do it consistently, but they will regularly give you insight into whats important to your customers and why. All of this translates into how well youre solving your customers problems and helps you along the path of becoming stronger experts on your target markets.
STEP TWO: Write Down Your Target Markets Problems
An important part of this ongoing process is validating what you currently think about our customers problems as well as qualifying the impact those problems are having on our customershow important is it to our customers to solve these problems?
An interesting exercise that you can do in conjunction with this is to send an e-mail out to others in your organization (sales, operations, executives, etc.) and ask them what they believe the top three problems are that you solve for your customers, and ask them to rank them in order of importance to the customer.
In preparation for this article, I reached out to our sales, operations, and executive personnel and asked them to identify the top three problems we solve for our customers. I asked them to rank them from most important problem to least important problem. I then took the responses and weighted them to come up with the following list of problems our organization believes we solve for our customers:
Top Three Problems We Solve
According to Us
| Weighted Importance | |
|---|---|
| Better Competitive Intelligence | 41 |
| Understanding why they win and lose | 30 |
| Sales Process (SWOT) | 24 |
| Better understand target market needs | 14 |
| Understanding market perceptions | 11 |
| Improve sales performance and win rates | 8 |
| Product (SWOT) | 7 |
| Marketing program and messaging effectiveness | 7 |
| Identify previously unknown areas of improvement | 4 |
| Non-biased win loss analysis & expertise | 4 |
| Identify sales best practices | 4 |
| Better understand buying process | 4 |
| What happened in this deal? | 3 |
| Disseminate W/L intelligence throughout organization | 3 |
| Identify sales training opportunities | 3 |
| Understand pricing impact on decisions | 3 |
You will notice that the list above isnt necessarily organized as problem statements. However, organizing things in this way allows us to figure out which category of problems are most important to our target markets. This is perfectly acceptable, as long as you have a deep understanding of the problems your customers are facing and youre able to articulate them using the same words your customers use. The trap that you have to be careful of here is taking a short-cut and falling back onto your benefit statements instead of really dissecting the problems as articulated by decision makers in your target markets.
For example, when one of our decision makers was asked what business problem they were trying to solve when they selected our solution, he responded, We need to gain specific competitive insight on [One of their competitors]. They have been beating us with greater frequency and we need to figure out how to counter this. We can see that this is clearly a Better Competitive Intelligence category problem. In contrast, the following response really belongs in the Better understand target market needs problem category:
We need to get to a more refined understanding of our markets so we can figure out why there is a gap between what we think customers should be spending and what they actually are–we need to find out why this delta exists and use this information to make changes to our go-to market strategies, our product plans and perhaps even our business model.
According to this internally-derived list, the top three problems that I should be looking to validate, in addition to trying to identify new problems, are Competitive Intelligence-related problems, helping customers understand why they win and lose, and helping customers understand the strengths, weaknesses, opportunities, and threats in their current sales process (Sales Process (SWOT)).
STEP THREE: Select Recent Wins and Losses to Review and Qualify Demographics
You want to examine the key demographic information of your recent wins and losses to make sure that they fit within your target markets. The best sales professionals and managers will tell you that there really are only two reasons you lose a deal (versus the more commonly voiced, but generally incorrect, reasons of price and product):
- You shouldnt have been in the deal in the first place (your solution wasnt a good fit based on the customers needs)
- You were outsoldthe competition did a better job selling than you did.
Because walking away from any deal is a very difficult thing to do for a sales representative trying to make quota, you will have losses in your database that are not in your target markets. It is very important that you do your best to remove these from the list of competitive wins and losses you will be using to perform your win loss analysis.
STEP FOUR: Interview Decision Makers
To help with best practices when interviewing decision makers, I went to our team of account consultants, the folks that spend at least half of every day talking to decision makers, and asked them for advice they would give. They provided both some best practice dos and donts that will help you improve your interviews and response rates:
Do these things:
- Pre-notify the responsible sales representative that you will be doing a win loss review on the deal.
- Let the sales representative know that you will share the results of the review with them so that they can learn from the effort as well.
- Ask the sales representative to notify the decision maker that you will be calling to do a debrief on how they made their purchase decisionthis will improve response rates significantly. Its important that you craft an e-mail message for the sales representative to send to the decision maker that emphasizes the following:
- You are not calling to try to win them back, or re-engage the selling process (this applies to your losses).
- You are calling to get their feedback so that you can improve your solution and sales processes.
- You are in product management and not sales.
- Listen. Listen. Then listen some more. There are really only three situations where you should be talking:
- Youre asking the interview questions.
- Youre asking a follow-up question.
- Youre answering a question about one of the questions you just asked.
- Keep the interview conversationaldont sound like a call center survey.
- Always know their title, correct spelling of their name and gender prior to the call. Only verify information with them (e.g. I have your title as CFO. Is that the most accurate title?)
- Make sure you have follow-up, or delving, questions prepared beforehand for each of your questions so that you can get actionable feedback
- Smile while you interviewit comes through in your voice
- Thank them for providing candid and informative feedback, especially when the feedback might be considered negative for you, at appropriate times in the interviewthis will encourage them to continue to provide candid feedback.
Dont do these things:
- Dont go into the interview with a predisposed view of what happened in the deal.
- Dont lead person you are interviewing, or ask leading questions. For instance, dont ask Did you reject the solution because the price was too expensive for you? Instead, ask something like, Why did you not select the solution?
- Never assume that you know the answer to a question because of their responses to previous questions.
- Dont interruptthis will be a challenge because they will say something that you can answer, and you will want to jump in and respond to what they just said, but if you do, you will miss out on critical information.
- Dont frustrate them with an overly long interview; keep your interview as succinct as you can and be mindful of their time constraints.
- Dont try to solve all of your problems in one interview.
- Dont add your own opinion or additional commentary to their responses.
STEP FIVE: Build a Repository
While individual win reports or loss reports are very important to understanding your target markets problems and how well your solution addresses those problems, having a database that you build over time will allow you to validate these problems and identify important trends in the market problems that your solution addresses.
When trending or aggregating information from a group of interviews, keep the following in mind:
- Try to maintain the integrity of your marketsif you are interested in more than one target market, try to separate your interviews into groups that each fit a single target market and do a trend analysis for each group.
- When trending, having an equal mix of win reports and loss reports will help avoid positive or negative bias in your overall findings. Being able to see how performance differs between wins and losses will also help pinpoint areas of greatest impact.
- Remember that certain questions work better for determine overall performance (e.g. performance ratings) and some are more individual-based, making them better for identifying a range of best practices (e.g. an open-ended question asking how the vendor could better serve the customers needs).
STEP SIX: Analyze Stated Customer Problems
Once you have our database set up, you can get to the fun partfiguring out what problems are most important to your target markets and how your solution can help them solve those problems. To help illustrate this, I analyzed ten recent opportunities that evaluated Primary Intelligence (6 wins and 4 losses) to identify the market problems that caused them to seek a solution. The results are very interesting, especially when you compare them with our internally-derived list of customer problems that we solve:
Top Three Problems We Solve
According to Our Evaluators
| Weighted Importance | |
|---|---|
| Better Competitive Intelligence | 13 |
| Improve current win loss program | 12 |
| Non-biased win loss analysis & expertise | 10 |
| Understanding why they win and lose | 10 |
| Product (SWOT) | 5 |
| Sales Process (SWOT) | 4 |
| Better understand target market needs | 3 |
| Understanding market perceptions | 3 |
| Marketing program and messaging effectiveness | 1 |
| Disseminate W/L intelligence throughout organization | 1 |
| Improve sales performance and win rates | 0 |
| Identify previously unknown areas of improvement | 0 |
| Identify sales best practices | 0 |
| Better understand buying process | 0 |
| What happened in this deal? | 0 |
| Identify sales training opportunities | 0 |
| Understand pricing impact on decisions | 0 |
Keep in mind that this is only ten recent opportunities; however, we can immediately see that we missed an important market problem from our internal listevaluators indicated that improving their current win loss program was one of the most important problems that was driving them to look at our solution.
This process allows me to understand the big picture for our target markets, and analyzing the way each customer communicates their problems gives me fantastic insight into areas that we can/should be focusing on to better meet the needs of our customers. It is comments like the following that help me frame the newly identified customer problem of improving a current win loss analysis program:
“We were doing our own loss reviews, and we were finding that we were not getting good, actionable information from them. Then we found out that Primary Intelligence’s services were available and saw that the level of information was much deeper than we were able to get by doing our own loss reviews. We felt doing our own reviews internally wasn’t as deep because it wasnt our area of expertise.”
“We were trying to gain a more robust process. We were strapped for resources in-house and we were looking for a solution that would allow us to be able to touch more deals than we currently touch, and therefore get a bigger wealth of information as to why we are winning and losing. We have an internal process, but we think we are getting rote answers. We had not gone with a third party ever before.”
We can see that while they definitely want to improve their current win loss analysis programs, our customers are framing this problem around the idea of deeper information, actionable information, wealth of information, etc. These, then, are the real problems underlying the stated problem of improving their current win loss programs.
Analyzing the data in this way helps me understand the problems and their impact, but just as important is the fact that Im able to get a good feel for which problems are most important to our target marketsthat is, which problems are the ones our customers are willing to pay to solve.
The importance ranking that we did internally was quite a bit different than what actually resulted from listening to evaluatorsas you can see when we look at the top ten problems side-by-side:
Top Three Problems We Solve
Evaluator Ranking vs. Internal Ranking
| Evaluator Ranking | Internal Ranking | |
|---|---|---|
| Better Competitive Intelligence | 1st | 1st |
| Improve current win loss program | 2nd | - |
| Non-biased win loss analysis & expertise | 3rd | 10th |
| Understanding why they win and lose | 4th | 2nd |
| Product (SWOT) | 5th | 7th |
| Sales Process (SWOT) | 6th | 3rd |
| Better understand target market needs | 7th | 4th |
| Understanding market perceptions | 8th | 5th |
| Marketing program and messaging effectiveness | 9th | 8th |
| Disseminate W/L intelligence throughout organization | 10th | 14th |
STEP SEVEN: Make Market Problem Identification and Validation a Habit
Remember that win loss analysis isnt a project; its a program. Your recent evaluators are one of the best places (if not the best place) you can go to validate the problems you believe you are solving with your solution.
The things that influence and drive your target markets are fluid and always changing. If there is one constant that we have seen analyzing purchase decisions over the last ten years, it is that markets will change. The problems that you are solving today, while important, may not be the most important problems your customers need you to solve tomorrow. If you are not listening, you can bet one of your competitors will be.
About the Author: Ken Allred, Founder and CEO of Primary Intelligence, is a thought leader in SaaS-based sales intelligence, analytics and sales enablement solutions. He is committed to the optimization of sales, marketing and product management teams through the implementation of advanced Sales 2.0 intelligence solutions.
- Sue Massey (September 1st, 2009 at 11:31 am)
I found your site on Google and read a few of your other entires. Nice Stuff. I'm looking ...
Decision Maker-derived vs. Sales-derived Win Loss Analysis
by Ken Allred, August 11, 2009
Which Methodology is Best for Product Managers?
Recently, respected product managers have been discussing the value of win loss reports as they relate to product strategy and technology roadmaps. In an interesting blog post on Product Management Insights, Michael Shrivathsan argues Why Win/Loss Reports Shouldnt Drive Your Product Roadmap. In his article, Shrivathsan outlines why doing win loss reports with your sales team might be unreliable, including the fact that there might be hidden bias in the sales teams perceptions. He cites a humorous tweet from Mike Boudreaux, a fellow Product Management twitterer, to illustrate his point: Typical win/loss analysis from sales force: majority of losses due to product and price. Majority of wins due to relationship.
As I read this article, I could understand Shrivathsans argument, but only if you define and develop your win loss program in a narrow and short-sighted way. By operationalizing your win loss in the way described, or by abandoning win loss analysis altogether, youll be missing out on perhaps the most important intelligence you need to validate your product strategies and roadmaps.
To begin, an important distinction needs to be made concerning the win loss analysis methodology being discussed in Shrivathsans post. From his comments, we can infer that the win loss analysis methodology being utilized is to have the responsible sales representatives identify the reasons they won or lost a deal and report this information back to product management and other interested parties. I define this as a sales-derived win loss analysis methodology.
This is a common conception of win loss analysis. Based on our experience at Primary Intelligence, we know that approximately 1/3 of companies indicate that they perform win loss analysis; however, we have found that only about 50% of those companies that say they have a win loss program implement what we define as a decision maker-derived win loss analysis methodology as opposed to a sales-derived one.
The distinction between these two very different types of win loss programs is very important. In fact, to fully illustrate this point I went to our own win loss program and grabbed a recent real-world example. This is an analysis of a recent competitive win, but the principal holds true for losses as well (if there is interest, I can show a real-world loss example at a later date).
Lets compare the results between a sales-derived win review and a decision maker-derived win review.
Selection Reasons
When our sales representative was asked why we won the opportunity, his response was:
The reason we won the deal is because I was tenacious and kept working on the deal and building relationships until I got in front of the right person. I then built my relationship with that person so that we had a great relationship. It was good that we had technology, but the primary reason they selected us was because of the relationship I was able to build with the decision maker.
Does this sound familiar? This response sounds similar to the kind that Shrivathsan laments in his blog post. However, lets compare our sales representatives response to the decision makers response on why they selected Primary Intelligence:
There were a number of reasons. One is their extensive experience in this business. [PIs sales rep] and [PIs account consultant]s personalities were a compelling factor, as were the quality of the work and the technology. We really liked the dashboard. Were just now starting to tap into the power of the technology. Primary Intelligence also gave us the ability of integrating the win loss data into Salesforce.com, our CRM system. The others could not do that. Those would be the primary selection reasons.
We can see from the decision makers comments that our sales reps relationship with the decision maker (along with our account consultant) absolutely had an impact on the decision. However, if thats all the information we had, we would be missing valuable intelligence on how this customer actually arrived at their purchase decision.
More importantly, our product management team could be led to believe that the integration with our customers SFA tools we have been investing in may not be a high priority for our customers and we could begin to question our product strategy.
Competitor Weaknesses
An important element of every win loss program is gaining a better understanding of your competitors products. Lets take a look at the competitive intelligence gaps we had in the different win loss analysis methodologies.
Our sales representative was asked why our primary competitor lost the deal. His response was:
They werent pleased with the depth of the information [our competitor] showed in their samples. The decision maker was really looking for a strategic partner and they didnt feel they could get that from [our competitor].
And the decision makers response to why they didnt select our competitor:
Primarily it was the quality of their work. I found the quality of the work was not very good and the questions were not followed up on. The deliverable was fairly weak and looked very unprofessional.
Our sales representative did a pretty good job describing the situation, but the decision makers response is a lot clearer when it comes to trying to understand our competitors weaknesses. The decision maker identifies three distinct weaknesses, while our sales rep identified only two weaknessesone that was confirmed by the decision maker and one that wasnt.
Valued Solution Features
Now lets dig a little deeper and focus on the solution features to see if we can identify additional gaps between sales-derived and decision maker-derived win loss programs. We asked our sales representative what features of the evaluated solutions were valued the most by the decision maker:
Quality of interviewing, because we do a much better job than [PIs competitor] and really uncover the things that affect the outcome of purchase decisions.
Quality of interviewing is definitely an important feature of our solution, but was it really the feature that the decision maker valued most? In our win loss interview, we asked the customer to rate the quality of several product criteria for both PIs solution and the solutions offered by the competition:
We can see here that while our sales rep was partially correct in that we did perform well in the quality of interviewing decision criterion, the decision maker indicated that the quality of deliverable and tools and dashboards were what they valued most. The customer commented:
Primary Intelligences deliverable is very balanced between quantitative and qualitative information. The executive summary thats produced saves me a lot of time and effort and I get a lot of value from the analytics we get like the competitive advantage scores and predictive analytics. And the value of delivering everything that is produced through the dashboard is really important.
If our current product strategy is to invest in specific technology features and we were to see several sales-derived win reviews like this one, we might begin to question our current plans. Seeing enough reviews like this might even cause us to halt development and redirect those resources to other projects. You can see how it would be easy to make some very costly mistakes if you are ONLY reviewing sales-derived win loss data.
Conclusion
The following table clearly illustrates the large differences between the two methodologies we have been discussing:
| Sales-derived | Decision Maker-derived | |
|---|---|---|
| Selection Reasons | Relationship | Relationship Experience Quality of deliverable Tools & dashboards SF.com integration |
| Competitor Weaknesses | Depth of information | Depth of information Quality of deliverable Weak/unprofessional |
| Valued Solution Features | Quality of interviews | Quality of deliverable Tools & dashboards Executive summary Advantage scores Predictive analytics |
If you, or your organization, currently defines win loss analysis as a debrief from the responsible sales representative, I hope these examples can help shine some light on the big, risky gaps that are inherent in performing post-decision analysis via the sales representative.
To be perfectly fair, Shrivathsan does make a brief mention of including decision makers in win loss analysis, saying, theyre often much more open & honest to a product manager with whom they have no relationship, than a sales rep who worked with them for a period of time. As he alludes to, and as we have found in our own experience, these same principals weve been discussing will apply if you are having your sales representatives interview decision makers, so its important to have a third party of some sort perform the analysis (Ill be posting on this subject in more depth at a later point).
I have to agree with Shrivathsans assessment that you shouldnt allow sales rep-derived win loss reports to drive or affect your product roadmaps?but only if you are performing sales-derived win loss analysis. Even in our organization, where we specialize in helping companies implement win loss programs, if our product managers relied on win loss reports from sales reps alone to drive product strategy, we could quickly get ourselves into trouble. Fortunately, we have learned to incorporate a superior source of informationcustomers and prospectsthat provide us with data we need to make a roadmap we can trust and that will ultimately allow us to serve our customers needs more effectively.
In my next post I will write about how you can get a tremendous amount of value from sales-derived win loss reports and how to appropriately use them.
Note: If youve never had the opportunity to review a decision maker-derived win loss report I would like to change that now and extend an offer to do a couple win loss reviews for freeno expectation or obligation on your part. Contact me if youre interested in taking me up on this offer.
About the Author: Ken Allred, Founder and CEO of Primary Intelligence, is a thought leader in SaaS-based sales intelligence, analytics and sales enablement solutions. He is committed to the optimization of sales, marketing and product management teams through the implementation of advanced Sales 2.0 intelligence solutions.
Understanding Competitors
by Mark Larson, December 16, 2008
For an experienced salesperson, it is natural to navigate many potential hazards in a sales scenario because they have seen them before. They understand how competitors position themselves and how the market perceives their solutions.
The problem is, how does the inexperienced sales rep learn these valuable lessons? When it comes to understanding competitors, most salespeople learn in two ways: experience in competing against vendors over time and anecdotal comments made by colleagues. This isn’t good news for someone new. They end up losing plenty of potential business as they learn on the fly.
Primary Intelligence created Competitive Navigator to give all salespeople access to the competitive intelligence that would normally take years to obtain on their own. With a central intelligence location, all sales people receive the same training and information that helps them compete against any competitor.
Primary Intelligence does this by interviewing prospects and clients from your previous sales scenarios, regardless of whether they were wins or losses. In doing so, Primary Intelligence receives perceptions of vendors and their products directly from the person who is evaluating and buying them.
Whenever a competitor is encountered, Primary Intelligence asks key information about the prospect’s perceptions of the company, its solutions, and its sales team activities. As more instances of competitor activity are encountered, the picture becomes clearer on the market’s view of the vendor’s products, reputation, and sales practices.
Primary Intelligence is also gathering intelligence about your own company, solutions, and sales practices at the same time. Therefore, it is possible to compare your strengths and weaknesses against those of each competitor. By doing so, Competitive Navigator points out strategic methods you can use to beat any competitor in any sales scenario.
For more information about Competitive Navigator, click here.