More on Product Management Metrics

by Ken Allred, September 8, 2009

This post continues our ongoing discussion about product management metrics. To catch up on the discussion thus far, youll want to review Saeeds post on product managements mandate and my post on two examples of key product management metrics. The lively conversation about product management metrics got me thinking about good metrics and bad metrics and how to tell the difference.

There are four key aspects we should use to evaluate a potential metric:

1. Do product managers have influence on the factor being measured (Do they have enough control over the factor to significantly affect it)?
2. Is the metric a predictor of success?
3. Is the metric actionable?
4. Can you tie compensation to the metric?

The four criteria above can help you evaluate any metric you may be considering and give you an idea of their potential effectiveness. Unfortunately, there is such a thing as a bad metric, and there is a very real risk to your strategic objectives if you measure the wrong thing. A bad metric will cause you to focus on the wrong thingsyou may be successful in that metric, but you will ultimately miss the mark. However, a good metric that meets the criteria above can be a powerful motivator and an incredible tool.

There has been a lot of debate that if a person doesn’t have complete control of the thing being measured they shouldn’t be held accountable for itor it shouldn’t be a metric used to monitor their success. While I agree that the more control a person has over the thing being measured the better, my experience has taught me that if the person can exert significant influence on the thing being measured, even if they don’t have complete control, it can still be a fantastic metric if the other three factors can also be met (actionable, predictor and compensation tied to it).

After the influence test, the next important test of a metric is to ask yourself if performing well in this metric will lead to success 100 percent of the timeis the metric a predictor of success? If you can perform well in a given metric, but still fail at your strategic objective, then you need a better metric.

The third key test of a metric is to ask yourself if you can determine specific actions to take based on the metricis the metric itself actionable? Can you look at a metric at any given point in time and see specific actions you can take to improve in that metric? If you can’t, the metric isn’t actionable and you need a better metric.

And the last test, and one of my favorites, is whether or not you can tie compensation, or a portion of compensation to the metric. This isn’t absolutely a requirementthe other three tests are the most important when it comes to identifying good metricsbut if you can tie compensation to the metric, “you’ll be cooking with gas” as a buddy of mine likes to say.

In my experience running Primary Intelligence, we have implemented, monitored and then discarded so many different metrics for every role in the organization that it would be difficult to list them all. The one thing I’ve learned from this exercise is that internal metrics (activity-based), while interesting, will never measure up to external metrics (results-based)the metrics that directly measure, without ambiguity, our progress towards our strategic objectives. In the case of product management, we have already defined the strategic objective as “optimizing the business at a product, product line, or product portfolio level over the product lifecycle.”

In my previous post, I recommended two potential metrics we could use to measure our effectiveness as product managers:

1. Product performance versus customer problems
2. Product performance versus competitors’ product performance

I’m still inclined to use these two metrics because I believe they meet the four tests described above, they’re results-based metrics, and they have significant impact on the three drivers of revenue:

Revenue Drivers	Product Performance
A prospect’s likelihood of purchasing our product	The probability that a customer buys our product directly correlates with how well they perceive our product will solve their problems
A customer’s likelihood of renewing, or purchasing more of our product	The probability that a customer renews with us directly correlates with how well our product actually solves their problems
A customer’s likelihood of recommending our product to a friend	The probability that a customer recommends us directly correlates with how well our product solves their problems

I also believe that these two metrics are relatively easy to monitor using product management activities that are already (or should be) part of our process: talking to customers and evaluators.

This is the approach that I am using to set these metrics up for our own organization:

1. Identify the key problems/business needs that our product solves for our customers
2. Identify the product features that solve, or help solve, a specific customer problem (repeat for each key problem)
3. Ask the customer to rate our performance in those features (talking to customers)
4. Ask the customer to rate our performance and our competitors’ performance in those features (talking to evaluators)
5. Track these metrics over time (probably quarterly)

The first step is probably the most important, as we have to make sure that we’re solving the right problems for our customers the problems they’re willing to pay for. For each key problem we want to solve for our customers, we need to identify the major features, or feature categories that help solve this problem for our customer.

For example, one of the key problem categories that we solve for our customers is their need for actionable, real-time competitive intelligence. Now that I’ve identified this problem, I have to examine our product for the key features that help solve this problem. The partial list that I came up with looked like this:

1. Real-time competitor SWOT analysis
2. Role-based CI dashboards
3. Reporting capabilities
4. Competitor pricing analysis

Once I have the key features identified, I am ready to measure the performance of our product in solving this specific problem for our customers. I do this through two types of interviewing:

1. Talking to customers through customer satisfaction interviews or impromptu customer interviews
2. Talking to evaluators in recent competitive wins and lossesour win loss analysis program

Performing this analysis can lead me to create a flow chart based on our performance scores that looks something like the following:

Additionally, I can create a similar flow chart to compare our performance versus each of our primary competitors’ product performance that would look something like the following:

Let’s ask the tough questions about these two metrics now:

Do we as product managers have enough control or influence over these two areas we will be measuring? I think we do. Do others affect these metrics? Absolutelybut I don’t think that should be used as an argument against these metrics because our mandate as product managers is to build products that solve problems customers are willing to pay for. Sales, marketing and support all play important parts in this, but product managers really are the foundation. If we have the foundation right, we can help fix sales, marketing and support problems that may be negatively affecting our metrics.

Will these metrics predict our success in optimizing our products over the product life-cycle? I think they will. The better we are at solving problems customers will pay for, the higher these metrics will be and the more likely we will be to meet our strategic product management objectives.

Can I look at these metrics and immediately identify specific actions to take to improve them? I think we can. The great thing about these metrics is they immediately identify both risks and opportunities that we can act on.

Can we tie compensation to these metrics? As the CEO, I can tell you that these are exactly the type of metrics I would want to tie compensation to.

The key to implementing these metrics is making sure that I am carefully aligning my product managers to focus on the most important thing they can do to impact our businessanalyzing and improving how well we are solving our customers’ problems.

About the Author: Ken Allred, Founder and CEO of Primary Intelligence, is a thought leader in SaaS-based sales intelligence, analytics and sales enablement solutions. He is committed to the optimization of sales, marketing and product management teams through the implementation of advanced Sales 2.0 intelligence solutions.