We are often asked what kinds of metrics should be used for product development (PD). Metrics are frequently seen as a waste of time by the project team because they don’t seem to drive improvements in product development. So why do we use metrics? We use them because, when properly defined, they help identify both areas that work well and areas for improvement.[i]
Metrics are simply a set of numbers that give information about a particular process or activity.[ii] How the measurements are used is what brings the value to collecting metrics. A non-product development example helps to illustrate what is meant by the analysis. Say the most recent lot of parts from your production line are all on one side of the target value. They are still in spec. Do you have a problem or not? This is where the analysis comes in. Look at the 2 plots below.
From these 2 plots you can see that the second one shows a need to take action, now. Simply relying on ‘everything is in spec’ wouldn’t cause you to take action until after the parts start failing. In evaluating new software ideas, the use of A/B testing has become widespread. Being able to identify problems or successes early and respond them is the biggest value to collecting and evaluating metrics.
It is also important to note that people also behave according to how they are measured. Hauser and Katz summarized pitfalls that lead to counter-productive metrics.[iii] Therefore, when you select metrics, you must consider the result you want to attain, plus the potential unintended consequences.
Types of Metrics
There are many ways to divide metrics. For this article, we will use two main groups reflecting the timeframe for use. Example of each group of metrics are shown below. Within each of these groups, there may be qualitative and quantitative measures. But you also need to look at how the measurement can be used to drive changes.
The first group of metrics are long term, strategic, or reactive. These metrics are designed to be used over an extended time. As such, the actual metric needs to stay unchanged so that the data can be compared from period to period or over time. Because of the timeframe needed to collect information, corrections to the process being measured is reactive.
The second group of metrics are short term, actionable, or predictive. These have a short life and are used to address a particular project or specific problem. The metric generally is used for a short time, until a particular issue has been resolved or it may be used to provide predictive information to allow a project to respond early enough to prevent problems.
When selecting metrics, start with a few to see if they add value. Too many metrics may drive conflicting behavior and place significant overhead on the organization to collect the information relative to the value obtained. You may also find that some metrics are better collected through a proxy metric (e.g., customer satisfaction through complaint data).
Below are some examples of product development metrics.
Long term, strategic, or reactive
Short term, actionable, or predictive
Selecting the right metrics is important for successful product development because they can help you focus on driving improvement into your PD process. Keep the number of metrics manageable and try different ones when the old ones no longer are effective.
—————————————————————————————
[iii] Hauser, John R. and Gerry Katz (1998), “Metrics: You Are What You Measure!.” European Management Journal, 16, 5, (October), 516-528.
[iv] Anderson, A. (2015) NPD Metrics as a Continuous Improvement Tool [PowerPoint slides]. Retrieved from https://www.pdma.org/p/do/sd/topic=724&sid=842
[v] Chan, A. (2005). Using an Effective Metrics Program to Support Business Objectives. In K. B. Kahn (ed.), The PDMA Handbook of New Product Development, Second Edition, (pp. 445-454). Hoboken, New Jersey, John Wiley & Sons
[vi] Reinertsen, D. G. (2009). The Principles of Product Development Flow: Second Generation Lean Product Development. Redondo Beach, California, Celeritas Publishing
