We attended a Lean Launch Pad seminar where an attendee indicated that they didn’t understand where hypothesis testing fit into the lean startup process. I believe one of the best explanations I have read on the topic is from Garvin in his classic book, Learning in Action.[i] He discussed two particular types of experiments: exploratory and hypothesis-testing.
First let’s start with a definition. An experiment is a test in which particular actions are taken and results are observed. Often experiments are done to learn about the range of possible results, or to prove a hypothesized result.
Garvin described exploratory experiments as being designed for discovery, usually through an open-ended search. This is sometimes referred to as the “probe-and-learn” process.
“Exploration is key to understanding market and technology uncertainty”
On the other hand, experimentation for hypothesis-testing has a different set of goals: proof is the desired end, not discovery. The best hypotheses are clear and unambiguous, they describe a relationship or connection among variables and are capable of being disproved.
When should you use exploratory versus hypothesis-testing experimentation? It really depends on what you’re trying to achieve.
For example, if you have a product you’d like to offer in a new market, you could learn about potential obstacles and value propositions through exploratory experimentation. You could offer the product in different configurations to a limited group. By carefully observing reactions to the offers and usage among actual purchasers, you will learn about this new market and whether the product may be a good fit.
Hypothesis-testing works better when you want a more refined understanding of causal relationships. For example, you believe you can increase sales by offering a new flavor of tea. In your experiment, you offer the new flavor in addition to existing flavors in several test stores. You identify similar stores as a control whether the new flavor is not offered. You compare same store sales and determine whether the new flavor actually increased sales.
Based on this brief explanation on the two types of experimentation, I believe both of these techniques can be used with the Lean Launch Pad process. You want to use exploratory experimentation when you don’t know much about your product (for e.g. technology/commercial viability) and you need to learn more about possible outcomes. If you have causality between variables, it is best to use hypothesis-testing especially if you’re trying to prove a particular relationship or pattern to the causality.
And of course, a combination of techniques are also highly desirable. Starting out with exploratory experimentation when your product is ill-defined, and then transitioning to hypothesis- testing when your product is more defined. If you want to learn more about these two types of experiments, I highly recommend that you read Garvin’s chapter on Experiments.
To learn more about our integration of experimentation in product development check out our new book Learn & Adapt: ExPD an Adaptive Product Development Process for Rapid Innovation and Risk Reduction.
“The true method of knowledge is experiment.” — William Blake
[i] David Garvin, Learning in Action: A Guide To Putting the Learning Organization To Work: Harvard Business School Press, 2000. Pages 139-183.