A Smarter Way to Run Business Experiments
Too many companies try out new ideas without knowing how to properly test them, a Harvard professor explains
--
In 2016, Jeff Bezos gave shareholders a rare insight into Amazon’s innovation engine. In his annual letter, he explained: “I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.”
So it should not have come as a surprise when Amazon acquired Whole Foods about 14 months after Bezos wrote the letter. Amazon’s reputation for fearless innovation was fueled by such radical business experiments — so-called big swings — and, equally important, the tens of thousands of smaller and disciplined experiments that have led to a highly optimized user experience in its web store.
If the case for business experimentation is so compelling, then why don’t more companies conduct rigorous tests of their risky overhauls and expensive innovation proposals in order to make better decisions?
Many organizations are reluctant to fund good business experiments and have considerable difficulty executing them. Most tests of new business initiatives are too informal. They are not based on proven scientific and statistical methods, and so executives end up misinterpreting statistical noise as causation — and make bad decisions.
To ensure that each experiment yields better decisions — companies should ask themselves seven important questions.
Question 1: Does the Experiment Have a Testable Hypothesis?
In 2013, the large retailer Kohl’s was looking for ways to decrease its operating costs. One suggestion was to open stores an hour later on Monday through Saturday. Company executives were split on the matter. Some argued that reducing the stores’ hours would result in a significant drop in sales; others claimed that the impact on sales would be minimal. The only way to settle the debate with any…
