The randomized control trial (RCT) has become a standard fixture in corporations over the past 10 years. The idea is, to drive growth, companies just need to run RCTs to find out what works and roll those out.
Sounds good, but having lived it, I’ve seen that a downside to the ‘RCT culture’ is that it can kill innovation.
Experiments are like lottery tickets, each has a low chance of winning. So, the more experiments an organization runs the better their chances at finding winners. Anything that limits the number of experiments also limits innovation.
RCT culture limits the number of experiments because managers believe they can only make decisions from statistically significant results.
Obtaining that requires large enough efforts that organization are limited in how many they can try at once.
Another problem is that the quest for statistical significance kills other types of learning. Managers put no stock into the findings of pilots, qualitative research, natural experiments or just plain old common sense, because it’s not statistically significant.
This mindset was captured on a podcast I listened to recently, (paraphrasing) “the fire department won’t use water to put out a fire because there is not an RCT that proves that it works.”
These are all learnings that can dramatically increase the number of effective experiements the company is running, but managers shut those out because their statisticans have led them astray with statistical significance.
This might be a good to link to my old saying, If I need a statistician to tell me if something worked, it didn’t work.
While statisticians are very good at math, before letting them drive too much of the business decisions, consider carefully how many successful products they have launched and whether their actions have meaningfully and tangibly improved the company’s performance.