Downside of Randomized Control Trials culture

Randomized control trial (RCT) has become a corporate trend over the past 10 years. The idea is, to drive growth, companies just need to run a bunch of RCTs to find out what works and roll those out.

Sounds great, but having lived that, I’ve seen that a big downside to the ‘RCT culture’ is that it counterintuitively causes stagnation.

RCT’s tend to turn management into ‘statistical significance’ snobs.

They couldn’t explain to you what statistical significance means, but they have been trained, often by the company’s analytics groups, that it sounds smart to only approve things that have statistical significance.

There’s a couple of problems with this.

One is that to get a statistically significance read often requires large sample sizes. So, RCTs become big effort that takes considerable organizational resources. That means, only a handful can be conducted at any one time. In a world where only 1-in-10 things pan out, this means that you might find one good thing every two cycles. At organizations I’ve worked with, a cycle can be a year long. So, the cadence is finding one good thing about every 2-3 years. That seems to hold.

It’s also worth mentioning, that ‘success’ doesn’t necessarily mean growth. Most times, success just means it is slightly better than what you have today. So, finding one thing that moves the needle slightly every 2-3 years isn’t a recipe for high single digit or double digit growth.

Two, any trial too small to have a statistical read is sneered at because it doesn’t have ‘statistical significance.’ So, small pilots, experiments and natural experiments are not conducted at near the pace to keep the fires of innovation going for growth or ignored entirely, even when results are big.

Sometimes, those small, but big results, are discounted even when it has statistical significance due to the second bit of statistical snobbery, small sample size.

This might be a good to link to my old saying, If I need a statistician to tell me if something worked, it didn’t work.

Because of this, management of mature organizations have a tendency to chase big efforts that have small rewards and ignore small efforts that could have big rewards. This leads more to them having to ‘spin’ their results (i.e. putting lipstick on the pig) to try to convince their bosses of success, rather than letting the results speak for themselves.


Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s