The world may be too complicated for RCT’s, there are better feedbacks

Randomized controlled trials (RCT’s) are hot.

The general public knows common forms of these these as pilots (as in a pilot episode for TV series), test marketing, experimentation or trials.

Those in public policy circles leverage RCT’s to identify whether a policy or program works. A recent Freakonomics podcast, When Helping Hurts, explores the issue and how RCTs have been used to determine whether programs, like mentor programs for disadvantaged youth, help or not.

The podcast explored long-term research at such a mentor program that, to the chagrin of the researchers, didn’t appear to help and may have even hurt. Ouch.

The moral of the story was that things that sound good may not be, but that may be difficult to figure that out.

RCTs can help.

But, even RCTs are limited. A common limitation is the definition of success is often too limited.

Charter schools are often measured on whether they improve student test scores. That assumes a lot. For example, it assumes that test scores matter. It also assumes that schools can have an influence on test scores.

But, what matters most is always the individual students and the parents. Those are things schools won’t have much impact on.

A success measure that is normally overlooked when discussion school choice is whether parents were happy with their choice. If they were, it was a success, regardless of what happened to test scores.

This would also have been difficult to pull out of an RCT, because happiness is subjective.

We don’t need RCTs to figure out if we like McDonald’s better than Starbucks. What we have there is pure, old fashioned accountability from customer to corporate HQ. It’s not perfect, but it’s better than RCTs.

We also don’t RCTs to figure out if being polite is useful. We have accountability to help us figure that out. When you aren’t polite, you get the stink eye.

I think researchers get excited about RCTs because it gives them something to do and they don’t yet realize how limited their tool really is.






Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s