If I need a statistician to tell me whether something worked or not, it didn’t work.
That’s a sentence that I think I came up with, but I wouldn’t doubt that I lifted it off someone and can’t remember who. Until proven otherwise, I’ll take credit.
This sentence came from my years working with statisticians employed by managers of for-profit companies to find out if something they tried worked or not.
My sentence ruffles the feathers of statisticians. Many statisticians understand the limitations of their trade, but many don’t. Many non-statisticians don’t either. When managers use statisticians to determine the effectiveness of a project, beware.
To me, the project needs to offer what I call a clear advantage. I should be able to look at the results and clearly see that what I tried was better.
That’s how we all make judgments every day about what we buy and what we sell. We don’t need statisticians to tell us that the coffee shop downstairs is good enough for rainy days, but the walk to Starbucks a block away is worth it on nice days.
When evaluating business success I usually look at a few things to make my judgment.
First, I see what happened to revenue. I consider this a true measure. It’s hard to fake. This is a true reflection of what customers thought . If ask customers what they thought, their answers may not be that helpful, so it’s always good to look at their actual behavior, not just their stated behavior. Revenue is the truest measure. Did they buy it or not?
A statistician might tell you that a 0.4% difference in revenue, depending on the sample size, is a “statistically significant” result, meaning that there’s a good chance that the change I made was the cause of the difference.
I don’t care. I don’t want more than a 0.4% difference in revenue because I do not trust the chances well enough to believe that my experiment caused it. I want something bigger. Give me 5% growth. Maybe 3%. Enough to get me over the hump.
Second, I do ask the customers what they think to see if what they say lines up with what happened with revenue. This can give me important clues as to why it worked or didn’t. Often, things that do work, don’t work for the reasons you thought they would.
Third, I determine if the cost to make the change is worth it to the owners.
If you take a look at just about any business today, their success was not determined by statisticians. Their success was determined by the customers. Customers either buy enough of what they’re selling to to make it worth the owners while or they don’t.