Subtitle: The Dangers of Reporting Numbers Without Proper Benchmarks
A few years ago shark attacks seemed to be on the rise given all the media attention they were attracting.
It caused people to ask, Have sharks suddenly developed a taste for humans? Has global warming or agricultural runoff diminished their food supply, so they are more willing to try new things?
But, at the end of the year, we learned that number of shark attacks that year was average or below average.
That doesn’t diminish from the shark attacks themselves. I don’t want to be attacked by a shark. It’s terrible when someone is attacked by one.
But, even in the years where shark attacks were big news, I still got in the water. Why? Because I did my research and felt comfortable that while getting attacked by a shark was possible, it wasn’t any more probable than it had been before. The only probability that had changed is me making the national headlines if I did get attacked.
This caused me to be skeptical when I see the reporting of raw numbers without proper benchmarks. It tells me the author is either ignorant of the importance of proper benchmarks or is intentionally leaving those benchmarks out for a ‘good story.’
If ignorant, they should not be reporting on things with numbers involved and I’m not interested in reading their work.
If intentional, then they are likely trying ‘not to let the facts get in the way of a good story.’
Unfortunately, with COVID-19, a lot of reporting has been done without proper benchmarks.
Here’s a good example of an article that lacks proper benchmarks. Here’s a paragraph from the article:
On Thursday, EMS received more than 6,000 calls. EMS lieutenant Vincent Variale, who also heads a supervisors’ union, said, “We’ve broken every call volume record we’ve ever seen before.”
6,000 is a raw number. That’s the number the author wants me to remember.
But, what’s the benchmark? A benchmark tells me how to think of this number.
The author hints at a benchmark with the quote that it breaks every call volume record.
But, as a reader I’m left to guess by how much. When I’m left to guess, I get skeptical. Too many times before when I’ve been left to guess, like with shark attacks, I’ve learned that the benchmarked numbers were less than alarming.
If a typical Thursday gets 1,000 calls and the previous record was 3,500 calls, then 6,000 might be alarming.
But, if the typical Thursday is 4,800 calls and the previous record was 5,900, then we are talking about an outlier, for sure, but still not too far out of the realm of possibility.
Does the paragraph that follows the one quoted above provide a clue to what might be happening?
Fire Commissioner Daniel Nigro on Friday asked the public to avoid making 911 calls except in real emergencies.
It’s hard to tell.
The way the author includes it, it could just be a public service reminder, or it could be that a good percentage of the 6,000 calls were non-emergency calls driven by the attention C-19 has been getting. For example, calls from people with a mild cough asking where they can get tested and how to get there without possibly infecting 10 others on the way.
If it turns out that the number of non-emergency calls was 2,000 for Thursday compared to 200 on a typical day, then the real story is the number of non-emergency calls, not necessarily the record number of calls.