As a qualitative analysis guy, people assume I live and die by statistics. The problem is that quantitative analysis can mislead if you use the numbers incorrectly. It is essential that the person using statistics think carefully about the purpose of the measure and what they really represent. It is easy to draw a misleading conclusion.
A recent social media post notes that the 10 biggest drops in the stock market (Dow) occurred under the Trump administration. The information is both correct and incorrect. The list is based on point drops and that is true. However, the scale has changed over the years so the point drop is not a fair comparison.
If you look at percent drops, NONE of the top 10 occurred under Trump (this list may not include the latest Coronavirus drop). https://www.snopes.com/fact-check/largest-stock-drops-trump/
How do you defend against misleading statistics?
- Smell test: does it seem right? Does it seem TOO right?
- Question how the numbers were drawn and how they may have been drawn incorrectly.
- Consider the scale, consistency, and reliability of measures.
- Consider the motivations of the source (who is the source?).
- How are the numbers normally used? Is this a different use?
There is a lot of statistical analysis that use measures differently than the original intention. That is not necessarily bad. It is just important to apply statistics correctly especially when we depend on them to draw conclusions.