Data analysis is a funny thing. When the analytic results are clear, you gain the supporting evidence to make solid data-driven decisions. When the analytics results aren’t clear, however, you get…well, frustrated. A quick Google search will return plenty of articles explaining reasons why your analysis failed. The reasons will include everything from using the wrong data, to a lack of political interest by executives. And while there are many strategies available to improve the quality of your analytic projects, one challenge always remains. This article will focus on the real reason your analytic results aren’t always clear: probability.
Please Don’t Run
Now, if your eyes just glazed over from a flashback to your college probability and statistics course, shake it off. I promise not to dive into all the heavy math. But my main points are ones that consumers of analytic results frequently forget. Hang in with me and you’ll finish this article stronger than when you started.
Probability: The Driver of Analytics
Descriptive statistics, like averages, medians, and standard deviations, help you make sense of the characteristics of numeric data. You learn how to describe the typical observations, and the differences between observations. Once you move beyond description, however, you enter the world of inferential statistics. And this is the world where your analytics results can become less clear.
At the heart of all inferential statistical analyses is the concept of probability. In plain language, probability is a description of the certainty with which a specific outcome occurs. If an outcome has a probability of zero, then it never happens – like seeing a baseball transform into a unicorn. In contrast, if the outcome has a probability of one, then it always happens, or is certain – like water freezing below 32 degrees Fahrenheit.
In plain language, probability is a description of the certainty with which a specific outcome occurs.
Mathematicians developed probability and inferential statistics to understand the reality of a world where very few outcomes are certain. Typically, we find that sometimes one outcome occurs and other times different outcomes occur. Most probabilities, therefore, fall somewhere between 0 and 1. Importantly, if the probability is not 0 or 1, then the outcome (or lack of outcome) is not certain.
Outcome A might happen with a probability of 0.30, or 30 percent of the time. In comparison, outcome B might happen with a probability of 0.70, or 70 percent of the time. For any given event, we won’t know which outcome will happen, only that outcome B is more likely to happen.
And to be clear about this, there are often more than two possible outcomes for any given event. For example, when we predict customer purchases, each product or service offered is an outcome with a probability of purchase.
The Traditional Statistics Approach
If you have taken a traditional introductory course in statistics (also called frequentist statistics), you likely learned about hypothesis testing. The concept of hypothesis testing boils down to creating two competing views of how things happen, and then using your data to determine which view it’s most compatible with. For example, we might want to know if younger customers are more or less likely to purchase energy drinks than older customers. We could frame our hypothesis test with the following two statements:
H0: Customers in the 18 – 24-year-old age range purchase energy drinks at the same rate as customers age 25 and up.
HA: Customers in the 18 – 24-year-old age range purchase energy drinks at a different rate than customers age 25 and up.
In this example, H0 is the null hypothesis – or the idea that there is no difference between the age groups in how often they purchase energy drinks. In comparison, HA is the alternate hypothesis – or the idea that there is a difference between the two groups.
We use this approach in hypothesis testing to create a decision point where only one answer can be logically true. Our entire question is boiled down to an either/or decision. Traditional statistics, however, simplify the question even further, focusing on whether your data are compatible with the null hypothesis (H0). Essentially, the traditional statistical test is a decision about whether the null hypothesis is probable, or improbable. Logic dictates that if the data are not compatible with H0, then then they are likely compatible with HA. We reduce the entire process down to a thumbs-up/thumbs-down decision.
The Confusion Created by the Traditional Statistics Approach
The traditional statistics approach trades one probability for another, which causes confusion for readers about how to interpret the results. The logic of hypothesis testing focuses on the probability that H0 is true, based on the compatibility of our data.
But that is not where we started our discussion on probability. We started off by asking how certain a specific outcome was to occur. The confusion, therefore, is that many people interpret the thumbs-up/thumbs-down decision in hypothesis testing as a sign of the certainty of a specific outcome.
The logic that many readers use goes something like this:
The hypothesis test was statistically significant; therefore the null hypothesis is not true. We can then conclude that younger customers will purchase energy drinks more often than older customers.
When readers view statistical results in this way, they are ignoring the fundamental nature of probability. Instead, they are attempting to make conclusions stronger than they actually are.
How P-Values Really Work
The truth is, we rarely ever really “know” what the truth is. We are only searching for a signal of the truth in the noisiness of our data.
Remember the hypothesis test is focused on the probability the data are compatible with H0. Since probabilities range from 0 to 1, we need to determine the threshold where we will stop believing the data are compatible with H0. A standard threshold in applied statistical analysis is a probability less than 0.05 (written as p < 0.05). You can read more about why the weird reason is the standard here. So, if the chance the data are compatible with H0 is less than 5 percent, we declare the data incompatible with the null, and the alternative hypothesis becomes our preferred view of reality.
But notice we are still talking about probabilities. In the world of traditional inferential statistics, if the probability of data compatibility with H0 is 5.1 percent, we accept that as being the truth. At the very least, we say that there isn’t enough evidence to reject the null hypothesis as likely.
There is nothing mathematically magical about using a 5 percent threshold. Depending on how much data you have, setting a p-value threshold at 0.05 could handicap your analysis. In small enough samples of data, you might not be able to mathematically reach this threshold.
Some analysts argue that if you only need rough results, setting a higher p-value threshold makes sense. Consequently, you might set your threshold for a significant result at 0.10 or 0.20. With these thresholds, you accept the null hypothesis when there is a 10 or 20 percent chance of data compatibility.
Ultimately, your key question about the p-value is: At what point is the uncertainty low enough that I believe I know the answer?
Data Analysis is About Describing Uncertainty
At its core, we use data analytics to solve problems through the process of removing uncertainty. We begin the process by posing a question. If we’re lucky, we may have some sense about the answer to that question. Business experience and knowledge of the industry, customers, and environment we work in can help guide us toward an answer. Very quickly, however, honest people realize they have no certainty that their intuition is correct.
The only certainties in life are death and taxes.
Did our proposed product for development resonate with customers?
Does our new marketing campaign drive increased sales in the target demographic?
Will reducing the cost and quality of materials reduce customer satisfaction?
Our business experience drives our gut, but can’t tell us if we are right or wrong. We construct analyses to reduce uncertainty until we feel comfortable saying we know the “right” answer.
But the answers we get from data analysis are never the “right” answer. At best, we learn what the most probable answer is.
What To Do About Uncertainty?
At this point, you might be a little concerned that your analytic results aren’t as certain as you once thought. I understand your concern, my friend. After all, if we can’t remove uncertainty and determine the “right” answer, then doesn’t that pose a risk?
Yes.
Risk is an inherent part of business. You will always face risks that come in many forms like changes in:
- Consumer tastes
- Technology
- Competition
- Government Regulation
- Accidents
- And More…
Your analytic strategy can’t remove those risks. From our discussion of probability, you also know that analytics won’t show you the perfect path to certain success. Your analytic strategy will, however, help show you a path that is more likely to be successful.
The unfortunate part is that sometimes your analysis will not yield clear results. For example, I have seen evaluations where the analysis gave very promising, substantive results, that were not statistically significant. In other evaluations, I’ve seen conflicting results across different analyses, making it unclear whether the program or policy “worked”.
When this happens, you need to dig deeper into the analysis and results. Was your sample size large enough to be able to identify a significant result? If you had multiple outcomes, is it possible that your program or policy was only partially effective? Is it possible that other factors not accounted for in the analysis were masking a program effect?
While your analysis may not give you certainty in your results, it’s still better than relying only on your gut. Additionally, there is an entire analytic school of thought that attempts to make the uncertainty in an analysis clearer. Bayesian statistical analysis shines a spotlight on uncertainty, attempting to quantify how much uncertainty there is in any result.
Conclusion
You may never know with certainty what the “right” answer is from an analysis. C’est la vie. But knowing what you know now, you’re better equipped to interpret your analytic results for what they really mean. Additionally, you’re better equipped to dig deeper when your analysis presents results that are less than clear.
Like any other scientific pursuit, replication is the key to establishing truth. If you find the same analytic results over and over, you’ll be more confident that you know what is happening. Until then, accept the uncertainty of life, but minimize it with your analytic strategy.
