The question asked most commonly by non-analysts when looking at analytic results is, “Are the results significant?” If the answer is yes, many people assume these are important results, and the analytic punchline should be taken seriously. Fewer people understand what a statistically significant result actually means. In this article, I’ll pull the curtain back and make hypothesis testing (and statistical significance) easy to understand.

What is Hypothesis Testing?

To get right down to brass tacks, hypothesis testing is the process used to answer many analytic questions.

Dig back in your memory and you probably learned the scientific method in school.

Recall that when you use the scientific method, you make a clear statement about what you think is happening in the world.

This is your hypothesis.

Ideally, you want to make your hypothesis clear and concrete enough that you boil your question down to one of two options: the null hypothesis and the alternative hypothesis.

The null hypothesis is simply the idea that there is nothing going on. No difference between groups, no relationships between variables, no impact of a program, etc.

The hypothesis you believe is true is called the alternative hypothesis. It should be a clear statement about the relationship, difference, or impact you anticipate in the real world.

In every hypothesis test, you will be testing the null hypothesis.

The logic goes, if the null hypothesis is unlikely to be true, then your alternative hypothesis is probably a better description of reality.

Hypothesis Example #1:

• Alternative: Women are more likely than men to make purchasing decisions for the family. Versus….
• Null: Women are equally as likely as men to make purchasing decisions for the family.

Hypothesis Example #2:

• Alternative: There is a positive correlation between alcohol use and tobacco use (i.e., greater alcohol use is associated with greater tobacco use). Versus….
• Null: There is no correlation between alcohol use and tobacco use.

Once you have your two options, hypothesis testing is the process of using data to decide which option is most likely true.

Hypothesis Testing and Statistical Significance

When you perform a hypothesis test, the analysis searches for a signal that your data agree with the null hypothesis.

As part of the hypothesis test, you must determine how you will decide whether the null hypothesis is likely to be true.

You make this decision by setting an alpha level. The alpha level is a probability threshold below which you are comfortable saying that the null hypothesis is unlikely to be true.

Obviously, the alpha level must be set to a low value. A common choice of alpha is 0.05, or 5 percent. Of course, this is not an iron-clad value, and other values might make more sense.

When your analysis calculates the test statistic, your software will provide a p-value or probability value.

The p-value is what you will compare to your alpha level to decide whether the null hypothesis is likely to be true.

If the p-value in the resulting analysis is less than the alpha level you defined earlier, then the null hypothesis is unlikely to be true. At this point, analysts would call the test statistically significant.

Statistical Significance Is Not the End

Importantly, whether you identify a significant result or not, your work is only half done.

When your hypothesis test is completed, you need to be able to translate the findings into a plain language description of what is going on in your data.

Additionally, you need to consider the implications of the findings for your larger research question(s).

After all, you’re performing this analysis to answer a question.

If your hypothesis test was not statistically significant (i.e., p-value > alpha level), then your data agree best with the null hypothesis.

In contrast, if your hypothesis test was statistically significant (i.e., p-value <= alpha level), then your data agree best with your alternative hypothesis.

Now that you know which hypothesis your data supports best, what does this mean to your project?

Answer this question clearly, and you’ll have a solid data story to tell.

Conclusion

With a clear and simple understanding of hypothesis testing, you are better prepared to review analytic results. For any analysis, if you understand what the null and alternative hypotheses are, you’ll be able to translate the results back into your original research question. Keep in mind that the most important result is not determining whether the result is significant or not, but what the results mean in practical terms.

Caveat: I am aware that the American Statistical Association (ASA) released a position paper in 2016 about significance testing and the use of p-values. The paper appropriately cautions against the over-reliance on hypothesis testing and statistical significance as the sole evidence of meaningful results. In this article, I am attempting a far simpler goal in the form of conveying the fundamentals of hypothesis testing.

Interested readers can find more information on the ASA position paper << Link Here >>.