Your Rapid-Cycle Sample Sizes May Be Too Small for Healthcare QI

Hi there, Friend! I’m trying out a new format with shorter articles, and more pointed, actionable information. Let me know how you like it. This post is the first in a six-part series on data analytic issues I continue to see in healthcare Quality Improvement (QI). The six parts will include:

Let’s get to it!

The Issue: Rapid-Cycle Sample Sizes

Rapid-cycle quality improvements are performed using an iterative process. Often described using the Plan-Do-Study-Act (PDSA) framework, the process iterates through four steps to test and evaluate a change.[1]

Rapid-cycle frameworks have been used for several years in healthcare quality improvement, particularly in hospitals, nursing homes, and other settings.

Yet, I continue to see many QI projects with sample sizes too small to be useful.

An Example of the Rapid-Cycle Sample Size Issue

I was once participating in a large hospital-based quality improvement initiative. A cornerstone of the effort was to allow local facilities to adapt and tweak quality improvements at their local facilities.

During a planning meeting, a director on the initiative said that a sample size of five (5) patients was sufficient to determine if a process improvement was working.

Take a step back to think about that statement. With only 5 cases in the study, you can observe something happening (or not happening) to a patient in 0, 20, 40, 60, 80, or 100 percent of the cases.

Your QI effort would need to make a 20 percent change in the outcome to see it in your data.

That’s a big change for rapid-cycle improvement without making major policy changes (e.g., hard stop policies for early elective deliveries).

Of course, if you go from 20 percent to 80 percent, you won’t need many observations to see success.

But if you want to move from 70 to 80 percent, you’ll need many more cases to see success.

Importantly, if your sample is too small your analysis might suggest the intervention didn’t work, when in reality you simply didn’t have enough data to draw conclusions with much certainty.

How to Address the Small Sample Issue

To ensure you have a large enough sample size for your rapid-cycle improvement project, have your analysts calculate the minimum required sample size for the analysis you will be performing.

You will need to know roughly what your performance rate is currently, as well as the target rate you want to achieve. You will also need to set a statistical level of certainty, which is often set at 0.05 – meaning there is a five percent or lower chance of finding a significant result when there really isn’t one.

With this information, analysts can perform sample size calculations to determine the minimum sample needed for your project. If you collect data from at least this many cases, your results should be reliable and useful for making decisions.

Take the time to perform these calculations, so you don’t spend the time and resources to produce bad information from a study with too small of a sample.

Conclusion

The punch line is this. Collect as much data as you can give your resource constraints. But take the time to calculate the minimum sample size required for the analysis you are performing.

Don’t proceed with the study unless you can collect a sample at least as large as the minimum requirement.

[1] For more information on the PDSA framework, see the Agency for Healthcare Research and Quality Health Literacy Universal Precautions Toolkit, 3^rd Edition. Accessible at https://www.ahrq.gov/health-literacy/improve/precautions/tool2b.html