Welcome back to the second in a six-part series of short articles with more pointed, actionable information. In this article, I’m raising a persistent issue I’ve seen across many interventions: attempting to evaluate healthcare Quality Improvement (QI) projects without a comparison group.
The Issue: Lack of Comparison Group
Every QI project seeks to either make healthcare better for the patient, save money, or preferably do both.
To determine if a specific project improved care, it makes sense that you want to compare the outcomes for those receiving the intervention to outcomes for patients not receiving the intervention.
By making this comparison, you get a sense of how much of an impact your QI intervention had.
However, a problem arises when your QI project is implemented on the entire target population all at once.
In this situation, there is no clear comparison group that can be used to determine the impact of the intervention.
An Example of QI Interventions Without Comparison Groups
I often see this problem occur in large-scale QI programs that impact state-level or higher policy.
To be fair, many projects are planned to include comparison groups. However, the comparison groups defined in the plan frequently are not practical after implementation
Program directors sometimes plan to use national-level patient data to construct synthetic control groups – pulling a combination of patients from other locations that look like the target population.
Unfortunately, this is extremely difficult to do in practice because the necessary data fields may not be available in a national data set.
Additionally, patients from different states experience different policy contexts for receiving healthcare, making them systematically different from the target population.
Program directors also often find that accessing national data sets can be cost-prohibitive and may not align well with the targeted population.
So what starts out as a well-intentioned plan can quickly turn into an evaluation nightmare when the program team finds the intended comparison group isn’t viable.
Ensuring Your QI Intervention Has a Comparison Group
To prevent this from happening to your QI intervention, it is imperative that you confirm that your intended comparison group is viable before implementing the project.
Even if the target population is an entire state, there are options for creating comparisons that allow you to evaluate the program
Pre/Post Assessment
Compare outcomes before and after program implementation. Be careful that other external factors are not influencing the results over time (e.g., other QI efforts, changes in healthcare technology, etc.).
This can be extended to include multiple pre- and post-periods.
Staged Roll-out
Implement the program across the target population in phases. Compare the results for segments of the population receiving the program at different times.
Stages can be defined in a variety of ways (e.g., geographically, by risk or need, etc.)
Comparing Results to Next-Closest Cases
If your target population is defined by a threshold value of a continuous variable (e.g., patients over 50 versus those under 50), then compare the results for the target population to results for those just on the other side of the threshold.
You expect to see a difference in outcomes at the point of the eligibility threshold. This is a specialized scenario, but very useful when the conditions are right.
Conclusion
To be clear, these three options are not an exhaustive list of possible ways to evaluate a program that will be implemented on the entire population. However, they provide a few common ways of approaching such a challenge.
Importantly, it’s imperative that you consider your comparison group before implementing a QI intervention. You want to be certain that the proposed comparisons are feasible and will provide you with the information you need for decision-making.
If you are uncertain about the best comparison group for your next QI project, reach out to someone with research design experience for assistance. With over 25 years of experience designing and executing research projects, I’d be happy to discuss your challenge with you.
Other Articles in This Series
If you found this brief article useful, see the other articles in the series here:
