This is my fifth post in a six-part series focusing on critical issues in healthcare Quality Improvement (QI) analytics. This week, I want to focus on metrics. Specifically, the challenges we all face when using metrics that were initially intended for different uses from the current application.

Let’s get into it.

The Issue: Metrics Intended for Other Uses

Performance metrics that track quality over time are some of the most important elements of the QI world. There is no shortage of measures in use across healthcare settings, different payers, or different populations. If you throw in professional disagreements about relevant thresholds for some measures (for example, should it be INR < 3, < 4, or <5?), the number of QI performance metrics becomes staggeringly large.

So which performance measures are the right or best to use?

Frequently, QI project directors are looking for metrics that are already available and align with the processes and outcomes of interest in the project. However, you may find that existing metrics are not always well-suited to the project at hand.

Typical mismatches between performance measure intentions and QI projects include the following:

• Different settings of care: there are many patient safety measures specified for hospitals; however, many of these would not be useful in a primary care setting.
• Different populations: the inclusion/exclusion criteria for an existing measure may not align well with your QI project’s inclusion/exclusion criteria.
• Different time frames for rate calculations: Existing performance metrics, often calculated on an annual basis, may not be effective for measuring improvements made during a one- to two-month process improvement project.

When you import metrics intended for other uses into a QI project with mismatched specifications, the resulting data will suffer from concerns about validity (i.e., accuracy) and reliability (i.e., consistency). In other words, are you accurately capturing the information you want in a consistent manner from one patient to the next?

Why Do QI Staff Still Use Mismatched Metrics?

This is a more challenging question to answer, and I’m not sure there is systematic research on the subject. If you are aware of any research on the issue, please comment to let me know.

Anecdotally and through my own experience, there seem to be two main reasons:

1. A lack of experience: those developing QI programs and selecting metrics may be excellent clinicians but have far less experience evaluating measures for QI research. As a result, metrics that might seem superficially adequate may not be when you examine the details more closely.

2. A desire to use validated measures: QI program developers try to follow a fundamental research design tenet by using measures already validated for accuracy and reliability. Even the most basic research design courses teach this concept. Developers are tempted to justify selecting an existing measure, especially if it has been previously validated. However, a measure validated in one setting and one population may not be valid in another.

The QI community has thoroughly researched many of CMS’s performance measures across its various programs. Measure developers create specs with a high degree of detail in identifying qualifying events, algorithms, and inclusion and exclusion criteria. That level of detail, however, does not automatically make a measure suitable for use in all settings, all populations, and all time frames.

It is important for QI project directors to carefully select the metrics they will be using to align with the purpose and targets of the project.

How to Select Appropriate Metrics

During the planning and design phase of your QI project, you will select the metrics used for tracking whether the project was successful or not. This is the time when you must carefully evaluate potential measures for alignment with your program implementation.

Your evaluation of the suitability of a metric should cover the following aspects of the measure, at a minimum:

• Clinical Intent: At its core, this is the most important issue to align on. What is the clinical intent of the performance measure you are considering? Does it align with your intent for the QI project?
• Target population: is the measure intended for use with your population? If not, consider whether your target population experiences the same conditions and comorbidities as the measure’s intended population. Also, consider whether the clinical processes and procedures referenced in the measure specs are aligned with those your target population might receive. If not, the measure is likely not well suited for your needs.
• Setting of Care: Is the measure intended for use in the same setting of care as your QI program? If not, verify whether the data intended to calculate the measure are available in your setting.
• Measurement Time Frame: If your QI project will be fully implemented within a few weeks, does it make sense to use a performance metric that requires years of data to calculate? Additionally, the faster your measurement frequency becomes (e.g., weekly or monthly), the less likely an annual measure will work well of duplicative counting of patients or events across smaller time spans. In general, shorter time frames for project implementation require a scrappier measurement strategy.

These are four important items to consider in any measure selection scenario. Depending on the circumstances of your QI project, there may be other important factors to consider.

Conclusion

Selecting good performance measures for your QI project is an activity to be taken seriously. Many well-intentioned QI initiatives have been derailed by poor measurement strategies.

If you are considering using existing performance metrics for your analysis, be sure to consider how well the candidate metrics align with your project intent.

In some cases, a performance measure can easily be adapted to a closely aligned project. In other cases, the measure’s intended use and specifications may be too far removed to be helpful in your project.

Keep in mind that most existing clinical performance measures have been developed to have a high degree of validity when basing their calculation on administrative claims data. If your QI project is capturing data in closer proximity to the patient (e.g., at the point of care), existing performance measures may not be of much use. A more direct and highly aligned measure of project processes and outcomes may be better suited to your goals.

Other Articles in This Series

 If you found this brief article helpful, see the other articles in the series here:

• Small Sample Sizes
• Lack of Comparison Groups
• Sufficient Time Frames
• Poor Data Collection Strategies
• Reliance on Metrics Intended for Other Uses
• Misuse and Misinterpretation of Statistical Tools and Results