Patient-Reported-Outcome-Measures.pdf

Supplemental Material to the CMS MMS Hub Patient-Reported Outcome Measures

December 2023

Patient-Reported Outcome Measures

This document provides information about patient-reported outcome measures (PROMs). These

measures have special considerations outside the more common structure, process, and

outcome measures. This information supplements the information found in the Blueprint content on

the CMS MMS Hub, Measure Specification , Measure Testing , and Measure Use, Continuing

Evaluation & Maintenance .

1 PATIENT-REPORTED OUTCOME MEASURES (PROMS)

PROMs are tools used to capture patients’

reports of their outcomes, which measure developers can

use as the basis for patient-reported outcome-based performance measures (PRO-PMs). PRO-PMs are a

high priority for CMS and other organizations. PROMs present some design challenges. In this

document, we describe some of these challenges and approaches to address them.

Ensuring patients and families are engaged as partners in their care—a CMS goal—can also be an

effective way to measure the quality of patient care. Although patient reports of their health and

experience with care are not the only items that should undergo measurement, they are an important

component. Historically CMS used surveys to collect patient-reported data, but the continued

development of the infrastructure allows more timely collection and use of alternative collection

methods (e.g., using mobile devices) of these data. Currently, academic settings develop and test

PROMS for use in clinical care. Measure developers must do additional testing to use a PROM as a basis

The Blueprint content uses the terms persons and patients interchangeably.

1 Patient-Reported Outcome Measures (PROMs) ... 1

1.1 Patient-Reported Outcomes (PROs)............. 2

1.2 Patient-Reported Outcome

Measurement (PROM) ................................. 2

1.3 Patient-Reported Outcome-Based

Performance Measures (PRO-PMs) .............. 3

1.4 Approaches to Developing Patient-

Reported Outcome-based Performance

Measures ...................................................... 3

2 Patient-Reported, Outcome (PRO)-based

Performance Measure Evaluation ......................... 6

3 Key Points .............................................................. 7

References .................................................................. 8

Supplemental Material to the CMS MMS Hub Patient-Reported Outcome Measures

December 2023

for a quality measure. Figure 1 depicts the relationship between patient-reported outcomes (PROs),

PROMs, and PRO-PMs.

Figure 1. Relationship between PROs, PROMs, and PRO-PMs

1.1 PATIENT-REPORTED OUTCOMES (PROS)

CMS defines a PRO as any report of the status of a patient’s health condition or health behavior coming

directly from the patient, without interpretation of the patient’s response by a clinician or anyone else.

Self-reported patient data provide a rich data source for outcomes. This definition reflects the key

areas:

• health-related quality of life (including functional status)

• symptoms and symptom burden (e.g., pain, fatigue)

• health behaviors (e.g., smoking, diet, exercise)

1.2 PATIENT-REPORTED OUTCOME MEASURES (PROMS)

PROMs are tools used to collect patient-reported outcomes. Some examples of patient self-reported

data collection tools include

• Patient-Reported Outcomes Measurement Information System (PROMIS®)

—Funded by

the National Institutes of Health (NIH), these tools measure patient self-reported health

status.

• Medicare Health Outcomes Survey (HOS)—The HOS was the first outcome measure tool

used in Medicare Advantage plans. The goals of the Medicare HOS program are to gather

valid and reliable health status data in Medicare managed care for use in qualit

Supplemental Material to the CMS MMS Hub Patient-Reported Outcome Measures

improvement activities, plan accountability, public reporting, and health improvement. All

managed care plans with Medicare Advantage contracts must participate.

• FOTO Patient Outcomes —This tool measures the functional status of patients who

received outpatient rehabilitation through the use of self-reported health status

questionnaires. The FOTO tool assesses change in functional status by comparin

mea

surements taken at intake, during, and at discharge from rehabilitation.

However, the outcomes collected by the tools are insufficient individually for measuring performance

and accountability programs cannot use them directly. Measure developers should construct quality

measures that apply the outcome data collected by the tools to measure the quality of care.

1.3 PATIENT-REPORTED OUTCOME-BASED PERFORMANCE MEASURES (PRO-PMS)

A PRO-PM is a way to aggregate the information from patients into a reliable, valid measure of

performance at the measured entity level, e.g., clinician. The same measure evaluation criteria and

justification principles that apply to other outcome measures also apply to PRO-PMs.

Sev

eral PRO-PMs are available. Examples include

• Back Pain After Lumbar Discectomy/Laminotomy (CMIT Measure ID 85)

• Functional Status Assessment for Total Knee Replacement (CMIT Measure ID 279)

1.4 APPROACHES TO DEVELOPING PATIENT-REPORTED OUTCOME-BASED PERFORMANCE

MEASURES

Although PROs are a special type of outcome measure, the principles for development are the same.

The supplemental material, Risk Adjustment in Quality Measurement , details the procedure for risk-

adjusting outcome measures. Patient Reported Outcomes (PROs) in Performance Measurement

outlines a pathway for PROs to move from simple patient-reported data to measurement, to

performance measurement, and finally to endorsed measures in use for reporting and accountability.

1.4.1 Choose and Define a Patient-Reported Outcome

Throughout their health care journey, patients provide many kinds of data to their clinicians. Sometimes

a patient may share information with their clinician, which their clinician then interprets in the patient

record. PROMs collect information directly from patients, without clinician interpretation. To choose

patient-reported outcomes that will become quality measures, measure developers must first identify

quality issues for a target/initial population. An appropriate outcome has clinical or policy relevance.

For example, whether the patient did or did not develop a surgical site infection after cataract surgery

would not be a good PRO. A patient could report redness, swelling, and drainage, but not actually

whether they have an infection. A better outcome measure in this instance might be a clinically

meaningful measure of improvement in vision.

Outcome quality measures must also be meaningful to the target population and usable by measured

entities. Whenever possible, measure developers should consult clinical experts, including patients and

patient advocates, to help them define appropriate and meaningful outcomes.

1.4.2 Respecify or Create a De Novo PROM

As with other types of measures, measure developers can respecify existing measures or PROMs or

create a de novo PROM. There are some advantages to creating a de novo PROM, especially if its

December 2023

Supplemental Material to the CMS MMS Hub Patient-Reported Outcome Measures

December 2023

development is principally for the intended PRO-PM. Whereas when measure developers are selecting

from existing PROMs, they must find the best fit among PROMs created for other purposes. Additionally,

developing digital PROMs may require de novo development since some existing PROMs and their

owners may not allow for or their data elements are not conducive to mapping to interoperable data

standards. There are many challenges inherent in developing PROMs, including time, resource, and cost

constraints, and methodological and logistical challenges, which the measure developer must

acknowledge and address.

There are resources available to help measure developers respecify or create de novo PROMs. For

example, the PROMIS website provides existing tools and numerous resources for PROM developers,

such as standards for instrument development and validation. The journal Psychometrika routinely

includes information relevant to PROM developers across fields. Psychometrika published a relevant

special issue in September 2021, Advancing Methods to Assess Patient-Reported Outcomes: Lessons

Learned from the Patient-Reported Outcomes Measurement Information System (PROMIS) Initiative .

However, guidance tailored for health care quality measure developers to develop PROMs to support

the development of PRO-PMs in health initiatives is sparse.

Other peer-reviewed journals, such as the Journal of Patient-Reported Outcomes, provide articles

dedicated to these topics to help measure developers stay up to date with methods to assess PROs. CMS

funded a technical expert panel (TEP), Building a Roadmap From Patient-Reported Outcome Measures

to Patient-Reported Outcome Performance Measures (Building the Roadmap). The TEP produced

several reports, the most recent a technical guidance report . The purpose of the TEP was to identify

the attributes of high-quality PROMs and provide guidance on how to select PROMs to develop PRO-

PMs. Measure developers of PROMs should consider assembling a PROM TEP or consulting experts in

item set creation, psychometrics, statistics, and other specialty areas.

1.4.2.1

The Measure Lifecycle and PROMs

When developing a PROM, the Measure Lifecycle still applies. The usual starting point is information

gathering with an environmental scan and literature review to identify whether there are existing

tools to collect the outcome in the target population. Measure developers can use the Environmental

Scan Support Tool (ESST) and the De Novo Measure Scan (DNMS) to assist with environmental

scans. You must have a free CMIT account to access the DNMS. The PROMIS site has publications for

PROMIS measures back to 2004. See the Environmental Scan for Quality Measurement supplemental

material for more information on conducting environmental scans.

Measure developers may consider using tools with established psychometric properties (e.g., adequate

data element and tool reliability and validity). While the tools are not themselves necessarily PROMs,

with further testing in the health care environment, measure developers may use the information from

these tools to develop and test the construct of a PROM.

PROMs use the same basic building block for specifications, e.g., title, target/initial population,

description, numerator, denominator, exclusions. Questions to answer:

• How to best collect the data? One method or multiple methods? Using multiple methods adds

to testing complexity.

• What is the content of the PROM?

Supplemental Material to the CMS MMS Hub Patient-Reported Outcome Measures

• What is the phrasing of the items? For example, consider the PROM population and reading

level.

• What is the order and layout of the items? For example, should the measure developer includ

kip logic?

Measure developers should follow the scientific principles of questionnaire and survey development.

There is a lot o

f published literature on these principles. PROM development is iterative with testing.

Measure developers should test early and often. It is important to test these tools with the population

and setting on which the PROM focuses. If respecifying an existing tool, there may be differences

between the reliability and validity of a PROM in more controlled settings (e.g., clinical trials, academic

research projects) compared with use in real-world practice settings, but to date, testing of most PROMs

has only been in controlled settings. The test plan should assess the sources of variation that can affect

validity, reliability, and usability of the PROM. If creating a PROM for electronic collection, consider the

interoperability and test like other digital measures using multiple test sites and vendor products. When

assessing feasibility and interoperability, the measure developer should review the concepts in the

Logical Observation Identifiers Names and Codes (LOINC) terminology. LOINC has numerous health

care screening, evaluation, and survey instrument items and the acceptable answers. If there are no

existing pertinent concepts, The measure developer may want to request LOINC add new concepts .

1.4.2.2

Attributes of High-Quality PROMs

The Final Interim Report from the Building the Roadmap TEP identified 12 attributes of high-quality

PROMs for use in PRO-PMs, which the measure developer can use in conjunction with the Measure

Evaluation Criteria :

• covers desired PROs from the patient and/or caregiver perspective

• P

ROM measured outcome is a result of care for which relevant clinical quality is the target of

measurement

• interpretable scores, defined and actionable cut points or targets, and anchors and/or defined

meaningful change

• c

lear conceptual and measurement models

• psychometric soundness

o reliability

o validity

• usability/feasibility of use

o low burden (e.g., length, time/effort to complete) and feasibility

o fits with standard of care and related workflows (e.g., actionable, incorporated, and

discussed at point of care)

o readability (e.g., Flesch Kincaid score)

o cultural appropriateness

o translated with culturally appropriate items

o availability of standardized clinical terminology and codes

o guidance on standardized data collection (including modes and methods)

Measure developers must be aware that many of the desirable attributes for a PROM are subjective,

making them difficult to quantify. Additionally, gold standards for what constitutes a “high-quality”

PROM are lacking. When developing a new PROM, the measure developer will also face the challenge of

the PROM not being in use at the outset to test the measure attributes, including gathering feedback.

December 2023

Supplemental Material to the CMS MMS Hub Patient-Reported Outcome Measures

December 2023

This underscores the need for gathering feedback continuously from multiple interested parties

throughout the PROM development process.

1.4.2.3

Additional Considerations for PROM developers

A major consideration for measure developers developing PROMs is to ensure the ability to attribute the

items to patient-reported outcomes accurately and fairly to the measured entity. Measure developers

have a unique charge and perspective in survey and item development, as the design or most surveys

and items is not to track provider performance. PROM developers must keep the intended use (e.g.,

attribution to care provision) in direct focus throughout the development and testing for eventual

implementation.

Measure developers must test the PROM on a representative sample of the target population, including

all sub-groups to ensure no bias, for instance, across gender, race, or social and cultural groups. If the

PROM is multimodal (e.g., phone, mail, and online) the measure developer will have to test the effects

of the various modes. If translated into additional languages, the measure developer will also need to

test each translated version of the PROM. The measure developer needs to consider whether to allow

proxy responses in the PROM and how well they estimate the target population’s responses.

PROM developers will need to conduct cognitive testing to ensure items are clear, relevant to the

concept, and valid. PROM and PRO-PM developers must assess non-response to the instrument and

specific items. Measure developers must assess the items and the whole instrument using Item

Response Theory (IRT) and PROM-relevant methods, such as structural equation modeling and latent

variable models. Measure developers should be aware different methods can lead to different

interpretations of the tool’s performance. They will also need to evaluate any scales and determine

protocols for implementing the tool.

1.4.3 Determine the Appropriate Performance Measure: the PRO-PM

The measure developer should report the outcomes for target/initial populations as average change or

percentage improvement determined by the topic of interest. The measure developer must test all

measures for reliability, usability and use, feasibility, validity, and threats to validity, including how to

handle missing data and appropriate risk adjustments. To appropriately distinguish variations in

performance between measured entities, the outcome must capture the results of the care given and

not the influence of comorbidities or other extraneous variables. However, as in any other outcome

measurement, the measure developer should not allow risk adjustment to mask disparities. The

supplemental material, Risk Adjustment in Quality Measurement , contains a discussion on

determining the need for risk adjustment and development, and evaluation of risk adjustment models.

2 PATIENT-REPORTED, OUTCOME (PRO)-BASED PERFORMANCE MEASURE

EVALUATION

The measure developer should evaluate outcome measures, including those based on PROs, against

standard criteria in the same way that measure developers evaluate all measures under development.

Supplemental Material to the CMS MMS Hub Patient-Reported Outcome Measures

Some of the unique considerations (in addition to the others in each category) that apply to evaluating

PRO-PMs include

• Importance—The measures must be patient-centered. Patients must be involve

d in

identifying the PROs used for performance measurement.

• Scientific Acceptability—Specifications must include methods of administration,

handling of proxy responses, response rate calculations, and how the responses affect

results. The measure developer must establish reliability and validity not only for the data

measurement instrument (i.e., PROM) but also for the derived performance measurement

(i.e., PRO-PM).

• Feasibility—Minimize burden to respondents. Illness may complicate accessibility issues.

easure developers should consider language, literacy, and cultural issues.

• Usability and Use—Not only must patients find the results of PRO-PMs useful, but measured

entities must also be able to use the information to improve quality of care

ROs in Performance Measurement suggests endorsement criteria for PRO-PMs.

Evaluation for PRO-PMs is a special case of overall outcome measure evaluation. PROs in Performance

Measurement outlines criteria specific to PRO-PMs. The overarching principle is these measures

should consider the patient foremost. Quality measures designed to capture performance on PROs

should be

• Psychometrically sound—In addition to the usual validity and reliability criteria, the measure

developer should consider cultural and language considerations, and patients’ burden of

responding.

• Person-centered—Quality measures should reflect collaboration and shared decision-

making with patients. Patients become more engaged when they can give feedback on

outcomes important to them.

• Meaningful—Quality measures should capture impact on health-related quality of life,

symptom burden, and achievement of personal goals.

• Amenable to change—Outcomes of interest must be responsive to specific health care

services or intervention.

• Implementable—Data collection directly from patients involves challenges of burden to

patients, health literacy of patients, cultural competence of measured entities, and

adaptation to computer-based platforms. Evaluation should address how to manage these

hallenges.

3 KEY POINTS

PROMS are quality measures derived from patient-reported outcomes and are a high priority for CMS

and other organizations. These measures present some design challenges and require measure

developers to construct PRO-PMs that apply patient outcome data to measure quality of care. PRO tools

measure developers can use include PROMIS , HOS , and FOTO

. The same measure evaluation

criteria and justification principles that apply to other outcome measures also apply to PRO-PMs. It is

important for measure developers to create patient-centered PROMs and produce data that measured

entities can use to improve quality of care.

December 2023

Supplemental Material to the CMS MMS Hub Patient-Reported Outcome Measures

December 2023

REFERENCES

Centers for Medicare & Medicaid Services. (n.d.-a). Back pain after lumbar discectomy/laminotomy.

CMS Measures Inventory Tool. Retrieved September 28, 2023, from

https://cmit.cms.gov/cmit/#/FamilyView?familyId=85

Centers for Medicare & Medicaid Services. (n.d.-b). CMIT environmental scan. CMS Measures Inventory

Tool. Retrieved September 28, 2023, from https://cmit.cms.gov/cmit/#/EnvironmentalScan

Centers for Medicare & Medicaid Services. (n.d.-c). Functional status assessment after total knee

replacement. CMS Measures Inventory Tool. Retrieved September 28, 2023, from

https://cmit.cms.gov/cmit/#/FamilyView?familyId=279

Centers for Medicare & Medicaid Services. (n.d.-d). Welcome to CMIT. Retrieved September 28, 2023,

from https://cmit.cms.gov/cmit/#/login

Centers for Medicare & Medicaid Services. (n.d.-e). Welcome to the Medicare Health Outcomes Survey

(HOS). Retrieved September 28, 2023, from http://www.hosonline.org/

FOTO. (n.d.). FOTO patient outcomes. Retrieved September 28, 2023, from https://www.fotoinc.com/

LOINC. (n.d.). Home. Retrieved September 28, 2023, from https://loinc.org/

LOINC. (n.d.). Submissions. Retrieved September 28, 2023, from https://loinc.org/submissions/

National Quality Forum. (2012). Patient-reported outcomes in performance measurement.

https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=72537

National Quality Forum. (2021, August). Building a roadmap from patient-reported outcome measures to

patient-reported outcome performance measures: Interim report.

https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=95978

National Quality Forum. (2021, November). Building a roadmap from patient-reported outcome

measures to patient-reported outcome performance measures: Technical guidance – final draft.

https://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=96460

Northwestern University. (n.d.). HealthMeasures. Transforming how health is measured. Retrieved

September 28, 2023, from http://www.healthmeasures.net/explore-measurement-

systems/promis

Reeve, B. B. & Hays, R. D. (Ed.). (2021). Advancing methods to assess patient-reported outcomes:

Lessons learned from the patient-reported outcomes measurement information system®

(PROMIS®) initiative [Special section]. Psychometrica, 86(3), 669-842.

https://link.springer.com/journal/11336/volumes-and-issues/86-3