I'm a research professor at the Duke Clinical Research Institute.
And I've been doing patient-reported outcomes research for about 15 years or so.
And I'm very happy to be invited to speak to you tonight about it.
This is actually a great time to be talking about this because
there are so many exciting developments in patient-centered research these days.
And so, we've got this lecture.
And I think our -- the next lecture will also be about self-report
approaches to research.
So, I'd like to go over a few different things today.
It's a huge, huge field, but I'll try to pick out what I think
are some of the most basic issues that would be useful for everyone to know.
First of all, why measure patient-reported health status?
What are the different types of patient-reported outcomes?
When I say PROs, it's going to be patient-reported outcomes.
How are patient-reported outcome measures developed and evaluated?
Just very briefly, I want to go over that.
We want to talk about some "new" methods for doing patient-reported
outcomes research using item response theory.
And I put new in quotes because it's new for health measurement,
but it's not new for measurement in general.
Education testing folks have been doing this for a long time.
So, we're a little late to the game, but there are some exciting developments.
So, we want to mention those.
And then finally,
I want to say a word about interpreting scores that you'd get out of patient-reported
outcome measures. Okay?
So, let's start with the first.
Why measure patient-reported health status?
Well, let's think very generally about a situation where
we're interested in trying to measure the benefits of some intervention.
And so, how do we assess benefit?
If the intervention is the kind of intervention that's supposed to act on
the body, it's a pill or some procedure, we've got all sorts of measures
that we use to assess the effect of some agent on the body, right?
We've got copies of some virus in the blood, tumor size, blood pressure, peak VO2.
These are all examples of these endpoints we use in studies
that tell us how the body has changed, all right?
But for endpoints to really matter, to have endpoints that will inform decisions,
they've got to be endpoints that are relevant to the stakeholders
here, chiefly patients, clinicians, and payers.
And there are probably even more stakeholders, but those are sort of the big
three.
And it's not always the case that some of those endpoints that we use
to measure the exact effect on the body translate well into day-to-day functioning.
So, let's take peak VO2, all right?
Peak VO2 is a measurement that's used in a lot of trials for COPD or heart failure.
And one might ask, "Well, how does someone's peak VO2 relate to their
-- the extent of physical functioning that they can perform during their day?" Right?
Well, across a few different studies,
we see that it's a -- there's a moderate size correlation in what our group did.
The correlation between peak VO2 and PROMIS Physical Function scores is 0.53.
Now, if you want to get a little geekier on the statistics, you know, that tells you
that there's about 72 percent of the variance
in physical functioning that's not accounted for by peak
VO2, right?
So, it's still the case that the majority of the differences among people
in their day-to-day physical functioning is not explained by their peak VO2, right?
They are used for making an assessment of their physical functioning,
because we know that just measuring peak VO2 is not going to give us the best indication
of how this new treatment
we're testing is actually going to change their day-to-day lives, all right?
Another quick example, overactive bladder syndrome, all right?
So, for a lot of studies on overactive bladder syndrome,
one of the things they'll use to measure volume is pad weight, all right?
They'll have people bring in their pads in Ziplock bags
or whatever, and actually measure the volume that has leaked, all right?
Well, one might ask, "Well, what is a meaningful reduction in volume?"
I might have an intervention that is reducing the volume to some degree.
But for a patient, what constitutes a meaningful reduction in volume?
It might be the case that, you know, "Unless you can reduce my volume
to the point where I don't have to go out wearing a pad, I don't care.
You might be very excited that it's a statistically
significant reduction of 30 percent volume, but it's not particularly relevant to me,
because it's not going to drive what I'm doing so much." Right?
So, it's not always straightforward
how to assess benefit in terms of sort of these harder outcomes.
And we want to turn to actually querying patients to find out
what effect some of these treatments are having on their day-to-day lives.
So, when we talk about treatment benefit, you know, the FDA is quite explicit
in their definition of benefit in terms of feeling, function, and survival.
And when we're talking about feeling and functioning, we're really in the domain
now of trying to figure out what's happening in people's day-to-day lives.
And the best way to assess that oftentimes is through a patient-reported outcome measure.
And we'll stick with the FDA for a second here,
and they've got a very good definition -- whoops.
Sorry about that.
No signal.
I'm sorry.
Oh, here we go.
Hopefully, this will work.
Because if it doesn't, I'll have a very awkward
interpretive dance version of this entire talk.
Okay. Here we go.
So, the FDA defines a patient-reported outcome as "a measurement based on a report
that comes directly from the patient about the status of a patient's
health condition without amendment
or interpretation of the patient's response by a clinician or anyone else." Okay?
And we're talking here about all sorts of things that match this definition.
It could be symptom diaries. It could be multi-item surveys.
It could be in person interviews where responses are being recorded.
The data can be captured electronically, paper and pencil, on a phone, all right?
All these things could be patient-reported outcome measures.
And just to throw up an example of one, here's the NIH PROMIS fatigue short form,
right, where we're asking multiple items to try to get at someone's fatigue.
And the responses to those items will be combined in some way
to arrive at a single score that reflects their self-reported fatigue.
All right? Okay.
So, there we have the sort of, why are we interested in patient-reported outcomes?
Okay?
Let's talk about the different types of patient-reported outcomes.
And we're going to walk through a model that I've adapted from a wonderful article
by Wilson and Cleary.
It was published in JAMA in '95.
Their terminology is a little bit older, but it still is one
that sort of drives our field.
So, we begin with these biological and physiological variables.
And we've talked a bit about these, right?
So, things like CD4 count, HbA1c, or tumor size.
Again, these are things that are measuring how
the state of the body or how the body has changed, right?
Those bodily changes give rise to different symptoms, right?
And so, symptom status like pain intensity, nausea, urinary straining.
Symptom status is the first time we start to see a potential
for a patient-reported outcome here, right?
Changes in symptoms can affect the person's day-to-day functioning.
And so, functional status then is sort of the next step in the causal chain.
It might include things like walking or what
people referred to as activities of daily living, or social interactions, right?
There are all sorts of ways in which the person needs to function
during the day that could be compromised by certain symptoms,
right?
Now, the person's symptoms and their functional status together
might inform how the person sort of views their overall health.
And so, general health perceptions is the next type of patient-reported
outcome in the chain here.
And sometimes we assess that very generally
with a simple question like one from PROMIS or the SF-36.
"In general, would you say your health is excellent, very good, good, fair, poor?
All right?
It doesn't seem like a particularly exciting item, but
it's amazing that responses to this item are independent
predictors of mortality after controlling for other clinical characteristics.
So, it's a good kind of a catchall type of item.
People's overall health
then informs their evaluations of their overall quality of life, right?
And so, we might ask something like, "Overall,
how satisfied are you with your life as a whole these days?" Right?
So, much broader question, and it encompasses much more than just their health.
And so, we've got a lot of these nonmedical factors, financial, you know, socioeconomic
considerations that are informing one's overall quality of life.
But certainly, one's health related quality of life could be a major determinant of that.
Okay?
So, we've got this sort of chain, this causal chain of patient-reported concepts here.
Let's talk a little bit more about these.
One of the things we know is that any of these things, symptom status,
the functional status, my overall health perceptions, any of those things
could be influenced by other characteristics about the person.
So, it's not just what's happened to my body.
It's not just the drug I've got, or the disease I have.
But there are individual characteristics, all right?
Any clinician here knows that
certain patients seem to have personalities that amplify the experience of pain.
And so, you have two patients with the exact same structural or physiological problem
who have very different pain experiences, right?
So, there are characteristics
of the individual that can also influence all these different things.
And I've drawn thicker arrows there going from left to right to indicate
the general understanding that these other characteristics
of the person have an increasing effect as we go out here.
So, the individual state of the body is probably not going to be a strong determinant
of one's overall quality of life as other characteristics of the person, right?
Similarly, not only there are interesting characteristics about the individual
that influence
their symptom status, functional status, their general health perceptions,
but there are also characteristics of the person's environment, all right?
If you think about how much your functional status is affected
by the physical environment you're in, it's quite profound, right?
What types of supports are there?
We've got some right here, right?
If I have a little bit of trouble walking, these rails are wonderful, right?
And I might report when I got up to the top that I don't have that much difficulty
going upstairs in the environment I'm in.
But if you put me in an environment where you didn't have these awesome rails,
I might be reporting
that I have a bit more difficulty walking upstairs,
because there's a parentheses in the environment I'm in, right?
So, the environment can play a large role in the person's experience of their symptoms,
the impact on their functioning, their general health perceptions.
And again, that influence might be increasing as we go from
left to right in the -- these concepts here.
Okay?
I've talked -- I've alluded to this a little bit,
but there's an important point here -- there's actually two important points here.
One of the important points is that we see a dilution of effects
of biological interventions as we go from left to right, all right?
So, if I have an intervention that is to change the state of the body,
it's going to change
some balance of chemicals in the body in order to improve the disease, right?
Or if it's a surgical intervention, it's going to take something out that's
causing a problem, right? Those are changes on the body.
We're going to try to observe the effects of that intervention.
But one of the things implied by this model is that the effect of that intervention
will be seen most acutely in those things that I measured in terms
of their -- of the biology or physiology of the body, right?
Next, it will be symptoms.
Next, it will be functioning.
But because there are a lot of other things, other characteristics of the individual,
other characteristics of their environment that contribute to the symptom experience,
functional status,
general health perceptions, that effect of that biological intervention gets diluted.
And so, it's harder for me to show an effect of my intervention
when I'm measuring things further out here, all right?
It's harder to show an effect
when I'm measuring things further out, because so much more contributes to it.
It's just a lot messier. It's more complex.
At the same time, though, for many patients, those are the things they are interested in.
They want to know how this is going to improve my functional status during the day.
That would be really important to show that.
But we're saying that it's going to probably be a little bit
more difficult to show that than it would be to show symptom improvement, right?
Or improvement in some blood levels of some relevant chemicals, right?
Okay.
So, dilution of effects of biological interventions as we move from left to right.
That's one of the implications of this.
And another one that shouldn't be too surprising
is that the correlation between successive boxes decreases, right?
So, the correlation between the biological and physical -- physiological variables
and symptom status is much higher than the correlation between biological
and physiological variables and general health perceptions, right? Okay?
So, those are also things to be aware of.
And again, and why is that?
It's because the -- all these other things are affecting the chain as we go down.
So, the further out you are, the more of those things are affecting
the boxes differently.
Okay?
So, this is a helpful overall model, I think, to try to characterize
the main types of patient-reported outcomes that we might be interested in,
and to plan our study and really ask, "Well, what do we want to measure here?
Are we interested just in symptoms?
Are we interested in impact on functioning?
Are we interested in how the person thinks about their overall health?" Right?
And we know now that if we are interested in things that are further out to the right,
we might have to work a little bit harder to show that effect.
We might have to do things like, for example, try to measure
some of these characteristics in the environment
and characteristics of the individual and include those in our study
as a way of trying to improve our measurement of our treatment effect, all right?
Okay.
All right.
So, that was one of the types of different PROs.
Let's go through a quick review of developing and evaluating PRO measures.
And I don't want to spend too long on this.
And I know in our next lecture there will be some similar material covered.
But I do want to give you an appreciation of the overall shape of how this goes.
Okay? All right.
So, the first thing we're going to do is to determine
what PRO concept we want to measure and why we want to measure it.
And let's just take the example of fatigue, all right?
So, a bunch of investigators are sitting around, "We think we need
a better fatigue measure." Okay?
Well, when we're all talking about it, we've got an -- a kind of a notion
of what we mean by fatigue and sort of where the boundaries of that concept might be.
But it's still kind of muddy, all right?
But we've got some rough concept of what might constitute fatigue, all right?
But to check ourselves, though, we want to go and we want to talk to patients
and maybe their caregivers
who actually are having experiences that we think would be described as fatigue.
And so, the second general step is to collect some qualitative data
to understand the meaning of this concept, right?
So, might do focus groups or individual interviews.
Talk to people about, you know, "Tell me what it's like to have this.
What's your day like?
What -- is fatigue the same as tired?" Or I'm going to talk about --
and as we do that, that fuzzy concept we have, it should hopefully become a bit
better defined for us, so that we can say that as a result of talking to folks,
we now have a much clearer picture of this thing we're trying to measure.
And let's try to write now a preliminary definition of fatigue
and what the parts are that we're going to try to measure fatigue.
Okay?
Once we've got that, we're going to need to write some items.
So, that's the next step.
Let's write items that we think are going to measure that concept, all right?
I just put eight items here because they fit nicely on the screen.
But you can write four items, or you can write 400 items.
It really depends on the concept you're measuring, okay?
And one important thing here is that as soon as we start writing
the items, we've got a preliminary measurement model.
It's a preliminary model or a picture of what -- how we think this concept is defined,
all right?
We think that there's this thing, fatigue, that's inside the person that is going
to cause their responses to each of the items we're going to ask them, all right?
Fatigue is going to cause them to respond in certain ways.
If you're more fatigued, you're going to be more likely to say, "I've -- you know, I have
-- I did not have the energy I needed to get things done today." All right?
Okay. So, that's our kind of picture.
And by the way, there are different types of measurement models.
We're not going to get into them, because it's just too mind-numbing
for a late afternoon evening.
But I've presented the most popular type of model we see here.
It's called the unidimensional model
where all the items are thought to be measuring one thing, and that's fatigue.
Okay? So, I've got this preliminary model.
We're going to come back to that, though.
Okay?
So, now I've got these items that I and my other investigator friends wrote, right?
And they're probably awesome items, because we all have PhDs, right?
Absolutely wrong. Some of them will be terrible items.
And so, the next thing we've got to do is to go test these items with the patients
or respondents from the population we are interested in studying.
And we're going to sit down and talk to people about the items.
We're going to ask them how they understand the item.
What do they think the item is asking?
How do they come up with their response? All right?
We want to make sure that people
are thinking about these items the same way we're thinking about them.
And we will definitely, as a result of that, if we do it well, find out
that some of our items are really crummy, and that we might have missed
some aspects of this thing we're trying to measure.
A person says, "Well, okay, I don't really understand
what you're getting at with this question, you know, but I'll tell you that
it reminds me a much better thing to be asking here is blah." All right?
So, we might ideas for a new item.
So, a very important step to be talking with the patients and people with the disease
is to find out how they're understanding the items and make sure
we've got items that are understood by everyone at the same way.
Once we've got that set of items, we can now go and administer
that set of items to a large sample of people.
And we want people who have the diseases of interest,
and we would love to get people who represent the whole range of symptom severity,
all right? The whole range of severity.
So, collecting data on a big set of people allows us then to use psychometric analyses.
And it's just another word
for statistical analyses that are specific to developing and evaluating measures.
Okay?
We're going to use psychometric analyses to see how -- to do a couple of things here.
The first thing is we want to see how well are those items working statistically,
all right? Remember I said we developed that sort of preliminary model?
Well, we've got a whole bunch of analyses we can do to say, "Hey,
how well do those items fit together like that?
How well do the items really seem to be caused by the same thing?" Or, "Are
some of these items really not hanging well with the other ones?" And
it may be that they're measuring something a little bit different than fatigue, right?
We do those psychometric analyses to see whether our items are behaving,
they're working the way we thought they would work to be
telling us something about this common thing called -- we're calling fatigue.
The other reason why we're going to do psychometric analyses
is, once we've got the right model, we want to find out how to get a score, right?
We've got eight items right now, but we would like to figure out
how to estimate a score for each person,
one number that will now represent that person's fatigue for us, okay?
Okay.
And I'm happy to take lots of questions afterward
about more details of the types of analyses that we do.
They're -- they can get a little bit squirrelly, but I do like talking about them.
So, if we have time after, I'm happy to go into that.
Then we're ready for our next step.
We've got a measure that we think measures fatigue.
We've got scores that we can get out of this measure now.
And now, the question is to evaluate the reliability and the validity of this measure.
And by validity here, we mean that the measure is measuring
what it's supposed to measure and nothing else, all right?
So, we're sort of hitting the bull's eye, all right?
In terms of reliability,
we mean here that we're measuring this thing with very little error.
Another word for reliability might be precision here.
And so, in the metaphor of a target, validity is how close you are to the bull's eye,
reliability is how tight a cluster do you have, all right?
Okay.
Now, if you'd get into the patient-reported
outcome literature a bit and start to look at tests -- or I'm sorry,
measures, you'll see that there are a lot of different kinds of validity mentioned.
There are lots of different kinds of reliability.
We don't have time to go into all of them, but I just want to make you aware of this.
So, you might see, for example, reference to content validity, or construct validity,
or responsiveness.
These are all different ways of assessing the degree to which
the measure is measuring what it's supposed to measure, all right?
I'll give you one quick example of something called convergent validity.
The rationale for convergent validity is that if I've got a measure that's supposed
to measure fatigue, and over here is another well-established measure that is thought to
measure fatigue, well, scores on my measure and the established measure should converge.
They should be correlated in different samples.
And so, here are actual data
from the NIH PROMIS depression domain, some measure of depression.
And the distribution of scores is on the histogram at the bottom
there, and off to the left is the distribution for another
really well-known depression measure, the CESD.
And when we noticed we're developing the PROMIS measures,
we looked at the convergence between scores on PROMIS measures
and scores on corresponding measures of the same domains here.
And so, we can see for this one, correlations 0.84,
there's very strong convergence, right?
It gives us more confidence that the PROMIS depression measures
are measuring the same type of depression that the CESD is measuring, right?
Although minor geek point here, if you'll compare the two frequency
distributions, you'll see the CESD exhibits a fairly significant floor effect, we'd say.
Everybody's piled up at the bottom, right?
The CESD is not quite as sensitive for lower levels of depressive symptomatology, right?
So -- whereas the PROMIS depression distribution is much closer to normal,
it does a better job of discriminating among people with lower levels.
So, those are some other fun things we checked out when we're comparing measures.
Okay?
So, that's just an example of a particular kind of validity.
Okay? Let's talk about reliability for a second.
And I'm just going to go through three quick examples of reliability.
The idea here is that if my health state has not changed, right,
if I'm basically in the exact same health, I should get the same score
if you use different sets of items from the same measure.
So, if I've got a 40-item measure, and those items are at a reliable measure,
then any random subset of those items should be giving me the same score, right?
And that's known as internal consistency.
If you see -- ever see Cronbach's alpha, that's a measure of internal consistency
reliability, all right?
This idea that different sets of items
from the same measure would all end up giving me the same score, all right?
If they don't, I mean,
some of the items aren't hanging together with the other ones too well.
If I've not changed, I should get the same score over time on this measure, all right?
If I know my true health state has remained the same
but the scores are bouncing around, that means there are some error there, right?
There's just noise.
And that's known as test-retest reliability.
And then in the special case where we've got measures
that require some raters to assign some ratings to things they observe
or written responses people provide, this doesn't happen
that often, but that kind of reliability is known as interrater reliability.
It means that I'm going to get the same score on this measurement system
regardless of who is scoring the thing, all right?
So, internal consistency, test-retest, and interrater
reliability are three of the big kinds of reliability you see.
Okay?