All right.
I talked fast again. I have caffeinated.
So, let's do a quick recap here.
We're going to determine what concept we want to measure and why.
We're going to go and collect
qualitative data from people to make sure that we really have defined this thing.
Once we've got it defined, we can write items for it.
We're going to go take the items to the actual types of people
that are going to be taking this measure, find out if they understand it,
revise the items, make sure our revisions work with new people.
Then we're going to go and give the items to a whole bunch of people,
so that we can do psychometric analyses.
And those psychometric analyses
are going to allow us to test how well our original model fits the data.
Well, there's a better model we've got to use.
And it's going to allow us to derive a score, right,
or a set of scores maybe, but some score that we can use.
And then finally, once we're in possession of this, we can take that out
and go test it now to see what kind of reliability and validity it exhibits.
Okay?
All right.
So, that was the development and evaluation process in a very small nutshell.
So, I'd like to now talk about some very interesting developments
that have occurred, and how we developed patient-reported outcome measures,
and some of the cool things we can do with them now.
And it's especially relevant because NIH has actually invested
a good deal in developing measures using some of these newer approaches.
And remember
I said new, it's new for health, but not new for the measurement world, okay?
What we're going to be talking about is something called item response theory.
Okay?
And what I want to focus on here are -- is what's different
between a regular old measure and a measure developed using item response theory.
And then I want to tell you about one very cool little thing
you could do called differential item functioning.
Okay?
So, let's start with contrasting the different types of measures.
And let's start with -- here is -- here's your basic off-the-shelf measure.
This is the FACIT-Fatigue Scale, right, developed by Dave Cella and colleagues
to measure fatigue especially in cancer populations.
And that -- this is a great example of your traditional off-the-shelf measure.
I'd say off-the-shelf PRO measure
because you just go and you grab it and you're ready to go.
Everyone's going to complete the same items on the FACIT-Fatigue, all right?
All of the items are necessary to obtain a score.
You got to complete all of the items, because you average or sum
up all the items to get the score, all right?
And the score on the FACIT-Fatigue
might not be on the same metric as other measures of the same thing.
So, for example, the SF-36 has an energy domain that's supposed to be measuring
about the same thing.
But those scores are on a different metric.
The absolute value of those scores mean a different thing for the FACIT measure,
right?
It's kind of like having two different thermometers that are both supposed
to be measuring temperature, but one gives it in one set of numbers and another
gives it in another set of numbers, and they're not comparable at all.
Okay?
This is what we get with traditional off-the-shelf measures.
In contrast, when we use item response theory,
we're able to create something called an item bank.
An item bank is a large collection of items measuring a single domain
like fatigue, right, or pain intensity. Okay?
Any and all of the items from the bank can be used to provide a score for that domain.
So, you can pluck out one item and estimate a score on fatigue.
Or you can pluck out 40 items and combine them all and estimate a score on fatigue.
Okay?
That means that item banks can be dynamic, not fixed.
We can add items to them overtime if we end up getting some better items.
We can sit in some new items and maybe get rid of some items
that are -- that don't seem to be quite as useful.
So, it's a dynamic resource.
All right.
Let's look in an item bank
for the domain of physical functioning just as an example here.
Okay.
So, we know there's this range of physical functioning out there in the world.
We know that we would like to systematically measure that range, right?
And so, to do that, we will create an item bank for physical functioning.
And in this case, the item bank is a collection of items
that all measure physical function.
Okay? And so, we've got examples here.
"Are you able to get in and out of bed?" "Are
you able to stand without losing your balance for 1 minute?" All right.
"Are you able to run 5 miles?" Okay.
Any item or set of items from this bank
could be used to generate a score for a person for physical functioning.
Any of them.
And why is that?
It's because they -- we've
put these items through the ringer to make sure that they fit in item
response theory model.
It's a statistical model that allows all of these properties to hold, okay?
And note here, too, that these items, as I've laid them out
here, are kind of arrayed in order of severity, all right?
Are you able to get in and out of bed?
That's kind of question
you'd ask of a person who looked like they were incredibly weak, right?
That's an item that's good for discriminating among people
who are at the lower end of functioning, right?
"Are you able to run 5 miles?" is something
that would be informative to ask of a person who's in the higher end.
And this tells us another interesting property about item
banks, is that with item response theory, not all items are the same.
Some items are better used for lower severity or higher severity.
And when we're thinking about which items we might want,
we have to think about the population we're using.
And we have the option to tailor our measures.
More about that in a second here.
Okay?
So, back to off-the-shelf,
if I'd just use my traditional off-the-shelf measure, I take it off the shelf,
and I just give it to the patient or send a web link or
however it is, and they all fill out the same measure and that's it.
Okay?
So, it's really easy to figure out what to do.
You take the off-the-shelf measure and administer it.
It's not as straightforward
with an item bank, because you've got some decisions, all right?
Let's go through those here.
Basically, I've got one of two main strategies here,
a fixed-length or computerized adaptive tests.
With the fixed-length measures,
everyone is going to get the same number of items.
It's a fixed-length for everyone, all right?
And I've got a couple different options there.
There are some very handy ready-made fixed-length measures.
So, if you want to use measures from the NIH PROMIS system,
you can go to the Assessment Center website and you could find ready-made short forms.
These are forms with about six
to 10 items per domain that are good for general populations.
They're generally sensitive
to the range of functioning there that the PROMIS folks have picked.
It's sort of a good kind of off-the-shelf version of PROMIS measures.
So, there's some ready-made fixed-length measures, all right?
I could also go in -- if I've achieved the requisite
level of geekiness, I could go into the system
and I could pick out the items that I would like that
I think are going to be most sensitive for my population, all right?
So, if I'm -- if I've got a trial where I've got severely ill heart failure
patients, I don't want to be asking people how much trouble they have running 5 miles.
It's a waste of their time, right?
I'm going to go in and I'm going to select the items that are going to be most sensitive
for my population.
And I'm going to give everyone in my trial those items, all right?
Okay. So, I could tailor them.
I can make my own.
So, those are both versions of fixed-length.
The other
main way of using items from an item bank is through computerized adaptive tests.
And, you know, a lot of you might be familiar with computerized adaptive testing,
because they use it a lot for education
like the SAT, GRE, a lot of the certificate --
certification tests in medicine, nursing, and government safety tests a lot.
What it does have is these big banks of items, and you sit down
with a computer and they show you one at a time, all right?
So, same deal here.
The idea is that the next item that you receive is dependent
upon your answer to previous items, all right?
And so, if I've got a range of items across this -- the spectrum of severity,
a computerized adaptive test or CAT might start with one right in the middle.
And if I seem to have a lot of difficulty doing that, it's going to put me
on the more severe side and say, "This guy sounds more severe.
Let me ask him a question on the more severe end and see how he answers, all right?
I'm going to answer that." I'm just going to say, "Oh, it seems like it's around here.
I'm going to ask a question here." And so, in some ways it's very similar to the way
a clinician might be asking questions of a patient.
Let's start with a general one.
And then as you're getting the sense of how severe or mild they are,
ask more questions to get a more precise picture of what that person looks like.
But the idea with the -- with a CAT is that in theory
every person could get a different set of items, right,
because the items that you get pulled out of that big item bank
are the ones that are getting us to the most precise estimate of your score
as quickly as possible. All right?
And we find often that you can get
a very precise estimate of people's scores by asking them only four questions.
But they have to be the right four questions.
And how do you figure out which are the right four questions?
Well, you have an algorithm,
a computerized adaptive testing algorithm that is drawing items from that item bank.
All right?
Having an item bank now with these different options allows a few things to happen.
Imagine we got this fatigue item bank here, and it has said 40 items in it.
Well, I could use items from that item bank in all different settings, right?
And I could use them in different ways too.
So, the chemotherapy trial might have items one through 10.
For the osteoarthritis trial, I might be using a CAT, a computerized
adaptive test, right?
And what's great is because all of the items
being pulled for all these different studies were from the item bank,
they all have the same metric, the same meaning.
The scores are comparable across all of those different studies, all right?
You can imagine how that might help facilitating things like meta-analysis
and the like. Here's another great application.
Right now, there's a lot of interest in pragmatic clinical trials or comparative
effectiveness research where we're trying to do a research within healthcare systems.
I might have different healthcare systems each of whom are using different measures,
all right?
Well, right now, each of those measures results in a different metric,
and it's really difficult to pull results then.
If, however, all of those items from the different measures
could be placed within an item bank, which means we do the kinds of item
response theory analyses that test whether these guys can all sit
in the same bank, and maybe we've got some other items in there,
now we're in a great position because we can get one metric out of that.
Right?
So, now,
even though different healthcare systems might be using different sets of items,
if they're all on the same item bank, we can -- we've got a cross-walk going
and we can get the same metric out of all of them, right?
Okay.
So, a quick review here comparing this traditional PRO
measures, the off-the-shelf measures with the PRO item bank.
With the traditional, all items are required to compute a score.
With item banks, any and all subsets of the items can generate a score.
With the traditional, everybody's got to take the same items.
With an item bank, different people could get different items.
With the traditional measure, you've got a -- you can just pull it off the shelf.
But when we've got an item bank, you've got to use items in the bank
to create measures for a specific use.
And we went through some of the different options there, right, fixed-length or CATs.
If it's fixed, you can get ready-made ones that someone might have created for you,
or you can go and create your own custom one.
And finally, in traditional PRO measures, the scores are not easily
comparable to other scores from other measures of the same thing.
Whereas with an item bank, there's much greater opportunity for cross-walk
between scores from different measures, as long as the items from
the different measures can be put in the same bank.
Okay? All right.
I mentioned that NIH has invested in item banks
quite heavily and integrate benefit, I think.
A few examples here.
The biggest one is probably PROMIS, the Patient-Reported Outcomes Measurement
Information System.
And over about 10 years of development, PROMIS has amassed
quite a few excellent measures of the main types of health concepts
that are relevant across chronic diseases for adults and children and adolescents.
They're freely available.
You can go to nihpromis.org for more information there.
NIH also supported the NIH Toolbox, which is for assessing
neurological and behavioral function, ages 3 to 85.
Again, freely available. You can go to nihtoolbox.org.
And finally, Neuro-QoL is for use in chronic neurological conditions.
And it's free. And you can go to neuroqol.org.
So, you might want to check those out, and you can get more information
from those sites too about how item banks work, all right?
I said that there was
a very cool thing you can do -- there are a lot of cool things
you can do with item response theory, but I wanted to tell you about a specific one.
It's differential item functioning, okay?
So, check out this very simple question, all right?
"In the past seven days, did you cry?
Yes or no?" This is -- I've just made a yes-no version of an
-- a kind of item you often see in depression measures, all right?
And I think that we would all agree that, in general, responses to this item
might inform us to some degree about the likelihood
that a person might be a bit depressed, all right?
The more likely you already have cried in the last week,
the more likely you are to have been depressed.
This has been shown to be the case, okay?
Imagine that we wanted to compare men and women's depression though,
and we had an item like this and a few other similar items.
We found that men and women differ in their depression levels.
What we'd like to know is that men and women truly do
differ in their underlying depression levels and not that
they actually differ in the way the items work for them, okay?
So, hang on for a second.
We're talking about differential item functioning.
It means the item behaves differently for two or more groups.
So, check this out.
There's a very simple item response theory curve here.
And you can see
we're modeling the probability of answering yes to that question, on the y-axis,
as a function of the person's underlying depression, on the x-axis.
Okay?
This, by the way, is why it's called item response theory.
They're models of the likelihood of responding
with a given answer based on your underlying level of whatever.
In this case, depression, okay?
So -- and you can see here as the person's depression -- as depression increases, so too
does the probability they're going to answer yes to "Did you cry in the past week?" Okay?
And so, what we're saying when we say that this is informative about
depression, is we're saying, well, we know that of people who answer yes,
those folks are most likely to be on the high end of depression.
So, we can make an inference that they are on the higher end of depression.
That's how we're using that item.
This is a map that maps the responses to that item to their underlying depression, okay?
And so, we take someone who is, say, one standard deviation above the mean of 0
on depression, they've got a 0.9 probability of answering yes on that, right?
Okay.
So, we've said that differential item functioning is what happens
when an item behaves differently for two or more groups.
Another way of saying that is that the map between depression
and the item is different for two or more groups.
I was talking about men and women with this crying item.
Okay. Let's look at this.
This is the map computed separately for females and males.
And what is it showing here?
If we've got a male who we know because we've suddenly become omniscient
for just a second is one standard deviation above the mean on depression,
the male's likelihood that they're going to answer yes is 0.3, right?
It's a 30 percent chance that they're going to answer and say, "Yes,
I've cried in the past week." A woman though who is one
standard deviation above the mean has a 0.9 probability of answering yes, right?
Now, why?
On average, women will respond that they cried in the past week
more often than men independent of their underlying level of depression, right?
But you can see how messed up you'd get here though if you just used one
version of the map and applied it the same for males and females here.
So, this is an example of differential item functioning.
And when we're developing measures using item response theory, one of the things
that's really important for us to do is to think, "Are there groups
for whom these items might behave differently?
Would they behave differently for young versus older people?
Would they behave differently for different cultural groups?" Right?
And we can explicitly test for that using item response theory
and conducting tests of differential item functioning.
And it will identify for us those items that are exhibiting
differential item functioning. And then we've got a choice.
We can either chuck those items, if we've got other items
that are just as good, but don't exhibit that. We've got to get rid of them.
Or we can say, "You know what?
We're not going to get rid of it, because it's really informative item.
But now we know we need to interpret it.
We need to score it differently for
men and women." Still useful, but we got to score it differently, right?
By the way, you know, when you see tests for potential race bias in achievement
tests and IQ tests, what they're doing is differential item functioning.
They're looking to see, for two people
who are at the exact same underlying level of analytic reasoning, for example,
do they have different probabilities of getting this item correct or not?
Because of maybe different cultural backgrounds, right?
So, differential item functioning is very important.
Very, very -- isn't it really a terrific technique we've got?
Okay.
So, that was our quick little foray into some of the things that item response
theory has brought us, namely measures based on item banks, right?
And how they differ from your traditional off-the-shelf measures.
And some other really cool tools like being able to assess differential item functioning.
Okay?