Statistics, you guys have heard a lot about.
And so, I'm going to focus less on the methods in stats
and a little bit more on kind of the practical implications of those statistical choices.
And so, again, these are all detailed in your protocol
so that you don't get to the end of the study and go, "What did we want to look at?
What was it that we were going to analyze?
We had this great idea." And then you go back to your application
and you're like, "Yeah, we described that in two paragraphs
because that was all the space we had." So, this is where you have the opportunity.
Lay it all out, your analytic plan can be done,
so that as soon as the data is clean, your biostatistician can run it
because you already know what you're going to do.
So, it saves you a lot of time in the end if you have this
all spelled out ahead of time.
And more importantly, what we're seeing in the open science trends
and for rigor and reproducibility, is that people are having to put their a priori
hypothesis and in some cases, analytic plans out there
and -- before they even start enrolling participants.
And so, journals are now comparing back to, "Well, what did you say
your primary outcome was going to be and is that actually what you're reporting in
your primary paper here?" So, it's important to get those things specified and detailed.
I mentioned before that when I think
I was talking about blood pressure and how you might collect that.
So, this is an area of your protocol where you define really clear
how you're defining your outcomes.
So, for cancer studies, you might have an appointed panel of experts
that actually review the pathology slides to determine whether or not a person
actually has cancer
or not or meets the definition of nonalcoholic steatohepatitis for a liver study.
You may actually have a group of individuals who review those blinded
and make a determination as to whether they think that disease is present
or not in the sample that they receive.
They might see an imaging scan
and make a determination as to whether or not -- you know, what is the size tumor?
Has it shrunk or not?
And so, they actually give the measurements and those kinds of things.
So, you want to define how -- for your study, you're going to define your outcome.
And I really do encourage people
to use the standard definitions in the field, the agreed upon either
diagnostic criteria or methods that have been published by a given field.
And so, one way to find those is to look for other large clinical trials
that have been done in a particular area and see what definitions they have used.
The clinicaltrials.gov resource that lists all of the trials that are ongoing
is a great resource to be able to look at inclusion and exclusion criteria
for other studies, so that you can see that you're enrolling the same kinds of patients
as other trials are.
And also, the definitions of their outcomes so that you can use similar outcome
definitions definitions in your trial and that you're not creating something de novo
and totally new that you've come up with that would be difficult to include
in a meta-analysis later because you didn't use
an agreed upon definition that the rest of the field is using.
So, for example, the studies that I have listed here are the Women's Health
Initiative, the Systolic Hypertension in the Elderly Program which has definitions
for how do you define hypertension in older adults,
the Studies of Left Ventricular Dysfunction, the SOLVD study.
They have very clear definitions of how they've identified what is an
MI, what is a myocardial infarction, what is a heart attack.
And so, those are really good resources to look at those very large trials
that have spent a lot of time determining how they're going to make
those determinations as to what the definitions are of the primary outcomes.
One of the things that can be a challenge is specifying
not only what data are you going to collect, but in what units are you going to collect it.
I used to show a slide and I don't think it's in this slide deck
anymore, where people had collected height
and weight in different measures, and some of them were in --
weight was in kilograms and sometimes it was in pounds,
that you had different staff writing it down in different ways.
And if you tried to merge that data, you get really skewed information
and not really know how to interpret the summary data.
So, case report forms are what we typically collect the data on,
either electronically -- in electronic case report
forms, or in paper copies which we're seeing less of now.
It's really important
that they include the units as to how they're supposed to be measured.
And if you're collecting it electronically, you can actually create limits in your system
so that it won't allow you to put in, you know, an adult weight at ten pounds.
That it just wouldn't allow you.
You would have ranges as to what's acceptable.
So, those are our elements that are very important,
how you're measuring it.
We do occasionally get investigators that wanted to rely on self-report
for height and weight, which, if that's not a primary outcome,
and it's just kind of a demographics information that you're collecting, it's fine.
But if you're planning to use BMI and calculate BMI from that information
and use it as a confounding variable and to adjust for in your analysis,
you probably need to get accurate and measure height and weight.
And certainly, if you're ever going to share that data for secondary data analysis,
it would be important to label it as to whether it's measured height and weight,
or whether it's self-reported height and weight so that the data couldn't be
inappropriately used thinking that it was actually measured as opposed to self-report.
And, you know,
the measurement piece of this is -- the reproducibility thing comes up again.
Yeah? Is there a question?
Okay. Making sure that you are collecting
the measure in the way that other people do it as well.
So, I gave you the example of blood pressure.
So, for most studies its done sitting.
It's done after a period of rest when the person is not talking, is not eating.
They get quite frustrated because they don't like to sit for -- typically, about
five minutes and people's attention span, that is forever for people at this time.
But that's the typical.
If you need to do something for your study in a different way,
if you actually want to get a reading on blood pressure when someone's exercising,
it's really important that you label your variables
that way and make it very clear as to well, how much exercise do they need to do
and at what heart rate are you trying to get them to, and,
you know, what blood pressure would you try to get for that?
So, detailing all these aspects in the protocol is really important
so your staff can collect the data that you want it collected.
So, you've likely heard some issues around study instrument surveys and
how to think about using them, the importance of making sure that they're actually reliable
and valid the way that you're using them, and that they're actually sensitive to change.
So, there are some measures, particularly
in the psychology literature which we talk about as state or trait.
So, state is the current state that you're in
and you would expect that to fluctuate from day-to-day, time to time.
How anxious are you giving a talk versus how anxious are you relaxing
and getting a massage?
Likely very different, depending on who you are.
One may make you much more anxious than the other depending on who you are.
Whereas trait variables are thought to be quite static and not change.
If you choose an outcome measure and you're not very familiar
with the different instruments and you happen to choose something that is a trait
variable that's not going to change, then it's not a very good outcome
measure to see if your intervention is going to have an influence on that
if you really do think that trait variables and trait characteristics of individuals
are something that are static.
There certainly is a literature that's pushing to think
that there are some interventions that can shift peoples
trait tendencies in terms of their characteristics.
But for the most part,
people think about, say,
that something
that you would want to potentially measure as an outcome variable because it is variable
versus a trait that is something that is just a characteristic about the person.
So, those things
are really important to think about as you're selecting your outcome measures.
Over what time period do the things change, so some instruments are really
only developed to be used every six months or once a year.
Some of the stress characteristics -- the Perceived Stress Scale is one.
Some of the items on there are, "Have you moved in the last year?"
Well, that's not going to change very likely in six weeks.
So, that item is going to be pretty static for most people on that particular scale.
Have you lost your job?
Have you had a death in the family?
Those things are not something that on a day-to-day
basis shift very much for a given individual.
And so, a more daily stress measure might be something better if you're looking
for changes in whether someone's stress levels changed over time.
The other
thing that we're seeing more and more of with outcome measures,
most of the particularly survey instruments were developed as pen and paper instruments.
And we see them more and more being shifted to being collected online,
or being collected through EMA, Ecological Momentary Assessment.
So, they ping you with a text and ask you a question or two in real time
wherever you are to find out currently where you're at, how do you feel.
And if the instrument wasn't validated to be used that way, and to be assessed that way,
then it really needs to be validated to be used that way
because there could be confusion around the language.
There could be a lot of problems with how reliable it is
and whether you get the same response every time you ask the question.
And so, it's not that it can't be done, it's that someone -- before you use it
as your primary outcome measure,
make sure that you're choosing an instrument that has been validated
and is found to be reliable in the way that you're actually using it.
And this comes
down to kind of is it a good measure for you, for your study?
And as this says at the end here, there may not be published data on it,
but that might be an opportunity for you to work with a psychometrician
to do the validity study on delivering it in a different way.
We need more valid and reliable measures through a variety of different
delivery methods because patients and participants
and research studies get very frustrated filling out hours and hours of forms.
And you've likely heard some about the computer adaptive testing
that is available through some of these instruments now where
an individual takes an online -- does a set of questions
and it calibrates to the individual, the way the GRE is done this way.
So, it hones you into your score based on what your responses
are rather than asking everyone the exact same set of questions.
It's really customized to the individual who's responding to the questions.
It can dramatically shorten the number of items that an individual has
to answer to respond that way, but it's very foreign to an investigator.
And I think they're getting more comfortable to this, but to know that not everyone answers
the same questions is something that's very different than a
typical standard set of a 27-item questionnaire for example.
So, as I mentioned, these are the elements that you certainly want to include in your
protocol in terms of the general design issues, your treatment assignment.
The randomization procedure
is something that's really important and we're starting to see, particularly
at my center, more and more interventions that are delivered as group interventions.
And it's very important to specify whether you are individually
taking an individual that decides to participate
and there's a random chance as to them participating in Class A versus Class
B; or whether you've created a random allocation of classes and depending on
when that participant happens to call, they get the next class that's available.
And as you can imagine,
if that randomization allocation sequence becomes known to study staff
or to the people enrolling participants, it may influence who they decide to participate.
So, if Treatment A is a surgical procedure and Treatment B is a watch and wait,
they may hold off -- I mean, this is an extreme example.
This is not likely to actually happen in a trial.
But there certainly is published literature that if the staff is aware,
they may even unconsciously not refer people to a study
if they know that the next treatment allocation is for a watch and wait
if they think that patient really needs treatment.
They may actually not even refer patients.
Whereas if they know the next treatment allocation is to the actual intervention,
they're likely to refer a lot more patients and potentially much more severe patients
that they think would respond or do well with that particular treatment.
So, the blinding of the delivery and the sequence of class
delivery in particular is something that's really important
if you're doing a random class allocation, which we're starting to see more and more.
And the same is true
with other types of cluster designs and -- that need to be taken into account.
The other issue that comes up that I don't think gets enough
attention is the fact that when you randomize the participants in a study
is really important in terms of your study flow.
You know, one of the things I'm going to talk about
and I'm sure one of the terms you've already heard is intention to treat analysis,
which is everyone randomized gets analyzed.
And because of that, you want to shorten the time and the wait
period between when the randomization is done and when the participant actually starts
the intervention.
So, I've mentioned a few times that we often have these class delivered interventions.
Mindfulness-based stress reduction is one of them,
but this could be true for a group-based cognitive behavioral therapy,
or other group-based visits for diabetes management.
And if you have to wait until you get 15 or 20 participants
before you can start the classes, if the first time someone calls you
and you determine that they're eligible, you decide to randomize them and you say, "Okay.
You've been randomized to this class and it happens on Tuesday night.
And our first class will be on April 25th." And then you wait
and you contact people, you're going to lose a lot of people.
And so, you run the risk of having a lot of people drop out of your study
after they've already been randomized and you now have to include them in your analysis.
If instead you screen people and you let them know, "Okay.
Our classes will be starting around the middle or the end of April.
We'll call you again at the middle or the end of April
to make sure you still want to participate
and confirm some of the characteristics, that nothing has changed,
and you're still eligible for the study.
The study is planned -- your first class is April 25th,"
and you let them know the class will either be on this night, or on this night.
"Please make sure your schedule is clear, so that we -- you could participate
in the class." You call them, ideally, the day before if you can.
You wait, really, as long as possible to be able to confirm that they're still eligible
and then tell them which class they're going to go to after you confirm
they can still attend the class because so many things come up.
"Oh, my parents -- you know, I have a sick child
that I've got to do all of these visits with," or "Oh, no.
You know, my neighbor fell and broke their hip and now I have to help with care for them,"
or "My grandchild needs aftercare and my daughter got a new job.
And so, I have to pick her up instead." So many things happen in life between the time
someone decides to participate and potentially when a class might start for the class.
And so, putting off that randomization
until the last possible moment that's feasible for the study to pull off
can save you tremendously in missing data in the long run for your analysis.
So, that's one of the things that I certainly wanted to mention tonight.
The other thing that I'll mention is that if you have interim analysis,
you include those and these studies and your protocol as well,
and hopefully you've had some information about,
you know, really only doing interim analysis,
and having stopping rules if it's absolutely necessary for your study,
if there's a real reason to want to assess for futility in the study early.
And certainly, your full analytic plan needs to be included as well.
I think you guys have heard a lot about sample sizes
and the importance of basing your sample size on your primary outcome
and the primary hypothesis that you're going to test,
and that you've got enough information in which to base that.
And there's many different ways to analyze data.
My take home message as an investigator and someone giving advice to investigators is
have a really good biostatistician that you work with
from the beginning all the way through your study.
And don't just hand them your data at the end because it's likely they won't be able
to do the analysis that you hoped they would be able to do.
You want to involve them from the very beginning to make sure you're
designing your study to actually answer your hypothesis,
and you're collecting the data that they need to look at the outcomes,
and control for any potential confounders that may be measured in your study.
Again, I'm not going to go into a lot of the details
although I will emphasize the point on the analytic plan, that you do need to have
a very clear process laid out as to how you're going to handle missing data,
what you're going to do about it, how your statistician --
are they going to try to impute based on other data?
What is the method that they're going to deal with the issue of missing data?
And how are they going to evaluate whether it's missing at random, or whether
it's actually informative missing-ness?
And that
you had differential drop out, so your wait list control
that you thought would be great -- well, nobody came back for their follow up visits.
And so, you have very little follow up data on your wait list comparison group.
It's -- it can be a definite problem in studies that have very different types
of interventions for participants and different engagement of the participants.
I mentioned the interim analysis and the biggest thing here is
you want to be able to analyze your data, so that it will have an impact.
And so, all of these elements of including these pieces in the protocol
ultimately get you to the place where you have good quality data that's been collected
consistently across all participants that you've included, only participants
who are eligible for your study, and that if everyone followed the protocol
all the way through, you have very little to no missing data.
So, you have a great quality data set on which to base your analytics
and then conclusions of your study.
The more missing data
you have, the more difficult it is to interpret the results of your study
and making it -- make really conclusive statements
about whether or not your intervention was beneficial or not.
This, I'll
mention briefly just because it's this concept of if they're randomized,
it is someone that you need to include in your intent to treat analysis.
I'll refer you to your biostatisticians
about whether or not it's wise to do modified intent to treat analysis.
There's a lot of differing opinion about that.
Certainly, we do see per protocol analysis to see -- and often times, these are explored
as different types of sensitivity analysis to see whether there's a huge impact
of whether or not people are compliant or adherent to their protocol
in terms of their benefit with the treatment itself.