So, I'm assistant professor at University of Pennsylvania.
But before that, I was actually here at NIH for several years at NIAID.
And so, I've done trials here at NIH and then I'm involved in clinical trials,
both on investigative team and also Data and Safety Monitoring Boards.
So, we'll talk a little bit about from my own personal experience
and also just from the general wisdom about data and safety monitoring.
And I do want to acknowledge many people who contributed to this presentation
over the years including Laura Lee Johnson, who I think has taught
a few other lectures here; Dennis Dixon; Michael Proschan, either formerly
or currently at NIH; and Susan Ellenberg at UPenn who used to be at the FDA.
Long list of experience adding to the few of these slides today.
So, I'm going to give an introduction to this topic,
and I'm going to just pretty much almost immediately launch,
you know, a couple of case studies or couple of quick examples
because I think you'll really understand the what, the why,
and the how by seeing examples for this particular topic.
And then we'll cover the basics of Data and Safety Monitoring Boards,
how you form them, what they monitor and the basic principles,
when you need them, and when you may not need them.
And I'll finish with just some specific examples
of things that are monitored by the DSMB,
so going from the broad to the concrete through this.
So, the fundamental principle guiding Data and Safety Monitoring Boards and overall
concept to data and safety monitoring is, you imagine
you're doing a clinical trial of a treatment.
It's novel, you don't know if it works, it might have some risks.
You don't want to wait until the end of the trial to figure out
whether or not during the trial
something fundamentally went wrong with the conduct of the trial,
you know, essentially making results unusable.
Or that the new drug had unexpected harms or that there was convincing evidence
very early on that you could have stopped and giving this treatment
to people much earlier than you did, and trials could go on for years.
So, ultimately, the idea is that clinical trials, it's an endeavor
that's very important but it's not without risks
and we want as many safety measures in place.
And so, Data and Safety Monitoring Boards exist for that caution.
Another fundamental concept
is once you start, say, looking at data,
interim looks during your trial, you can't just do that
without any restrictions if you want a valid study at the end.
And hopefully you've gone through some of these lectures
on hypothesis testing, on randomization, and maintaining the blind.
Everything we know about clinical research really comes through ahead
in the concept of Data and Safety Monitoring Boards.
So, one of the things that we learned from hypothesis testing,
which I believe was already on your schedule, is that if you look
at the data multiple times, you increase the risk of a false positive.
Right?
So, you know,
you flip a coin with a 5 percent chance, eventually it's going to come up heads.
And so, if you think you need to look at data in interim looks
because you want to make sure that if you need to stop the trial,
you stop it, you need to have a statistical plan in place so that you do
that; and at the end still maintain the statistical validity in your analysis.
That you want to maintain that acceptable rate, say, 0.05
or whatever you decided rate is, it could be 0.01 for certain trials,
that you maintain that no matter how many times you needed
to look at the data in the interim, and that's possible.
But you need a data and safety monitoring plan essentially.
Once you have interim looks planned, even by an independent body,
you need a statistical plan that needs to be part
of your protocol, generally preferable from the outset, from the design.
And also realize if the goal is to be looking at data
in interim looks, then the Data and Safety Monitoring Board is going to need people
that can consume that data, that they're competent in understanding the difference
between unlucky streaks in the data or statistically convincing trends.
And that they can understand all the complexities of clinical research
including risk benefits and anything else that you might need to consider in order
to make the decision or whether the patients are best
served by continuing the trial or stopping the trial.
And finally, if you have a group of people now charged perhaps at looking at data
were the only ones that may be unblinded, you need them to be objective.
Right?
If they're going to be having this
responsibility of deciding is it safest for the patients?
Is it best for the -- to continue or to stop?
They can't have any vested interest in the trial continuing, right,
or perhaps some relationship
with the study investigators or investment in the company or -- so, suddenly,
you realize based on the responsibilities of this board that initially
we started with, we just want to protect patients
quite quickly becomes a very complex endeavor where you need to consider
very carefully the people on the board and the responsibilities in order
for them to be able to fulfill their -- for their responsibilities.
So, this definition comes, I think, after many years of trials,
you know, from their infancy just people doing them
to having quite structured guidance from various bodies on
when and how you create the Data and Safety Monitoring Board.
So, the basic definition is that it's a group of independent experts
that reviews the ongoing conduct and involving data,
so emerging data of a clinical trial, to ensure continuing patient safety
as well as the validity and scientific merit of the trial.
So, not just that the data is not showing any early signals
but the trial that is being conducted is the trial
that was planned, that the protocol is being faithfully followed.
That's another sort of part that perhaps I haven't initially introduced
but I'm using now that essentially that this trial
is as intended and is ethical to continue.
And this is just a brief note on terminology
because if you start reading, right away, you'll start seeing that there are DSMCs
and DSMBs, and mostly that's a cultural thing.
Europe and the FDA tend to use committees.
DSMCs and NIH uses the Bs, Data and Safety Monitoring Boards.
If you're just early on being confused, if these were different types of bodies
or they have the same boy, but it's just generally general preference.
But what's interesting is these groups are entrenched.
And sometimes when they collaborate, they argue over what term to use.
In any case, let's get into our case studies.
So, the first example, I think both of these studies are --
that I'm going to talk about are historical but
incredibly exemplary in the sense of being the poster children
for why you need a DSMB and the benefits of having a DSMB.
So, the first one is ACTG 076.
So, that the Aids Clinical Trial Group Protocol 076.
And this is from the early days of HIV and zidovudine was just shown,
just recently shown at that time to be able to slow progression of HIV.
It was the first therapeutic breakthrough and the first treatment approved
by the U.S. government to treat HIV.
So, the trial on the docket that was going to be done
was to see whether or not there was safety and efficacy in using zidovudine
or AZT in preventing transmission of HIV from infected mothers to their infants.
And in particular, these women couldn't have advanced disease.
And this trial, I give the results papers which is the reference there.
So, at the time, NIAID had a group that basically ran
all the trials that they were -- the NIH was launching -- funding for HIV;
and they had a DSMB that was monitoring several of these trials.
And so, there were various ethical considerations
from the get-go they had to think about in designing this trial.
And the first is, if partway through, you know, they already have evidence
that this works, that AZT works in general at slowing the advancement of HIV.
There was a good chance that that could reduce infectivity of the mother.
And so,
if you want to stop as early as possible, once you have convincing evidence, right,
that you can prevent babies from being infected,
this is a very great diagnosis in the early '90s.
So, that's the first ethical principle
that needed to be in the design of this trial.
And the second was, there was a lot of pressure
just to give drugs immediately,
but you didn't want to put out a drug that didn't actually work.
And particularly for pregnant mothers and unborn infants and newborn infants,
you know, there are some extra risks there that we don't quite
fully understand when we have new drugs given to pregnant mothers.
So, we wanted a convincing evidence there was overall benefit and not undue
harm.
So, you know, that becomes a statistical analysis, and it becomes
interim analyses that a body can look at during the trial.
Basically, instead of waiting until the end,
they're going to look at different times during the trial at these issues of,
are there -- what are the adverse events, and what are the relative
event rates of infection on those on a placebo versus an active form.
So, it was powered to have an 80 percent
chance to detect the 33 percent reduction of HIV transmission
through the first 78 weeks of a baby's life.
And so, that was relatively to a projected rate of 30
percent in what they knew about transmission at that time.
So, the target sample size was 748 and accrual began in 1991.
So, you might remember from the survival lecture
at the end of last year, when you have time to advance time
to transmission of HIV babies getting infected, the sample size
is the number of -- typically, the number of observed
events that you need in order to determine the power.
And so, you have the idea
of how many you need to accrue to get the total number of events.
So, accrual began in 1991,
and the projected accrual was to take at least five years.
And actually -- and then I'm reading the slide
-- I'm remembering the 748 is the number of mother-baby pairs
you needed for the expected event rate to have 80 percent power.
So, already, you have to do a little of a guessing game
for sample size which is another reason why you might have a DSMB look partway
through to make sure you're on track in terms of your projections
in having adequate power.
So, the projected accrual
was to take 5 years, and they were planning for 15 percent drop out.
So, the monitoring plan in place was to meet twice a year just to look at safety.
Automatically, that was happening.
And in addition, they were going to do efficacy reviews.
And that was after each third of the projected infant infections.
And then they projected that the first efficacy review should take place in order
to capture about a third of the events around February of 1994.
And so, then that would happen, you know, this is a little bit of the nitty-gritty.
You can't just say, okay, we'll use data up through January 31, right?
So, they basically locked the data for all mothers enrolled up
to December of 1993 and then they would look at the events up to that point.
So, this is the data analysis at the first look.
And so, what we're looking at here is sort of the opposite of a Kaplan-Meier curve.
So, normally, survival curves, when you look at the even rates,
nobody has the events, say, mortality at the beginning of the trial
and then they decrease
and then you can compare the mortality rates over time with percent of life.
Here, what we have is, you know, some babies might be born basically
instantly infected, and so you'll see the event rate, let's get this going
here, is not quite at zero, maybe it's a little bit higher.
So, babies are essentially born already.
There seem to be an apparent difference.
And as time went on, zidovudine arm
only 80 percent of the babies infected, whereas placebo
was projecting 25 percent by the time a baby got to 72 weeks.
That's, you know,
more than fourfold difference, and the P-value was really convincing.
And at this point, if you look at the total number enrolled, it was about 360.
And I think basically,
they had about half of the planned events that they were expecting.
And so, quite a lot of information had accrued,
events and accrual had happened a little faster than expected.
And so, this was overwhelming evidence, and certainly met some conditions
where you might think, okay, we definitely need to stop.
We have evidence of benefits.
But there were some discussions.
And so, the first discussion was that they knew
there were 46 babies that sort of in limbo.
They didn't know their HIV status, and they didn't want to stop for efficacy
and then have those babies evaluated and realize retrospectively
the boundary wasn't actually met for stopping.
And so, because there is more evidence that's needed, not the usual 1.96,
and we'll get into that in a little bit, when you're stopping early.
And so, there was some statistical analysis to look at,
reasonable trends of what could happen to the 46
and what could happen to the P-value depending on different outcomes.
That was considered.
They looked very carefully at safety.
And that type of analysis that I'm talking about just to explain
all the words on the slide is what we call conditional power.
So, conditional power is essentially, given the data up to this point, what's
the probability that you will wind up with a positive result at the end.
And so, that is often a statistical analysis
that we presented to the DSMB when pertinent
when you're starting to think about is it time to stop?
So, after
careful review of that conditional power of the safety
of how much data was accrued, which was quite a lot,
the DSMB did recommend there was overwhelming and convincing
statistical evidence that there was benefit to this treatment.
And they recommended a trial stop.
And a key term here is that DSMB is a board
that's an independent body, they're not the sponsor,
they're not the investigators, they give advice, guidance.
But in the end, it's the trial leaders and ultimately the sponsor
that will make the decision, typically, on whether the trial should go or no-go.
And in this case, as in many cases, they follow
the advice of the DSMB, the recommendation, and they stopped.
And instantly,
the zidovudine was provided in the control group and guidance in the U.S.
and across the world was changed to allow distributing it for prevention.
So, that was a great success.
And the benefit of having the DSMB was that
they didn't have to wait for the trial's end.
They got to stop, actually, a couple years earlier and get that treatment out.
So, many babies were able to escape infection
because of that earlier stopping and distribution of the treatment.
So, here is a different kind of example
and exemplary in another way, and it's CAST.
There was actually many, many lessons that can be learned from
this trial as is typical when you do a large trial.
So, this was the Cardiac Arrhythmic
Suppression Trial, and it designed to evaluate hypothesis
of whether suppressing ventricular arrhythmias in patients with recent MI
or heart attacks, whether or not that would reduce sudden death,
that was the primary endpoint, or overall mortality as secondary endpoint.
And there were several active drugs that were known to be successful
in suppressing arrhythmias.
And so, this trial, well, the three of them, encainide, flecainide,
and moricizine -- I'm a little more shaky of how I pronounce that one.
You can see it on the slide.
So, we had three actives and three matching placebos.
Essentially, a large complex trial, and they were randomized 4,000 patients.
Partially because they're studying several treatments at once, also
partially because, you know, they need for that event rate.
Those are the number of patients you randomized and then not
all of them will wind up with those events.
And as we learned, it's the number of events that's your true sample size.
So, they needed to randomize 4,000 for 90 percent power in the end
to detect the reduction of sudden death with that 0.05 one-tailed
significance level. So, that was the original design.
And I don't know if any alarm bells are going off.
Well, Laura Lee jumped up and down about this.
It was Laura Lee Johnson who did the hypothesis testing? Yes.
But -- so, the DSMB met for the first time,
so this cardiovascular trial was launched in 1987.
The first meeting of the DSMB,
this is a blinded trial, large trial, death as an endpoint.
It's going to have an ethical monitoring board as independent.
You know, Phase 3 basically required to have a DSMB.
Their first review was before even the study started,
and that's often the case because they want to look at the protocol,
they want to make sure that this monitoring board is okay with the protocol
as is and agree on their charter; what are they going to monito,
is everything won on the same page about what's appropriate to monitor and how?
And as a result of that meeting, they made a recommendation
actually to change the primary analysis of the trial.
And you -- actually 1987 now, it's going on 30 years ago, and statistics
was getting, in clinical
trials, was still being formalized in terms of our best practices.
And perhaps, this is the time when it basically became convincing.
And certainly, there was a forward-thinking statistician at DSMB
that this was no longer acceptable to use one-sided 0.05 level tests.
So, the initial argument is we know the smaller the alpha,
the more difficult it is.
The more N you need to guarantee that level of a false-positive rate.
And someone might argue, why would I want to spend alpha to look in
either a strong positive direction or strong negative direction to decide
whether or not there's a treatment difference
when I'm only interested in the strong positive direction of benefit.
I want all my alpha going to the 0.05
and then I can use 1.64 and not 1.96.
But the opposing argument that has become the dominant one is that you
don't want a test, whether it's one-tailed or two-tailed, to influence
level of evidence you're going to use to declare benefits.
Right?
So, whether you can do a one-tailed test, but it shouldn't be easier to pass
that test than a two-tailed test in terms of deciding
what is the benchmark, what the critical value, in this case
1.96, to declare significance in terms of a difference
between two groups, say, if you're doing one look at 0.05 level.
And another argument is that, you know, you've -- almost rare
that I never see -- I've never seen a one-sided confidence interval.
When you do a confidence interval, you do the uncertainty plus or minus,
say, 1.96 standard deviations around the mean.
If you do your test as a one-sided test and a confidence interval,
the two-sided confidence interval, you could wind up doing this test,
declaring a difference, having a confidence interval,
and having the null inside of that confidence interval.
So, it's -- some might view a few more on the clinical side,
the second esoteric argument in one of the statisticians on board with this.
Well, they are now.
But at the time, perhaps, it was an emerging consensus.
And it was interesting that at the very first meeting,
essentially the principal analysis was changed.
Well, they made the recommendation and then the study team and sponsor
were convinced by the arguments, and they did change it.
And it only reduced the power a little bit.
So, they were already going for 90 percent power, which quite convincing.
So, they were still at 85 percent power when they did this change to be two-sided.
So, I think in the end, everyone was satisfied with that change in
the protocol.
And so, the lesson learned here is it's very important to have the first meeting
before the study starts.
It was -- ultimately, the DSMB is charged with a pretty delicate sense
of responsibility,
actually a very important role, which is they are going to be looking at data
no one else one has and be making recommendations that have
a strong influence on the future of this trial.
Everyone needs to be on the same page scientifically, believe in the trial,
believe in validity from the start, and then be able to trust each other
and get along through what could be potentially difficult times
in terms of unexpected challenges that make this trial more difficult
to complete than anticipated.
So, you want that first meeting, so everyone is
on the same page about roles and responsibilities and the science.
Another lesson learned was not only should you be on the same page about,
you know, the design but the monitoring plan should be in place.
Because what happened at CAST, which I think was just one of those,
every sixth month just, you know, right -- basically,
there are two kinds of meetings the DSMB has.
One is just regularly scheduled.
We'll keep an eye on safety,
and then you'll do these interim looks at data as you go.
It's not expecting,
you know, real efficacy trends until a lot of data has accumulated.
But in this trial, with only 5 percent of the expected information, total
-- you know, 5
percent of the total number of events that they needed for that power,
they actually already had a difference apparent between the treatment arms.
Now, as is typical in a DSMB report, once you start looking at efficacy,
the report is going to just have arm differences,
and you might have them labeled Arm 1 and Arm 2.
Because you never know, you want to be -- at the preponderance of caution,
you want that report to be masked in case it gets into the wrong
hands, mailed to the wrong person accidentally, left on a desk.
But then the DSMB would have the potential to say, "Okay,
tell me which arm is which because I want to make a decision."
And others will always have that, which is which, say, in a separate
envelope that they open up at the start of every meeting.
In this particular trial as was done at this time,
more commonly than it is now, they chose to remain blinded until,
you know, perhaps they decided there was a reason to look.
And so, they looked.
All they could see was there was a treatment difference.
At the time they hadn't had stopping boundaries in place, perhaps because
it was so early and they were still gelling the analysis plans.
And so, in the end
it was probably fine because even though there was a convincing
-- when they did the test, in this case the logrank test
because they're looking at survival curves between an active
and inactive or, say, placebo, the logrank looked like it was succeeding.
It was 3.43 for treatment difference, and the boundary that they were eventually
applying retrospectively looking back, it seemed that they had exceeded it.
But it's that issue if there were pending events,
and actually when they were adjudicated and added
in, it actually would have flipped it, so that would have been less convincing.
So, in the end, even if they had the boundary in place,
even if they had looked to see which arm is which, they may not have stopped.
But it was instantly alarming.
But they chose to remain blinded.
And at the next look, and there's a reference
for this trial that people talk endlessly about for many reasons.
At the next look, there were 48 events.
So, at the first look, it was roughly 22.
Now, there were 48 events.
And actually, looking at event rates and other reasons,
they realized that they would only expect 300 and not the 400
and some they'd originally thought
at the end of this trial, which meant the 48 was a little bit
higher of a percent in terms of total number of events.
And when they looked at the boundary, they saw that -- so,
the logrank was actually still at 3.22, and it was well
over the boundary for stopping, plus or minus 2.98.
And so, the DSMB requested to be unblinded to find out which arm is which.
And so much to their surprise, it was the placebo
that was doing the best.
And it was active arms that were having harm.
And all along, everyone had assumed it was benefit
that just they were waiting for it to be convincing.
And this becomes the compelling argument for why most people argue,
you should not say, well, to remain objective, I don't want to know
which arm is which until I really think there's a difference.
And the reason is you're probably not going to make a symmetric,
even decision in terms of how much evidence
you need to convince the medical community to adopt this new therapy
which might have harm as well as benefits versus evidence that, you know what, I'm
80 percent confident that, you know, we might be harming people
but I think we should wait to 90 percent confident because it's kind of early.
So, it's really not a symmetric operation.
And I think having seen this over and over overtime, people have just --
a lot of think tanks that make recommendations on how to do these things,
people who have served on these bodies many times in their careers
make an overwhelming recommendation that you -- the DSMB is not masked.
You know, the trial is double blinded, the investigators and patients.
It is not triple blinded.
We do not blind the DSMB. We want the DSMB
to have every bit of information possible to make the decisions.
And this is ultimately why
because you cannot but sort of believe in a new treatment.
And in fact, they were so convinced that something had to be wrong
that they investigated the -- they got random bottles,
they assumed
maybe the place where they were labeling the bottles had made a mistake
and that the placebo was the active and the active was the placebo,
so they started testing chemically bottled medications that were had those labels.
They look for imbalances and key variables and the placebo and the active arms
to see maybe the differences were because these folks
weren't actually having the same risk factors.
And then they continued to also look to see
if there were certain subgroups that maybe were being harmed versus others.
Again, thinking, well, of that subgroup was out of balance, that might have driven
this treatment difference.
And they also looked at the secondary endpoint,
which is overall mortality, to see if it was consistent.
And finally, when they were not -- they realized it was in fact the placebo
that was doing better, they wanted to see whether certain arrhythmia drugs
out of the three they were studying were actually the problem.
And it turned out early on that there were two
out of the three active drugs, encainide and flecainide,
luckily the two I can pronounce, that were driving the difference.
So, each of the three actives had a matched control.
And so, the overall score was 35
events on the active, 13 on the placebo.
And when they reduced down to those -- on the encainide
and flecainide versus placebo, it was 33 versus 9.
So, what had just been CAST became divided up into CAST
1, which was the trial with all three treatments,
and then at the end of this DSMB, they stopped the encainide and flecainide arms,
but they continue the third active placebo because there was evidence.
Certainly, there was no reason to stop those arms.
And so, the was the benefit of having sort of the structure
of multiple matched placebos that they continued one question,
even though they had to stop the other two.
So, other lessons learned here is that not only should
your monitoring plan be in place or DSMB be in place
all before your study starts, but your data management system,
the data entry, all those people in charge of monitoring your data, entering
data, organizing your data should be on board on day one.
And that data system has been built and tested
because you need that data at that first meeting if it's available.
Because very early on trends can become apparent, certainly for harm.
And also, it turns out for benefit, say, in the AZT trial.
And other principles are, you know, you need to be organized,
you need to have these meetings on the books
because there could be many reasons why you need to act quickly,
you know, to start your monitoring right away and also
you need a board that's going to be able to think about and investigators
contingency plans about what to do once you have this this information.
So, people need to be ready to receive the DSMB recommendations as well.
And I think -- as I mentioned earlier, I think probably went over most of this,
is that the ability of the DSMB to do their job can be compromised
if they remain masked.
And I
think initially, the idea of objectivity seemed to be the overwhelming principle
of why we would keep the DSMB mask unless, you know, and perhaps
always give the -- and always give them the prerogative to ask to be unblinded.
But I think it was, you know, examples like this, that in general,
the recommendation is that you keep DSMB body unblinded.
The report may stay masked because you never know
who could intercept it, but you would always have
that DSMB have the ability to see which arm is which.
And a great article, and it's just an editorial,
it's only about two pages, that really summarizes both sides,
I think, but it's definitely by the name of this article,
Masked Monitoring in Clinical Trials--Blind Stupidity?
Obviously coming down on one side of the debate here, and certainly
a good way to augment reading if you're interested in this topic.
Well,
they do present the opposing view, at least briefly.
Okay, so hopefully, those two examples, you know, sort of have
given you some ideas of why you need Data and Safety Monitoring Board.
You know, essentially, you know, it summarizes examples, you need to able
to identify any safety problem rapidly, you need to be able to identify
logistical problems if there are some, so the board can recommend amendments
or make sure those come to light such as, you know, if the accrual is so slow,
they'll never get full enrollment by the end of the grant period.
That could be a problem, so maybe you need to add sites.
So, they'll, you know, be actually evaluating the feasibility of the trial,
not just thing like efficacy or safety.
And ultimately, they want to make sure if the trial objectives
have been met early then the trial should be terminated early.