1 00:00:06,272 --> 00:00:08,808 So, I'm assistant professor at University of Pennsylvania. 2 00:00:08,808 --> 00:00:13,179 But before that, I was actually here at NIH for several years at NIAID. 3 00:00:13,179 --> 00:00:17,884 And so, I've done trials here at NIH and then I'm involved in clinical trials, 4 00:00:17,884 --> 00:00:21,354 both on investigative team and also Data and Safety Monitoring Boards. 5 00:00:21,354 --> 00:00:25,125 So, we'll talk a little bit about from my own personal experience 6 00:00:25,125 --> 00:00:28,862 and also just from the general wisdom about data and safety monitoring. 7 00:00:31,398 --> 00:00:35,769 And I do want to acknowledge many people who contributed to this presentation 8 00:00:35,769 --> 00:00:39,839 over the years including Laura Lee Johnson, who I think has taught 9 00:00:39,839 --> 00:00:43,543 a few other lectures here; Dennis Dixon; Michael Proschan, either formerly 10 00:00:43,543 --> 00:00:48,948 or currently at NIH; and Susan Ellenberg at UPenn who used to be at the FDA. 11 00:00:48,948 --> 00:00:52,986 Long list of experience adding to the few of these slides today. 12 00:00:53,386 --> 00:00:56,723 So, I'm going to give an introduction to this topic, 13 00:00:56,723 --> 00:01:00,060 and I'm going to just pretty much almost immediately launch, 14 00:01:00,060 --> 00:01:04,030 you know, a couple of case studies or couple of quick examples 15 00:01:04,030 --> 00:01:07,367 because I think you'll really understand the what, the why, 16 00:01:07,367 --> 00:01:10,703 and the how by seeing examples for this particular topic. 17 00:01:10,703 --> 00:01:14,707 And then we'll cover the basics of Data and Safety Monitoring Boards, 18 00:01:14,707 --> 00:01:18,378 how you form them, what they monitor and the basic principles, 19 00:01:18,378 --> 00:01:22,415 when you need them, and when you may not need them. 20 00:01:22,415 --> 00:01:25,485 And I'll finish with just some specific examples 21 00:01:25,485 --> 00:01:28,588 of things that are monitored by the DSMB, 22 00:01:28,588 --> 00:01:32,625 so going from the broad to the concrete through this. 23 00:01:32,625 --> 00:01:37,464 So, the fundamental principle guiding Data and Safety Monitoring Boards and overall 24 00:01:37,464 --> 00:01:40,934 concept to data and safety monitoring is, you imagine 25 00:01:40,934 --> 00:01:44,037 you're doing a clinical trial of a treatment. 26 00:01:44,337 --> 00:01:48,074 It's novel, you don't know if it works, it might have some risks. 27 00:01:48,074 --> 00:01:52,145 You don't want to wait until the end of the trial to figure out 28 00:01:52,145 --> 00:01:53,880 whether or not during the trial 29 00:01:53,880 --> 00:01:56,749 something fundamentally went wrong with the conduct of the trial, 30 00:01:56,749 --> 00:01:58,485 you know, essentially making results unusable. 31 00:01:58,485 --> 00:02:02,555 Or that the new drug had unexpected harms or that there was convincing evidence 32 00:02:02,555 --> 00:02:06,025 very early on that you could have stopped and giving this treatment 33 00:02:06,025 --> 00:02:10,063 to people much earlier than you did, and trials could go on for years. 34 00:02:10,530 --> 00:02:13,766 So, ultimately, the idea is that clinical trials, it's an endeavor 35 00:02:13,766 --> 00:02:16,169 that's very important but it's not without risks 36 00:02:16,169 --> 00:02:18,838 and we want as many safety measures in place. 37 00:02:18,838 --> 00:02:22,108 And so, Data and Safety Monitoring Boards exist for that caution. 38 00:02:24,611 --> 00:02:25,678 Another fundamental concept 39 00:02:25,678 --> 00:02:29,549 is once you start, say, looking at data, 40 00:02:29,883 --> 00:02:34,754 interim looks during your trial, you can't just do that 41 00:02:34,754 --> 00:02:40,827 without any restrictions if you want a valid study at the end. 42 00:02:40,827 --> 00:02:45,365 And hopefully you've gone through some of these lectures 43 00:02:45,365 --> 00:02:49,903 on hypothesis testing, on randomization, and maintaining the blind. 44 00:02:49,903 --> 00:02:54,941 Everything we know about clinical research really comes through ahead 45 00:02:54,941 --> 00:02:59,479 in the concept of Data and Safety Monitoring Boards. 46 00:03:00,013 --> 00:03:03,116 So, one of the things that we learned from hypothesis testing, 47 00:03:03,116 --> 00:03:06,786 which I believe was already on your schedule, is that if you look 48 00:03:06,786 --> 00:03:10,456 at the data multiple times, you increase the risk of a false positive. 49 00:03:10,456 --> 00:03:10,723 Right? 50 00:03:10,723 --> 00:03:11,558 So, you know, 51 00:03:11,558 --> 00:03:16,229 you flip a coin with a 5 percent chance, eventually it's going to come up heads. 52 00:03:16,229 --> 00:03:20,300 And so, if you think you need to look at data in interim looks 53 00:03:20,300 --> 00:03:24,237 because you want to make sure that if you need to stop the trial, 54 00:03:24,237 --> 00:03:29,709 you stop it, you need to have a statistical plan in place so that you do 55 00:03:29,709 --> 00:03:33,846 that; and at the end still maintain the statistical validity in your analysis. 56 00:03:33,846 --> 00:03:37,050 That you want to maintain that acceptable rate, say, 0.05 57 00:03:37,050 --> 00:03:41,221 or whatever you decided rate is, it could be 0.01 for certain trials, 58 00:03:41,221 --> 00:03:44,724 that you maintain that no matter how many times you needed 59 00:03:44,724 --> 00:03:48,228 to look at the data in the interim, and that's possible. 60 00:03:48,228 --> 00:03:51,431 But you need a data and safety monitoring plan essentially. 61 00:03:51,431 --> 00:03:54,968 Once you have interim looks planned, even by an independent body, 62 00:03:54,968 --> 00:03:58,137 you need a statistical plan that needs to be part 63 00:03:58,137 --> 00:04:01,674 of your protocol, generally preferable from the outset, from the design. 64 00:04:01,674 --> 00:04:05,878 And also realize if the goal is to be looking at data 65 00:04:05,878 --> 00:04:11,150 in interim looks, then the Data and Safety Monitoring Board is going to need people 66 00:04:11,150 --> 00:04:15,355 that can consume that data, that they're competent in understanding the difference 67 00:04:15,355 --> 00:04:18,891 between unlucky streaks in the data or statistically convincing trends. 68 00:04:18,891 --> 00:04:22,729 And that they can understand all the complexities of clinical research 69 00:04:22,729 --> 00:04:27,667 including risk benefits and anything else that you might need to consider in order 70 00:04:27,667 --> 00:04:31,170 to make the decision or whether the patients are best 71 00:04:31,170 --> 00:04:34,507 served by continuing the trial or stopping the trial. 72 00:04:34,507 --> 00:04:40,313 And finally, if you have a group of people now charged perhaps at looking at data 73 00:04:40,313 --> 00:04:45,218 were the only ones that may be unblinded, you need them to be objective. 74 00:04:46,152 --> 00:04:46,519 Right? 75 00:04:46,519 --> 00:04:49,088 If they're going to be having this 76 00:04:49,088 --> 00:04:52,425 responsibility of deciding is it safest for the patients? 77 00:04:52,425 --> 00:04:56,462 Is it best for the -- to continue or to stop? 78 00:04:56,462 --> 00:05:00,500 They can't have any vested interest in the trial continuing, right, 79 00:05:00,500 --> 00:05:01,968 or perhaps some relationship 80 00:05:01,968 --> 00:05:06,773 with the study investigators or investment in the company or -- so, suddenly, 81 00:05:06,773 --> 00:05:10,810 you realize based on the responsibilities of this board that initially 82 00:05:10,810 --> 00:05:14,113 we started with, we just want to protect patients 83 00:05:14,113 --> 00:05:18,584 quite quickly becomes a very complex endeavor where you need to consider 84 00:05:18,584 --> 00:05:23,056 very carefully the people on the board and the responsibilities in order 85 00:05:23,056 --> 00:05:27,794 for them to be able to fulfill their -- for their responsibilities. 86 00:05:27,794 --> 00:05:31,964 So, this definition comes, I think, after many years of trials, 87 00:05:31,964 --> 00:05:35,301 you know, from their infancy just people doing them 88 00:05:35,301 --> 00:05:38,638 to having quite structured guidance from various bodies on 89 00:05:38,638 --> 00:05:42,742 when and how you create the Data and Safety Monitoring Board. 90 00:05:43,376 --> 00:05:47,647 So, the basic definition is that it's a group of independent experts 91 00:05:47,647 --> 00:05:50,483 that reviews the ongoing conduct and involving data, 92 00:05:50,483 --> 00:05:54,787 so emerging data of a clinical trial, to ensure continuing patient safety 93 00:05:54,787 --> 00:05:58,691 as well as the validity and scientific merit of the trial. 94 00:05:58,691 --> 00:06:02,962 So, not just that the data is not showing any early signals 95 00:06:02,962 --> 00:06:06,532 but the trial that is being conducted is the trial 96 00:06:06,532 --> 00:06:10,103 that was planned, that the protocol is being faithfully followed. 97 00:06:10,103 --> 00:06:14,173 That's another sort of part that perhaps I haven't initially introduced 98 00:06:14,173 --> 00:06:17,477 but I'm using now that essentially that this trial 99 00:06:17,477 --> 00:06:20,613 is as intended and is ethical to continue. 100 00:06:20,613 --> 00:06:24,150 And this is just a brief note on terminology 101 00:06:24,150 --> 00:06:29,322 because if you start reading, right away, you'll start seeing that there are DSMCs 102 00:06:29,322 --> 00:06:32,258 and DSMBs, and mostly that's a cultural thing. 103 00:06:32,992 --> 00:06:36,028 Europe and the FDA tend to use committees. 104 00:06:36,028 --> 00:06:40,166 DSMCs and NIH uses the Bs, Data and Safety Monitoring Boards. 105 00:06:40,166 --> 00:06:45,471 If you're just early on being confused, if these were different types of bodies 106 00:06:45,471 --> 00:06:50,009 or they have the same boy, but it's just generally general preference. 107 00:06:50,009 --> 00:06:53,045 But what's interesting is these groups are entrenched. 108 00:06:53,045 --> 00:06:57,583 And sometimes when they collaborate, they argue over what term to use. 109 00:06:59,585 --> 00:07:02,789 In any case, let's get into our case studies. 110 00:07:02,789 --> 00:07:07,026 So, the first example, I think both of these studies are -- 111 00:07:07,026 --> 00:07:10,229 that I'm going to talk about are historical but 112 00:07:10,229 --> 00:07:13,766 incredibly exemplary in the sense of being the poster children 113 00:07:13,766 --> 00:07:18,371 for why you need a DSMB and the benefits of having a DSMB. 114 00:07:18,371 --> 00:07:20,840 So, the first one is ACTG 076. 115 00:07:20,873 --> 00:07:24,210 So, that the Aids Clinical Trial Group Protocol 076. 116 00:07:24,210 --> 00:07:29,415 And this is from the early days of HIV and zidovudine was just shown, 117 00:07:29,415 --> 00:07:34,587 just recently shown at that time to be able to slow progression of HIV. 118 00:07:34,587 --> 00:07:38,691 It was the first therapeutic breakthrough and the first treatment approved 119 00:07:38,691 --> 00:07:41,294 by the U.S. government to treat HIV. 120 00:07:41,294 --> 00:07:45,731 So, the trial on the docket that was going to be done 121 00:07:45,731 --> 00:07:50,937 was to see whether or not there was safety and efficacy in using zidovudine 122 00:07:50,937 --> 00:07:55,741 or AZT in preventing transmission of HIV from infected mothers to their infants. 123 00:07:55,741 --> 00:07:59,579 And in particular, these women couldn't have advanced disease. 124 00:07:59,579 --> 00:08:05,284 And this trial, I give the results papers which is the reference there. 125 00:08:05,284 --> 00:08:10,156 So, at the time, NIAID had a group that basically ran 126 00:08:10,156 --> 00:08:16,529 all the trials that they were -- the NIH was launching -- funding for HIV; 127 00:08:16,529 --> 00:08:21,601 and they had a DSMB that was monitoring several of these trials. 128 00:08:22,635 --> 00:08:24,904 And so, there were various ethical considerations 129 00:08:24,904 --> 00:08:28,774 from the get-go they had to think about in designing this trial. 130 00:08:28,774 --> 00:08:32,979 And the first is, if partway through, you know, they already have evidence 131 00:08:32,979 --> 00:08:37,483 that this works, that AZT works in general at slowing the advancement of HIV. 132 00:08:37,483 --> 00:08:41,687 There was a good chance that that could reduce infectivity of the mother. 133 00:08:41,687 --> 00:08:42,321 And so, 134 00:08:42,321 --> 00:08:47,159 if you want to stop as early as possible, once you have convincing evidence, right, 135 00:08:47,493 --> 00:08:50,363 that you can prevent babies from being infected, 136 00:08:50,363 --> 00:08:53,900 this is a very great diagnosis in the early '90s. 137 00:08:53,900 --> 00:08:56,035 So, that's the first ethical principle 138 00:08:56,035 --> 00:08:59,772 that needed to be in the design of this trial. 139 00:08:59,772 --> 00:09:03,476 And the second was, there was a lot of pressure 140 00:09:04,076 --> 00:09:05,278 just to give drugs immediately, 141 00:09:05,278 --> 00:09:09,882 but you didn't want to put out a drug that didn't actually work. 142 00:09:09,882 --> 00:09:13,786 And particularly for pregnant mothers and unborn infants and newborn infants, 143 00:09:14,253 --> 00:09:18,591 you know, there are some extra risks there that we don't quite 144 00:09:18,591 --> 00:09:22,562 fully understand when we have new drugs given to pregnant mothers. 145 00:09:22,562 --> 00:09:27,300 So, we wanted a convincing evidence there was overall benefit and not undue 146 00:09:27,300 --> 00:09:27,667 harm. 147 00:09:27,667 --> 00:09:31,637 So, you know, that becomes a statistical analysis, and it becomes 148 00:09:31,637 --> 00:09:35,641 interim analyses that a body can look at during the trial. 149 00:09:35,641 --> 00:09:38,177 Basically, instead of waiting until the end, 150 00:09:38,177 --> 00:09:43,249 they're going to look at different times during the trial at these issues of, 151 00:09:43,249 --> 00:09:47,954 are there -- what are the adverse events, and what are the relative 152 00:09:47,954 --> 00:09:52,658 event rates of infection on those on a placebo versus an active form. 153 00:09:53,225 --> 00:09:56,629 So, it was powered to have an 80 percent 154 00:09:56,629 --> 00:10:00,433 chance to detect the 33 percent reduction of HIV transmission 155 00:10:00,433 --> 00:10:03,836 through the first 78 weeks of a baby's life. 156 00:10:03,836 --> 00:10:08,007 And so, that was relatively to a projected rate of 30 157 00:10:08,007 --> 00:10:11,978 percent in what they knew about transmission at that time. 158 00:10:11,978 --> 00:10:16,716 So, the target sample size was 748 and accrual began in 1991. 159 00:10:16,716 --> 00:10:19,752 So, you might remember from the survival lecture 160 00:10:19,752 --> 00:10:24,657 at the end of last year, when you have time to advance time 161 00:10:24,657 --> 00:10:28,461 to transmission of HIV babies getting infected, the sample size 162 00:10:28,461 --> 00:10:32,231 is the number of -- typically, the number of observed 163 00:10:32,231 --> 00:10:36,002 events that you need in order to determine the power. 164 00:10:37,103 --> 00:10:38,738 And so, you have the idea 165 00:10:38,738 --> 00:10:42,708 of how many you need to accrue to get the total number of events. 166 00:10:42,708 --> 00:10:44,210 So, accrual began in 1991, 167 00:10:44,210 --> 00:10:47,213 and the projected accrual was to take at least five years. 168 00:10:47,213 --> 00:10:49,682 And actually -- and then I'm reading the slide 169 00:10:49,682 --> 00:10:52,685 -- I'm remembering the 748 is the number of mother-baby pairs 170 00:10:52,685 --> 00:10:55,955 you needed for the expected event rate to have 80 percent power. 171 00:10:56,455 --> 00:10:59,892 So, already, you have to do a little of a guessing game 172 00:10:59,892 --> 00:11:04,163 for sample size which is another reason why you might have a DSMB look partway 173 00:11:04,163 --> 00:11:07,600 through to make sure you're on track in terms of your projections 174 00:11:07,600 --> 00:11:08,734 in having adequate power. 175 00:11:08,734 --> 00:11:09,902 So, the projected accrual 176 00:11:09,902 --> 00:11:13,906 was to take 5 years, and they were planning for 15 percent drop out. 177 00:11:13,906 --> 00:11:18,778 So, the monitoring plan in place was to meet twice a year just to look at safety. 178 00:11:18,778 --> 00:11:20,312 Automatically, that was happening. 179 00:11:20,312 --> 00:11:23,649 And in addition, they were going to do efficacy reviews. 180 00:11:23,649 --> 00:11:27,319 And that was after each third of the projected infant infections. 181 00:11:27,319 --> 00:11:31,991 And then they projected that the first efficacy review should take place in order 182 00:11:31,991 --> 00:11:35,995 to capture about a third of the events around February of 1994. 183 00:11:35,995 --> 00:11:41,367 And so, then that would happen, you know, this is a little bit of the nitty-gritty. 184 00:11:41,701 --> 00:11:46,472 You can't just say, okay, we'll use data up through January 31, right? 185 00:11:46,472 --> 00:11:50,509 So, they basically locked the data for all mothers enrolled up 186 00:11:50,509 --> 00:11:56,582 to December of 1993 and then they would look at the events up to that point. 187 00:11:56,582 --> 00:12:00,419 So, this is the data analysis at the first look. 188 00:12:00,419 --> 00:12:06,325 And so, what we're looking at here is sort of the opposite of a Kaplan-Meier curve. 189 00:12:06,325 --> 00:12:09,462 So, normally, survival curves, when you look at the even rates, 190 00:12:09,462 --> 00:12:12,865 nobody has the events, say, mortality at the beginning of the trial 191 00:12:12,865 --> 00:12:13,999 and then they decrease 192 00:12:13,999 --> 00:12:17,970 and then you can compare the mortality rates over time with percent of life. 193 00:12:17,970 --> 00:12:21,674 Here, what we have is, you know, some babies might be born basically 194 00:12:21,674 --> 00:12:25,377 instantly infected, and so you'll see the event rate, let's get this going 195 00:12:25,377 --> 00:12:28,814 here, is not quite at zero, maybe it's a little bit higher. 196 00:12:28,814 --> 00:12:31,417 So, babies are essentially born already. 197 00:12:31,417 --> 00:12:34,487 There seem to be an apparent difference. 198 00:12:34,487 --> 00:12:37,523 And as time went on, zidovudine arm 199 00:12:37,523 --> 00:12:41,427 only 80 percent of the babies infected, whereas placebo 200 00:12:41,427 --> 00:12:47,099 was projecting 25 percent by the time a baby got to 72 weeks. 201 00:12:47,099 --> 00:12:48,400 That's, you know, 202 00:12:48,400 --> 00:12:52,972 more than fourfold difference, and the P-value was really convincing. 203 00:12:52,972 --> 00:13:00,179 And at this point, if you look at the total number enrolled, it was about 360. 204 00:13:00,646 --> 00:13:02,047 And I think basically, 205 00:13:02,047 --> 00:13:06,285 they had about half of the planned events that they were expecting. 206 00:13:06,285 --> 00:13:09,455 And so, quite a lot of information had accrued, 207 00:13:09,455 --> 00:13:12,958 events and accrual had happened a little faster than expected. 208 00:13:12,958 --> 00:13:16,829 And so, this was overwhelming evidence, and certainly met some conditions 209 00:13:16,829 --> 00:13:20,366 where you might think, okay, we definitely need to stop. 210 00:13:20,366 --> 00:13:22,101 We have evidence of benefits. 211 00:13:22,101 --> 00:13:24,036 But there were some discussions. 212 00:13:24,036 --> 00:13:27,406 And so, the first discussion was that they knew 213 00:13:27,406 --> 00:13:30,543 there were 46 babies that sort of in limbo. 214 00:13:30,676 --> 00:13:35,581 They didn't know their HIV status, and they didn't want to stop for efficacy 215 00:13:35,581 --> 00:13:38,617 and then have those babies evaluated and realize retrospectively 216 00:13:38,851 --> 00:13:41,187 the boundary wasn't actually met for stopping. 217 00:13:41,187 --> 00:13:45,724 And so, because there is more evidence that's needed, not the usual 1.96, 218 00:13:45,724 --> 00:13:50,296 and we'll get into that in a little bit, when you're stopping early. 219 00:13:50,296 --> 00:13:53,799 And so, there was some statistical analysis to look at, 220 00:13:53,799 --> 00:13:56,936 reasonable trends of what could happen to the 46 221 00:13:56,936 --> 00:14:00,806 and what could happen to the P-value depending on different outcomes. 222 00:14:01,340 --> 00:14:02,208 That was considered. 223 00:14:02,208 --> 00:14:04,009 They looked very carefully at safety. 224 00:14:04,009 --> 00:14:07,546 And that type of analysis that I'm talking about just to explain 225 00:14:07,546 --> 00:14:11,116 all the words on the slide is what we call conditional power. 226 00:14:11,116 --> 00:14:14,954 So, conditional power is essentially, given the data up to this point, what's 227 00:14:14,954 --> 00:14:19,091 the probability that you will wind up with a positive result at the end. 228 00:14:19,091 --> 00:14:21,460 And so, that is often a statistical analysis 229 00:14:21,460 --> 00:14:23,829 that we presented to the DSMB when pertinent 230 00:14:23,829 --> 00:14:27,066 when you're starting to think about is it time to stop? 231 00:14:28,234 --> 00:14:29,335 So, after 232 00:14:29,335 --> 00:14:33,439 careful review of that conditional power of the safety 233 00:14:33,439 --> 00:14:38,611 of how much data was accrued, which was quite a lot, 234 00:14:38,611 --> 00:14:42,848 the DSMB did recommend there was overwhelming and convincing 235 00:14:42,848 --> 00:14:47,086 statistical evidence that there was benefit to this treatment. 236 00:14:47,086 --> 00:14:49,922 And they recommended a trial stop. 237 00:14:51,123 --> 00:14:54,760 And a key term here is that DSMB is a board 238 00:14:54,760 --> 00:14:57,363 that's an independent body, they're not the sponsor, 239 00:14:57,363 --> 00:14:59,999 they're not the investigators, they give advice, guidance. 240 00:14:59,999 --> 00:15:03,936 But in the end, it's the trial leaders and ultimately the sponsor 241 00:15:03,936 --> 00:15:08,540 that will make the decision, typically, on whether the trial should go or no-go. 242 00:15:08,540 --> 00:15:11,810 And in this case, as in many cases, they follow 243 00:15:11,810 --> 00:15:15,114 the advice of the DSMB, the recommendation, and they stopped. 244 00:15:15,114 --> 00:15:15,848 And instantly, 245 00:15:15,848 --> 00:15:20,686 the zidovudine was provided in the control group and guidance in the U.S. 246 00:15:20,686 --> 00:15:25,124 and across the world was changed to allow distributing it for prevention. 247 00:15:25,124 --> 00:15:27,359 So, that was a great success. 248 00:15:27,359 --> 00:15:30,696 And the benefit of having the DSMB was that 249 00:15:30,696 --> 00:15:34,033 they didn't have to wait for the trial's end. 250 00:15:34,033 --> 00:15:39,238 They got to stop, actually, a couple years earlier and get that treatment out. 251 00:15:39,305 --> 00:15:42,474 So, many babies were able to escape infection 252 00:15:42,474 --> 00:15:46,512 because of that earlier stopping and distribution of the treatment. 253 00:15:46,512 --> 00:15:49,949 So, here is a different kind of example 254 00:15:49,949 --> 00:15:53,085 and exemplary in another way, and it's CAST. 255 00:15:53,085 --> 00:15:57,423 There was actually many, many lessons that can be learned from 256 00:15:57,423 --> 00:16:01,760 this trial as is typical when you do a large trial. 257 00:16:01,760 --> 00:16:04,129 So, this was the Cardiac Arrhythmic 258 00:16:04,129 --> 00:16:07,266 Suppression Trial, and it designed to evaluate hypothesis 259 00:16:07,266 --> 00:16:11,203 of whether suppressing ventricular arrhythmias in patients with recent MI 260 00:16:11,203 --> 00:16:15,541 or heart attacks, whether or not that would reduce sudden death, 261 00:16:16,108 --> 00:16:19,979 that was the primary endpoint, or overall mortality as secondary endpoint. 262 00:16:19,979 --> 00:16:24,183 And there were several active drugs that were known to be successful 263 00:16:24,183 --> 00:16:25,250 in suppressing arrhythmias. 264 00:16:25,250 --> 00:16:29,121 And so, this trial, well, the three of them, encainide, flecainide, 265 00:16:29,121 --> 00:16:34,059 and moricizine -- I'm a little more shaky of how I pronounce that one. 266 00:16:34,059 --> 00:16:36,528 You can see it on the slide. 267 00:16:36,929 --> 00:16:39,965 So, we had three actives and three matching placebos. 268 00:16:39,965 --> 00:16:43,669 Essentially, a large complex trial, and they were randomized 4,000 patients. 269 00:16:43,669 --> 00:16:46,705 Partially because they're studying several treatments at once, also 270 00:16:46,705 --> 00:16:50,075 partially because, you know, they need for that event rate. 271 00:16:50,075 --> 00:16:53,779 Those are the number of patients you randomized and then not 272 00:16:53,779 --> 00:16:56,815 all of them will wind up with those events. 273 00:16:56,815 --> 00:17:01,553 And as we learned, it's the number of events that's your true sample size. 274 00:17:01,920 --> 00:17:06,358 So, they needed to randomize 4,000 for 90 percent power in the end 275 00:17:06,358 --> 00:17:10,095 to detect the reduction of sudden death with that 0.05 one-tailed 276 00:17:10,095 --> 00:17:12,798 significance level. So, that was the original design. 277 00:17:12,798 --> 00:17:16,535 And I don't know if any alarm bells are going off. 278 00:17:16,535 --> 00:17:19,605 Well, Laura Lee jumped up and down about this. 279 00:17:19,605 --> 00:17:23,342 It was Laura Lee Johnson who did the hypothesis testing? Yes. 280 00:17:23,809 --> 00:17:27,579 But -- so, the DSMB met for the first time, 281 00:17:27,579 --> 00:17:30,749 so this cardiovascular trial was launched in 1987. 282 00:17:30,749 --> 00:17:33,118 The first meeting of the DSMB, 283 00:17:33,118 --> 00:17:37,456 this is a blinded trial, large trial, death as an endpoint. 284 00:17:37,456 --> 00:17:41,393 It's going to have an ethical monitoring board as independent. 285 00:17:41,393 --> 00:17:45,364 You know, Phase 3 basically required to have a DSMB. 286 00:17:45,364 --> 00:17:48,901 Their first review was before even the study started, 287 00:17:48,934 --> 00:17:53,372 and that's often the case because they want to look at the protocol, 288 00:17:53,372 --> 00:17:58,143 they want to make sure that this monitoring board is okay with the protocol 289 00:17:58,143 --> 00:18:02,581 as is and agree on their charter; what are they going to monito, 290 00:18:02,581 --> 00:18:07,386 is everything won on the same page about what's appropriate to monitor and how? 291 00:18:07,386 --> 00:18:11,156 And as a result of that meeting, they made a recommendation 292 00:18:11,156 --> 00:18:14,226 actually to change the primary analysis of the trial. 293 00:18:14,226 --> 00:18:18,997 And you -- actually 1987 now, it's going on 30 years ago, and statistics 294 00:18:18,997 --> 00:18:20,365 was getting, in clinical 295 00:18:20,365 --> 00:18:24,136 trials, was still being formalized in terms of our best practices. 296 00:18:24,136 --> 00:18:27,906 And perhaps, this is the time when it basically became convincing. 297 00:18:27,906 --> 00:18:31,076 And certainly, there was a forward-thinking statistician at DSMB 298 00:18:31,076 --> 00:18:35,948 that this was no longer acceptable to use one-sided 0.05 level tests. 299 00:18:35,948 --> 00:18:40,452 So, the initial argument is we know the smaller the alpha, 300 00:18:40,452 --> 00:18:42,421 the more difficult it is. 301 00:18:42,421 --> 00:18:47,526 The more N you need to guarantee that level of a false-positive rate. 302 00:18:47,526 --> 00:18:52,865 And someone might argue, why would I want to spend alpha to look in 303 00:18:53,132 --> 00:18:57,302 either a strong positive direction or strong negative direction to decide 304 00:18:57,302 --> 00:19:00,038 whether or not there's a treatment difference 305 00:19:00,038 --> 00:19:04,343 when I'm only interested in the strong positive direction of benefit. 306 00:19:04,343 --> 00:19:07,880 I want all my alpha going to the 0.05 307 00:19:07,880 --> 00:19:11,416 and then I can use 1.64 and not 1.96. 308 00:19:11,416 --> 00:19:16,421 But the opposing argument that has become the dominant one is that you 309 00:19:16,421 --> 00:19:20,359 don't want a test, whether it's one-tailed or two-tailed, to influence 310 00:19:20,626 --> 00:19:24,196 level of evidence you're going to use to declare benefits. 311 00:19:24,196 --> 00:19:24,563 Right? 312 00:19:24,563 --> 00:19:30,135 So, whether you can do a one-tailed test, but it shouldn't be easier to pass 313 00:19:30,135 --> 00:19:33,839 that test than a two-tailed test in terms of deciding 314 00:19:33,839 --> 00:19:37,910 what is the benchmark, what the critical value, in this case 315 00:19:37,910 --> 00:19:41,246 1.96, to declare significance in terms of a difference 316 00:19:41,246 --> 00:19:45,717 between two groups, say, if you're doing one look at 0.05 level. 317 00:19:47,553 --> 00:19:50,789 And another argument is that, you know, you've -- almost rare 318 00:19:50,789 --> 00:19:54,326 that I never see -- I've never seen a one-sided confidence interval. 319 00:19:54,326 --> 00:19:58,130 When you do a confidence interval, you do the uncertainty plus or minus, 320 00:19:58,130 --> 00:20:00,199 say, 1.96 standard deviations around the mean. 321 00:20:00,199 --> 00:20:04,036 If you do your test as a one-sided test and a confidence interval, 322 00:20:04,036 --> 00:20:07,806 the two-sided confidence interval, you could wind up doing this test, 323 00:20:07,806 --> 00:20:09,942 declaring a difference, having a confidence interval, 324 00:20:09,942 --> 00:20:12,678 and having the null inside of that confidence interval. 325 00:20:12,678 --> 00:20:16,648 So, it's -- some might view a few more on the clinical side, 326 00:20:16,648 --> 00:20:20,619 the second esoteric argument in one of the statisticians on board with this. 327 00:20:20,619 --> 00:20:21,820 Well, they are now. 328 00:20:21,820 --> 00:20:24,890 But at the time, perhaps, it was an emerging consensus. 329 00:20:25,290 --> 00:20:28,160 And it was interesting that at the very first meeting, 330 00:20:28,160 --> 00:20:29,895 essentially the principal analysis was changed. 331 00:20:29,895 --> 00:20:33,365 Well, they made the recommendation and then the study team and sponsor 332 00:20:33,365 --> 00:20:36,201 were convinced by the arguments, and they did change it. 333 00:20:36,201 --> 00:20:38,870 And it only reduced the power a little bit. 334 00:20:38,870 --> 00:20:42,341 So, they were already going for 90 percent power, which quite convincing. 335 00:20:42,341 --> 00:20:46,979 So, they were still at 85 percent power when they did this change to be two-sided. 336 00:20:46,979 --> 00:20:51,016 So, I think in the end, everyone was satisfied with that change in 337 00:20:51,016 --> 00:20:51,817 the protocol. 338 00:20:51,817 --> 00:20:56,622 And so, the lesson learned here is it's very important to have the first meeting 339 00:20:56,622 --> 00:20:57,889 before the study starts. 340 00:20:57,889 --> 00:21:01,927 It was -- ultimately, the DSMB is charged with a pretty delicate sense 341 00:21:01,927 --> 00:21:02,561 of responsibility, 342 00:21:02,561 --> 00:21:07,132 actually a very important role, which is they are going to be looking at data 343 00:21:07,332 --> 00:21:10,636 no one else one has and be making recommendations that have 344 00:21:10,636 --> 00:21:13,472 a strong influence on the future of this trial. 345 00:21:13,839 --> 00:21:18,644 Everyone needs to be on the same page scientifically, believe in the trial, 346 00:21:18,644 --> 00:21:23,782 believe in validity from the start, and then be able to trust each other 347 00:21:23,782 --> 00:21:27,486 and get along through what could be potentially difficult times 348 00:21:27,486 --> 00:21:31,523 in terms of unexpected challenges that make this trial more difficult 349 00:21:31,523 --> 00:21:32,991 to complete than anticipated. 350 00:21:32,991 --> 00:21:36,328 So, you want that first meeting, so everyone is 351 00:21:36,328 --> 00:21:40,565 on the same page about roles and responsibilities and the science. 352 00:21:40,565 --> 00:21:45,904 Another lesson learned was not only should you be on the same page about, 353 00:21:45,904 --> 00:21:50,342 you know, the design but the monitoring plan should be in place. 354 00:21:50,676 --> 00:21:54,813 Because what happened at CAST, which I think was just one of those, 355 00:21:54,813 --> 00:21:57,649 every sixth month just, you know, right -- basically, 356 00:21:57,649 --> 00:22:00,485 there are two kinds of meetings the DSMB has. 357 00:22:00,485 --> 00:22:02,087 One is just regularly scheduled. 358 00:22:02,087 --> 00:22:03,989 We'll keep an eye on safety, 359 00:22:03,989 --> 00:22:07,793 and then you'll do these interim looks at data as you go. 360 00:22:07,793 --> 00:22:08,727 It's not expecting, 361 00:22:08,727 --> 00:22:12,531 you know, real efficacy trends until a lot of data has accumulated. 362 00:22:12,531 --> 00:22:16,668 But in this trial, with only 5 percent of the expected information, total 363 00:22:16,668 --> 00:22:17,969 -- you know, 5 364 00:22:17,969 --> 00:22:22,107 percent of the total number of events that they needed for that power, 365 00:22:22,107 --> 00:22:25,811 they actually already had a difference apparent between the treatment arms. 366 00:22:25,811 --> 00:22:30,449 Now, as is typical in a DSMB report, once you start looking at efficacy, 367 00:22:30,449 --> 00:22:33,318 the report is going to just have arm differences, 368 00:22:33,318 --> 00:22:36,822 and you might have them labeled Arm 1 and Arm 2. 369 00:22:36,855 --> 00:22:40,859 Because you never know, you want to be -- at the preponderance of caution, 370 00:22:40,859 --> 00:22:44,863 you want that report to be masked in case it gets into the wrong 371 00:22:44,863 --> 00:22:48,033 hands, mailed to the wrong person accidentally, left on a desk. 372 00:22:48,033 --> 00:22:51,169 But then the DSMB would have the potential to say, "Okay, 373 00:22:51,169 --> 00:22:54,906 tell me which arm is which because I want to make a decision." 374 00:22:54,906 --> 00:22:58,643 And others will always have that, which is which, say, in a separate 375 00:22:58,643 --> 00:23:01,780 envelope that they open up at the start of every meeting. 376 00:23:02,547 --> 00:23:05,884 In this particular trial as was done at this time, 377 00:23:05,884 --> 00:23:09,921 more commonly than it is now, they chose to remain blinded until, 378 00:23:09,921 --> 00:23:13,592 you know, perhaps they decided there was a reason to look. 379 00:23:13,592 --> 00:23:14,926 And so, they looked. 380 00:23:14,926 --> 00:23:18,296 All they could see was there was a treatment difference. 381 00:23:18,296 --> 00:23:22,300 At the time they hadn't had stopping boundaries in place, perhaps because 382 00:23:22,300 --> 00:23:26,338 it was so early and they were still gelling the analysis plans. 383 00:23:27,906 --> 00:23:29,775 And so, in the end 384 00:23:29,775 --> 00:23:33,812 it was probably fine because even though there was a convincing 385 00:23:33,812 --> 00:23:38,250 -- when they did the test, in this case the logrank test 386 00:23:38,250 --> 00:23:41,586 because they're looking at survival curves between an active 387 00:23:41,586 --> 00:23:45,991 and inactive or, say, placebo, the logrank looked like it was succeeding. 388 00:23:45,991 --> 00:23:50,796 It was 3.43 for treatment difference, and the boundary that they were eventually 389 00:23:50,796 --> 00:23:54,299 applying retrospectively looking back, it seemed that they had exceeded it. 390 00:23:54,299 --> 00:23:56,868 But it's that issue if there were pending events, 391 00:23:56,868 --> 00:23:59,171 and actually when they were adjudicated and added 392 00:23:59,171 --> 00:24:03,175 in, it actually would have flipped it, so that would have been less convincing. 393 00:24:03,175 --> 00:24:06,645 So, in the end, even if they had the boundary in place, 394 00:24:06,645 --> 00:24:11,216 even if they had looked to see which arm is which, they may not have stopped. 395 00:24:11,216 --> 00:24:12,651 But it was instantly alarming. 396 00:24:12,651 --> 00:24:14,386 But they chose to remain blinded. 397 00:24:15,353 --> 00:24:18,256 And at the next look, and there's a reference 398 00:24:18,457 --> 00:24:22,027 for this trial that people talk endlessly about for many reasons. 399 00:24:22,027 --> 00:24:24,696 At the next look, there were 48 events. 400 00:24:24,696 --> 00:24:27,699 So, at the first look, it was roughly 22. 401 00:24:27,699 --> 00:24:29,367 Now, there were 48 events. 402 00:24:29,367 --> 00:24:32,370 And actually, looking at event rates and other reasons, 403 00:24:32,370 --> 00:24:36,374 they realized that they would only expect 300 and not the 400 404 00:24:36,374 --> 00:24:38,043 and some they'd originally thought 405 00:24:38,043 --> 00:24:42,714 at the end of this trial, which meant the 48 was a little bit 406 00:24:42,714 --> 00:24:46,384 higher of a percent in terms of total number of events. 407 00:24:46,818 --> 00:24:50,956 And when they looked at the boundary, they saw that -- so, 408 00:24:50,956 --> 00:24:54,793 the logrank was actually still at 3.22, and it was well 409 00:24:54,793 --> 00:24:58,063 over the boundary for stopping, plus or minus 2.98. 410 00:24:58,063 --> 00:25:03,435 And so, the DSMB requested to be unblinded to find out which arm is which. 411 00:25:03,435 --> 00:25:06,905 And so much to their surprise, it was the placebo 412 00:25:06,905 --> 00:25:08,640 that was doing the best. 413 00:25:08,640 --> 00:25:11,743 And it was active arms that were having harm. 414 00:25:12,444 --> 00:25:15,347 And all along, everyone had assumed it was benefit 415 00:25:15,347 --> 00:25:18,617 that just they were waiting for it to be convincing. 416 00:25:18,617 --> 00:25:22,187 And this becomes the compelling argument for why most people argue, 417 00:25:22,187 --> 00:25:26,424 you should not say, well, to remain objective, I don't want to know 418 00:25:26,424 --> 00:25:29,995 which arm is which until I really think there's a difference. 419 00:25:29,995 --> 00:25:33,899 And the reason is you're probably not going to make a symmetric, 420 00:25:33,899 --> 00:25:36,368 even decision in terms of how much evidence 421 00:25:36,601 --> 00:25:40,405 you need to convince the medical community to adopt this new therapy 422 00:25:40,405 --> 00:25:45,277 which might have harm as well as benefits versus evidence that, you know what, I'm 423 00:25:45,277 --> 00:25:48,847 80 percent confident that, you know, we might be harming people 424 00:25:48,847 --> 00:25:53,685 but I think we should wait to 90 percent confident because it's kind of early. 425 00:25:53,685 --> 00:25:55,987 So, it's really not a symmetric operation. 426 00:25:55,987 --> 00:26:00,625 And I think having seen this over and over overtime, people have just -- 427 00:26:00,625 --> 00:26:05,030 a lot of think tanks that make recommendations on how to do these things, 428 00:26:05,030 --> 00:26:08,833 people who have served on these bodies many times in their careers 429 00:26:08,833 --> 00:26:12,637 make an overwhelming recommendation that you -- the DSMB is not masked. 430 00:26:12,637 --> 00:26:16,107 You know, the trial is double blinded, the investigators and patients. 431 00:26:16,408 --> 00:26:18,176 It is not triple blinded. 432 00:26:18,176 --> 00:26:21,746 We do not blind the DSMB. We want the DSMB 433 00:26:21,746 --> 00:26:25,650 to have every bit of information possible to make the decisions. 434 00:26:25,650 --> 00:26:27,452 And this is ultimately why 435 00:26:27,452 --> 00:26:31,356 because you cannot but sort of believe in a new treatment. 436 00:26:31,356 --> 00:26:35,994 And in fact, they were so convinced that something had to be wrong 437 00:26:35,994 --> 00:26:39,230 that they investigated the -- they got random bottles, 438 00:26:39,264 --> 00:26:39,998 they assumed 439 00:26:39,998 --> 00:26:44,736 maybe the place where they were labeling the bottles had made a mistake 440 00:26:44,736 --> 00:26:49,507 and that the placebo was the active and the active was the placebo, 441 00:26:49,507 --> 00:26:53,878 so they started testing chemically bottled medications that were had those labels. 442 00:26:53,878 --> 00:26:58,984 They look for imbalances and key variables and the placebo and the active arms 443 00:26:58,984 --> 00:27:02,287 to see maybe the differences were because these folks 444 00:27:02,287 --> 00:27:04,856 weren't actually having the same risk factors. 445 00:27:05,590 --> 00:27:08,360 And then they continued to also look to see 446 00:27:08,360 --> 00:27:11,997 if there were certain subgroups that maybe were being harmed versus others. 447 00:27:11,997 --> 00:27:16,301 Again, thinking, well, of that subgroup was out of balance, that might have driven 448 00:27:16,301 --> 00:27:17,202 this treatment difference. 449 00:27:17,202 --> 00:27:19,638 And they also looked at the secondary endpoint, 450 00:27:19,638 --> 00:27:22,874 which is overall mortality, to see if it was consistent. 451 00:27:22,874 --> 00:27:27,579 And finally, when they were not -- they realized it was in fact the placebo 452 00:27:27,579 --> 00:27:32,117 that was doing better, they wanted to see whether certain arrhythmia drugs 453 00:27:32,117 --> 00:27:36,221 out of the three they were studying were actually the problem. 454 00:27:36,221 --> 00:27:39,924 And it turned out early on that there were two 455 00:27:39,924 --> 00:27:43,294 out of the three active drugs, encainide and flecainide, 456 00:27:43,294 --> 00:27:47,399 luckily the two I can pronounce, that were driving the difference. 457 00:27:47,399 --> 00:27:51,136 So, each of the three actives had a matched control. 458 00:27:51,603 --> 00:27:54,272 And so, the overall score was 35 459 00:27:54,272 --> 00:27:57,342 events on the active, 13 on the placebo. 460 00:27:57,342 --> 00:28:01,579 And when they reduced down to those -- on the encainide 461 00:28:01,579 --> 00:28:05,050 and flecainide versus placebo, it was 33 versus 9. 462 00:28:05,050 --> 00:28:09,287 So, what had just been CAST became divided up into CAST 463 00:28:09,287 --> 00:28:12,757 1, which was the trial with all three treatments, 464 00:28:12,757 --> 00:28:18,530 and then at the end of this DSMB, they stopped the encainide and flecainide arms, 465 00:28:18,530 --> 00:28:22,767 but they continue the third active placebo because there was evidence. 466 00:28:22,767 --> 00:28:26,204 Certainly, there was no reason to stop those arms. 467 00:28:26,204 --> 00:28:30,842 And so, the was the benefit of having sort of the structure 468 00:28:30,842 --> 00:28:34,279 of multiple matched placebos that they continued one question, 469 00:28:34,279 --> 00:28:37,749 even though they had to stop the other two. 470 00:28:37,749 --> 00:28:41,319 So, other lessons learned here is that not only should 471 00:28:41,319 --> 00:28:45,256 your monitoring plan be in place or DSMB be in place 472 00:28:45,256 --> 00:28:48,827 all before your study starts, but your data management system, 473 00:28:48,827 --> 00:28:53,331 the data entry, all those people in charge of monitoring your data, entering 474 00:28:53,565 --> 00:28:57,368 data, organizing your data should be on board on day one. 475 00:28:57,368 --> 00:29:00,972 And that data system has been built and tested 476 00:29:00,972 --> 00:29:05,744 because you need that data at that first meeting if it's available. 477 00:29:05,744 --> 00:29:10,115 Because very early on trends can become apparent, certainly for harm. 478 00:29:10,115 --> 00:29:14,886 And also, it turns out for benefit, say, in the AZT trial. 479 00:29:14,886 --> 00:29:19,290 And other principles are, you know, you need to be organized, 480 00:29:19,290 --> 00:29:22,861 you need to have these meetings on the books 481 00:29:22,861 --> 00:29:27,632 because there could be many reasons why you need to act quickly, 482 00:29:28,166 --> 00:29:31,436 you know, to start your monitoring right away and also 483 00:29:31,436 --> 00:29:35,974 you need a board that's going to be able to think about and investigators 484 00:29:35,974 --> 00:29:39,844 contingency plans about what to do once you have this this information. 485 00:29:39,844 --> 00:29:44,249 So, people need to be ready to receive the DSMB recommendations as well. 486 00:29:44,249 --> 00:29:49,587 And I think -- as I mentioned earlier, I think probably went over most of this, 487 00:29:49,587 --> 00:29:54,125 is that the ability of the DSMB to do their job can be compromised 488 00:29:54,125 --> 00:29:55,426 if they remain masked. 489 00:29:56,561 --> 00:29:57,228 And I 490 00:29:57,228 --> 00:30:01,266 think initially, the idea of objectivity seemed to be the overwhelming principle 491 00:30:01,266 --> 00:30:05,670 of why we would keep the DSMB mask unless, you know, and perhaps 492 00:30:05,670 --> 00:30:10,708 always give the -- and always give them the prerogative to ask to be unblinded. 493 00:30:10,708 --> 00:30:14,913 But I think it was, you know, examples like this, that in general, 494 00:30:15,246 --> 00:30:18,116 the recommendation is that you keep DSMB body unblinded. 495 00:30:18,116 --> 00:30:21,152 The report may stay masked because you never know 496 00:30:21,152 --> 00:30:24,189 who could intercept it, but you would always have 497 00:30:24,189 --> 00:30:27,892 that DSMB have the ability to see which arm is which. 498 00:30:28,226 --> 00:30:31,296 And a great article, and it's just an editorial, 499 00:30:31,296 --> 00:30:34,699 it's only about two pages, that really summarizes both sides, 500 00:30:34,699 --> 00:30:38,436 I think, but it's definitely by the name of this article, 501 00:30:38,436 --> 00:30:40,505 Masked Monitoring in Clinical Trials--Blind Stupidity? 502 00:30:40,505 --> 00:30:44,576 Obviously coming down on one side of the debate here, and certainly 503 00:30:44,576 --> 00:30:48,680 a good way to augment reading if you're interested in this topic. 504 00:30:50,682 --> 00:30:51,015 Well, 505 00:30:51,015 --> 00:30:54,252 they do present the opposing view, at least briefly. 506 00:30:54,252 --> 00:30:58,122 Okay, so hopefully, those two examples, you know, sort of have 507 00:30:58,122 --> 00:31:02,527 given you some ideas of why you need Data and Safety Monitoring Board. 508 00:31:02,527 --> 00:31:06,598 You know, essentially, you know, it summarizes examples, you need to able 509 00:31:06,598 --> 00:31:11,002 to identify any safety problem rapidly, you need to be able to identify 510 00:31:11,002 --> 00:31:15,039 logistical problems if there are some, so the board can recommend amendments 511 00:31:15,039 --> 00:31:20,812 or make sure those come to light such as, you know, if the accrual is so slow, 512 00:31:20,812 --> 00:31:24,849 they'll never get full enrollment by the end of the grant period. 513 00:31:24,849 --> 00:31:29,220 That could be a problem, so maybe you need to add sites. 514 00:31:29,220 --> 00:31:33,558 So, they'll, you know, be actually evaluating the feasibility of the trial, 515 00:31:33,558 --> 00:31:36,094 not just thing like efficacy or safety. 516 00:31:36,094 --> 00:31:40,064 And ultimately, they want to make sure if the trial objectives 517 00:31:40,064 --> 00:31:44,068 have been met early then the trial should be terminated early.