1 00:00:09,042 --> 00:00:10,377 My name is Dr. 2 00:00:10,377 --> 00:00:11,378 Laura Lee Johnson. 3 00:00:11,378 --> 00:00:14,347 I'm the associate director of the Division of Biometrics 4 00:00:14,347 --> 00:00:17,317 III at the United States Food and Drug Administration. 5 00:00:17,317 --> 00:00:20,153 I'm also one of the co-directors for IPPCR. 6 00:00:20,153 --> 00:00:24,624 This is our disclaimer, courtesy of my working at the FDA, you know, 7 00:00:24,624 --> 00:00:28,928 they hire me, they pay me, but they only want to take account 8 00:00:28,928 --> 00:00:33,566 of anything that they decide they like and disown anything that they don't like. 9 00:00:33,800 --> 00:00:38,638 Part of what's really important here is in fact there are a lot of different people 10 00:00:38,638 --> 00:00:41,374 who take this course from all around the world. 11 00:00:41,374 --> 00:00:44,110 And for some of us, it's the first introduction 12 00:00:44,110 --> 00:00:47,747 and for others we have a pretty advanced understanding of clinical research. 13 00:00:47,747 --> 00:00:52,919 So part of what I aim for in my lectures is to give some of the tricks, 14 00:00:52,919 --> 00:00:56,556 and the tips, and the concepts that I've learned over the years 15 00:00:56,556 --> 00:00:58,658 from the various investigators I've worked with. 16 00:00:58,691 --> 00:01:03,430 So my general objectives here, I want you all to be better 17 00:01:03,430 --> 00:01:06,199 consumers of the medical and scientific literature 18 00:01:06,199 --> 00:01:11,337 because while this course is focused on principles and practice of clinical research, 19 00:01:11,337 --> 00:01:15,275 those principles and practices are true for non-clinical research, too. 20 00:01:15,275 --> 00:01:21,214 There's a huge push right now also from the NIH and from many other groups. 21 00:01:21,214 --> 00:01:25,552 You'll see many journals that have written their editorials saying, "We 22 00:01:25,552 --> 00:01:30,690 are no longer going to accept really bad research just because it's pre-clinical." 23 00:01:32,459 --> 00:01:33,259 And all 24 00:01:33,259 --> 00:01:37,630 of these study designs, many of them, they have laboratory components. 25 00:01:37,630 --> 00:01:39,599 Not just because you do. 26 00:01:39,599 --> 00:01:45,572 But my biggest randomization problem happened in somebody who did not randomize on her 96 27 00:01:45,572 --> 00:01:51,377 well plate, and it caused a huge problem for her study, and her interpretation. 28 00:01:51,377 --> 00:01:54,747 So I want you to be better consumers. 29 00:01:54,747 --> 00:01:57,550 I want you to be better users. 30 00:01:57,550 --> 00:02:01,521 And I want to enhance the conversation inside research teams. 31 00:02:01,855 --> 00:02:05,558 And it might be with your study statisticians and epidemiologists, but 32 00:02:05,558 --> 00:02:09,629 it also will be across and between a lot of different folks. 33 00:02:09,629 --> 00:02:12,332 Realistically, what we really want is better science. 34 00:02:12,332 --> 00:02:13,333 With the information 35 00:02:13,333 --> 00:02:18,071 you learn, you're not going to be able to do your own statistical analyses. 36 00:02:18,071 --> 00:02:20,406 Like, some of you already can do 37 00:02:20,406 --> 00:02:24,144 your statistical analyses based on when you were raising your hands. 38 00:02:24,844 --> 00:02:30,717 But realistically, if you want to learn how to do stats, you want to learn 39 00:02:30,717 --> 00:02:37,724 how to be a data manager, you need to go take direct coursework in how to do that. 40 00:02:37,724 --> 00:02:42,428 But this should aid you to improve your abilities to critically evaluate 41 00:02:42,428 --> 00:02:44,764 grant applications, protocols, and the literature. 42 00:02:44,764 --> 00:02:51,404 No one's an expert in any given area, but we do all try to combine our expertise, 43 00:02:51,404 --> 00:02:55,875 and in this world, everything really is a team science work. 44 00:02:55,875 --> 00:03:01,548 Because it's really easy to write that your study's going to use a randomized 45 00:03:01,548 --> 00:03:05,051 double-blind control, parallel-arm design, and do an intent-to-treat analysis. 46 00:03:05,852 --> 00:03:10,290 It's really easy to say that subjects and participants are going to be consented. 47 00:03:10,290 --> 00:03:13,459 Those which you've already heard from the first two lectures, 48 00:03:13,459 --> 00:03:18,565 and you're going to learn throughout the entire course, it's really not easy to do it. 49 00:03:18,565 --> 00:03:22,702 And it's not easy to implement and maintain the integrity of your randomization. 50 00:03:22,702 --> 00:03:24,904 And Pamela [spelled phonetically], Shawn [spelled phonetically], 51 00:03:24,904 --> 00:03:26,172 and some other folks 52 00:03:26,172 --> 00:03:30,944 have written in the chapter on randomization, a lot of the threats to that integrity 53 00:03:30,944 --> 00:03:35,081 that show up, very well-meaning people, trying to do good studies, and how 54 00:03:35,081 --> 00:03:36,049 they were undermined. 55 00:03:37,383 --> 00:03:40,186 It's hard to maintain blinding and masking. 56 00:03:40,186 --> 00:03:44,190 One of the tricks one of my investigators taught me, 57 00:03:44,190 --> 00:03:50,563 she actually made badges for her study staff, and they had blinded with a little person 58 00:03:50,563 --> 00:03:55,368 with a little mask, almost like a racoon-looking thing, and then unblinded. 59 00:03:55,368 --> 00:03:58,571 And all of her staff wore these badges 60 00:03:58,571 --> 00:04:03,343 because she was doing a study where it was a physical intervention. 61 00:04:03,576 --> 00:04:07,947 And people knew that they were doing yoga, or they weren't doing yoga, 62 00:04:07,947 --> 00:04:10,950 but she didn't want them to talk about it 63 00:04:10,950 --> 00:04:15,622 to the people who were actually taking them through all of their study measurements. 64 00:04:15,622 --> 00:04:19,325 That was an innovative way to try to protect the blinding. 65 00:04:19,325 --> 00:04:20,660 It helped her participants 66 00:04:20,660 --> 00:04:25,331 remember who they couldn't talk to but then who they needed to complain to. 67 00:04:27,000 --> 00:04:28,167 Multiple study arms. 68 00:04:28,167 --> 00:04:33,373 How do you make sure your study arms aren't bleeding into each other? 69 00:04:33,373 --> 00:04:34,173 Data collection. 70 00:04:34,173 --> 00:04:39,345 How do you actually make sure that you're standardized your data collection process? 71 00:04:39,345 --> 00:04:43,349 We'll talk a lot about that throughout the several lectures. 72 00:04:43,349 --> 00:04:48,121 And also, how do you transfer data to regulatory and other groups? 73 00:04:48,121 --> 00:04:53,726 It might be that your studies don't fall under FDA purview, but they might. 74 00:04:54,727 --> 00:04:58,164 There are a lot of different regulatory organizations around the world 75 00:04:58,164 --> 00:05:01,567 and a lot of different rules that you have to follow. 76 00:05:01,567 --> 00:05:04,037 But many times, different people want your data 77 00:05:04,037 --> 00:05:09,642 and even if none of the regulatory groups do, this is a time and world of data sharing, 78 00:05:09,642 --> 00:05:14,580 and how do you adequately share that data, and make sure it's useful to other people? 79 00:05:14,814 --> 00:05:19,052 There're a lot of data standards you're going to hear about later 80 00:05:19,052 --> 00:05:23,623 in the winter and those will be interesting, useful for you, too. 81 00:05:23,623 --> 00:05:25,758 But that's a long view. 82 00:05:25,758 --> 00:05:30,330 Now just tonight, I'm going to talk about how to identify study designs 83 00:05:30,330 --> 00:05:33,866 that are used in clinical studies, epidemiology public health research. 84 00:05:33,866 --> 00:05:37,770 We're actually going to cover most of the epidemiology next Monday. 85 00:05:38,371 --> 00:05:42,608 I'm also going to talk about masking and blinding, different types 86 00:05:42,608 --> 00:05:44,544 of interventions, and comparison groups. 87 00:05:44,544 --> 00:05:49,949 So if you want to know what chapters in the book this is covered 88 00:05:49,949 --> 00:05:52,618 under, it's chapters 19 and 29. 89 00:05:52,618 --> 00:05:55,722 So what is your question of interest? 90 00:05:55,722 --> 00:05:59,559 Are you trying to interpret work in some new population? 91 00:05:59,559 --> 00:06:03,796 Are you trying to make a decision about an individual case? 92 00:06:03,796 --> 00:06:08,034 How many people in the room are actively, clinically doing work? 93 00:06:08,034 --> 00:06:09,202 A handful. Okay. 94 00:06:09,202 --> 00:06:13,806 So what we find out comparing two groups of people, or multiple 95 00:06:13,806 --> 00:06:18,044 groups of people may be very different than when you need 96 00:06:18,044 --> 00:06:22,682 to make a decision about that individual patient in front of you. 97 00:06:22,882 --> 00:06:26,652 And for each of us, like, I talk to my dad. 98 00:06:26,652 --> 00:06:29,055 What drug is he going to use? 99 00:06:29,055 --> 00:06:33,860 His doctor talks about like the pluses and minuses of all these different therapies. 100 00:06:33,860 --> 00:06:36,295 What in the end is the decision? 101 00:06:36,295 --> 00:06:39,031 And how does it work for him? 102 00:06:39,031 --> 00:06:41,434 We're looking at changing a population. 103 00:06:41,434 --> 00:06:46,239 Diabetes management, a large portion, has tried to shift the curve of a population. 104 00:06:47,573 --> 00:06:51,644 Sometimes, classically we look at those differences of groups in a study. 105 00:06:51,644 --> 00:06:54,347 But sometimes we're trying to do biomarker development, 106 00:06:54,347 --> 00:06:57,750 we have to figure out exactly what type of biomarker. 107 00:06:57,750 --> 00:06:58,751 People love biomarkers, 108 00:06:58,751 --> 00:07:03,156 but they forget to figure out exactly what the biomarker is for sometimes. 109 00:07:03,156 --> 00:07:06,225 Are you trying to develop a whole new outcome? 110 00:07:06,225 --> 00:07:09,629 Part of my job is it's patient reported outcome liaison. 111 00:07:10,329 --> 00:07:13,466 I help people develop new endpoints for clinical trials 112 00:07:13,466 --> 00:07:15,535 that involve the patient voice. 113 00:07:15,535 --> 00:07:17,270 The level of evidence. 114 00:07:17,270 --> 00:07:20,039 What is it that we're trying to establish? 115 00:07:20,039 --> 00:07:24,544 Are we trying to figure out what the current level of evidence is? 116 00:07:24,544 --> 00:07:29,048 You're going to hear about meta-analyses and other types of secondary data reviews. 117 00:07:29,048 --> 00:07:31,818 Is that what we're trying to do instead? 118 00:07:35,221 --> 00:07:40,092 Regardless of what you're doing, always remember the analysis follows the design. 119 00:07:40,092 --> 00:07:42,495 Your question will always come first. 120 00:07:42,495 --> 00:07:45,331 And we may have to edit it 121 00:07:45,331 --> 00:07:50,169 because it may not be directly answerable, but your question comes first. 122 00:07:50,169 --> 00:07:55,842 If at the end of the day, they're answering something that does not address 123 00:07:55,842 --> 00:07:56,642 your question, 124 00:07:56,642 --> 00:08:02,281 you need to say, "Hold up, people designing this study and analyzing the data. 125 00:08:02,615 --> 00:08:08,488 That is not what we need to do." Because your question is going to drive 126 00:08:08,488 --> 00:08:09,288 the hypotheses. 127 00:08:09,288 --> 00:08:13,593 We're going to actually design that experimental design for your study 128 00:08:13,593 --> 00:08:17,897 is in order to make sure we can test the hypotheses. 129 00:08:17,897 --> 00:08:23,369 We're going to do all of our sampling, all of that data collection, in-line 130 00:08:23,369 --> 00:08:24,937 with the experimental design. 131 00:08:24,937 --> 00:08:27,273 Your data comes from the samples. 132 00:08:27,540 --> 00:08:30,109 We analyze the data. We draw conclusions. 133 00:08:30,109 --> 00:08:35,214 That generally leads to more questions, and we start the whole process again. 134 00:08:35,214 --> 00:08:41,420 But sometimes I look at data analyses, and I read papers, and I'm like who cares? 135 00:08:41,420 --> 00:08:44,323 Like you answered a question that was answerable, 136 00:08:44,323 --> 00:08:47,260 but it wasn't the actual question of interest. 137 00:08:47,260 --> 00:08:48,728 So always strive -- 138 00:08:48,728 --> 00:08:54,000 like the reason statisticians have jobs, it's not just because we like to analyze data, 139 00:08:54,000 --> 00:09:00,206 but to say you have a new, cool question, and we don't have a method to do it. 140 00:09:00,206 --> 00:09:04,844 So we need to develop the methodology to answer the actual pertinent question. 141 00:09:04,844 --> 00:09:09,515 The other problem, though, is you need to take all your design information 142 00:09:09,515 --> 00:09:14,687 to a statistician early and often because part of our job is to give guidance 143 00:09:14,687 --> 00:09:17,456 about some of the assumptions for the methods. 144 00:09:17,456 --> 00:09:22,628 And to try to help make sure that we're going to do the best job 145 00:09:22,628 --> 00:09:26,065 possible with the fewest subjects possible to answer a question. 146 00:09:26,065 --> 00:09:30,202 Because, of course, you ask a statistician how they see research study, 147 00:09:30,503 --> 00:09:33,139 they say everything impacts the statistical analysis. 148 00:09:33,139 --> 00:09:37,677 But I will also say that that's not just my job security. 149 00:09:37,677 --> 00:09:38,844 That's because sometimes 150 00:09:38,844 --> 00:09:44,884 at the end of the day, investigators bring the information to me, and I say, well, 151 00:09:44,884 --> 00:09:49,422 we've undermined the integrity of the study by making the following decisions. 152 00:09:49,422 --> 00:09:51,324 So I can't help you. 153 00:09:51,324 --> 00:09:54,727 You collected all that data and it's basically worthless. 154 00:09:55,528 --> 00:09:56,929 You don't want that to happen. 155 00:09:56,929 --> 00:09:59,799 It's not good for you. It's not good for your study team. 156 00:09:59,799 --> 00:10:03,603 And it's not good for the human beings that agreed to participate in your study. 157 00:10:03,603 --> 00:10:06,439 So we're going to go through a little bit of vocabulary. 158 00:10:06,439 --> 00:10:07,173 But, of course, 159 00:10:07,173 --> 00:10:10,977 none of you will want that to happen and you sure don't want it to happen 160 00:10:10,977 --> 00:10:13,346 because you're here late at night listening to me talk. 161 00:10:15,214 --> 00:10:17,283 So when I go through 162 00:10:17,283 --> 00:10:21,787 vocabulary, part of this is to get us on similar footing. 163 00:10:21,787 --> 00:10:23,823 We will talk about arms. 164 00:10:23,823 --> 00:10:26,258 I do not mean my appendage. 165 00:10:26,258 --> 00:10:31,597 But in clinical research, we talk about study arms, or samples, or groups. 166 00:10:31,597 --> 00:10:34,066 We use these words fairly interchangeably. 167 00:10:34,066 --> 00:10:38,537 A lot of times we talk about wanting to demonstrate superiority. 168 00:10:38,537 --> 00:10:43,476 So remember John powers talked about this a little bit last night. 169 00:10:43,476 --> 00:10:47,079 When you want to demonstrate superiority, you're talking about 170 00:10:47,079 --> 00:10:51,684 detecting a difference between either groups, or between treatments, or study arms. 171 00:10:51,684 --> 00:10:57,089 The idea is that there's a difference in some way, shape, or form. 172 00:10:57,089 --> 00:11:01,727 Sometimes we say we want to demonstrate that the different arms 173 00:11:01,727 --> 00:11:03,663 are equally or similarly effective. 174 00:11:03,663 --> 00:11:05,965 So that is an equivalence trial. 175 00:11:05,965 --> 00:11:09,435 Sometimes we want to demonstrate that things are non-inferior. 176 00:11:09,435 --> 00:11:14,440 You got to be careful with non-inferiority because sometimes they start stepping away, 177 00:11:14,707 --> 00:11:19,845 so I have one non-inferiority study, then they say, well, now I have a new compound, 178 00:11:19,845 --> 00:11:25,017 so I need to show that it's non-inferior to the first thing you showed was non-inferior. 179 00:11:25,017 --> 00:11:28,854 I got to make sure it's not inferior to group one, right. 180 00:11:28,854 --> 00:11:29,655 So non-inferiority. 181 00:11:29,655 --> 00:11:34,960 You can also think of this while it's not exactly the same kind of like generics, 182 00:11:35,161 --> 00:11:35,561 right. 183 00:11:35,561 --> 00:11:40,766 When you think about equivalence, I sometimes think about generics, I have a generic 184 00:11:40,766 --> 00:11:44,503 cough syrup, I want to cough about the same amount 185 00:11:44,503 --> 00:11:47,473 plus or minus a little bit, that's equivalence. 186 00:11:47,473 --> 00:11:48,607 Non-inferiority is [negative] 187 00:11:48,607 --> 00:11:53,813 it may be a little bit worse but not enough that it really matters. 188 00:11:53,813 --> 00:11:56,816 Figuring out that margin, big, big difficult problem. 189 00:11:58,851 --> 00:11:59,618 Also, I'm 190 00:11:59,618 --> 00:12:04,724 a very bad lady and interact and use patient versus participant versus subject. 191 00:12:04,724 --> 00:12:10,196 Truth be told, what you're supposed to do is say a study subject. Why? 192 00:12:10,196 --> 00:12:14,867 That helps kind of differentiate, like, really when you're in clinical research, 193 00:12:14,867 --> 00:12:16,836 you are a guinea pig. 194 00:12:16,836 --> 00:12:22,708 I sign up for clinical trials, I recommend anybody who does work in clinical research, 195 00:12:22,708 --> 00:12:24,276 sign up for trials. 196 00:12:24,276 --> 00:12:25,845 Because you should understand 197 00:12:25,845 --> 00:12:30,516 what it means to have your data possibly out there and breached. 198 00:12:30,850 --> 00:12:38,023 You should understand what it -- pain in the neck it is to fill out all these forms. 199 00:12:38,023 --> 00:12:40,426 Understand what the burden is. 200 00:12:40,426 --> 00:12:45,598 But we also sometimes, in the literature especially, more of the behavioral 201 00:12:45,598 --> 00:12:48,000 social science, are talking about participants. 202 00:12:48,000 --> 00:12:50,770 We want people to feel like they're 203 00:12:50,770 --> 00:12:53,172 participating in research and actively engaged. 204 00:12:53,172 --> 00:12:58,344 I do a lot of my work now, though, with patient medical records, 205 00:12:58,344 --> 00:13:02,314 so they are literally patients that we are working with. 206 00:13:03,382 --> 00:13:05,317 But because of that, I 207 00:13:05,317 --> 00:13:09,989 have a tendency to flip in between all of these three words 208 00:13:09,989 --> 00:13:14,260 but what I should always be saying is participant and subject. 209 00:13:14,260 --> 00:13:15,828 So shame on me. 210 00:13:15,828 --> 00:13:17,596 Don't make my mistakes. 211 00:13:17,596 --> 00:13:20,499 A little bit about study design taxonomy. 212 00:13:20,499 --> 00:13:24,770 I kind of break the world into interventional and observational studies. 213 00:13:24,770 --> 00:13:27,506 Interventional means I do something to you. 214 00:13:27,506 --> 00:13:34,113 Observational means I watch the film of your life, or the photograph, as the case may be. 215 00:13:35,514 --> 00:13:39,051 We also break the world into longitudinal versus cross-sectional. 216 00:13:39,051 --> 00:13:41,020 So longitudinal is the film. 217 00:13:41,020 --> 00:13:46,525 So when I look at you at baseline six months, seven months, eight months. 218 00:13:46,525 --> 00:13:51,630 Cross-sectional is I give you all one survey right now and we're done. 219 00:13:51,630 --> 00:13:53,966 It's kind of like a census. 220 00:13:53,966 --> 00:13:57,136 We ask you something once, walk away. 221 00:13:57,136 --> 00:13:58,704 Prospective versus retrospective. 222 00:13:58,704 --> 00:14:02,641 Prospective means I'm going to follow you into the future. 223 00:14:02,942 --> 00:14:06,378 I'm going to kind of take my data real time. 224 00:14:06,378 --> 00:14:09,148 Retrospective means I look back in the past. 225 00:14:09,148 --> 00:14:13,619 So I may be looking back at employee health records that were gathered 226 00:14:13,619 --> 00:14:18,090 over the last 30 years by a Department of Defense or an army. 227 00:14:18,090 --> 00:14:22,228 So retrospective is looking backwards at data that was already collected, may 228 00:14:22,228 --> 00:14:27,032 or may not have been collected the way you wanted it, or the data 229 00:14:27,032 --> 00:14:30,302 you wanted, but you're using what's already there. 230 00:14:30,302 --> 00:14:33,706 Prospective you are moving forward and collecting data. 231 00:14:33,706 --> 00:14:39,645 Don't get too hung up on those, but it is a little bit important. 232 00:14:39,645 --> 00:14:45,618 So like if you're prospectively collecting data, and storing it, and you go back 233 00:14:45,618 --> 00:14:50,289 and analyze your stored data, it's still a prospective study. 234 00:14:50,289 --> 00:14:51,991 Blinded or masked. 235 00:14:51,991 --> 00:14:56,662 So this is when the investigator, the people running the study, 236 00:14:56,662 --> 00:15:00,933 the study participants do not know what intervention they're on. 237 00:15:01,467 --> 00:15:05,838 Sometimes we actually mask them instead to the hypothesis that we're testing. 238 00:15:05,838 --> 00:15:10,576 Sometimes we do not blind or mask a study and it's called open 239 00:15:10,576 --> 00:15:13,512 label study for some of the interventional trials. 240 00:15:13,512 --> 00:15:18,617 But depending on who all is blinded or not, single blind, double blind, unblinded. 241 00:15:18,617 --> 00:15:22,621 I used to do some work for the National Eye Institute. 242 00:15:22,621 --> 00:15:25,557 They do not like blinded. They prefer masked. 243 00:15:25,758 --> 00:15:27,192 You can understand why. 244 00:15:27,192 --> 00:15:31,630 Depending on where you train, you will see different names for each. 245 00:15:31,630 --> 00:15:32,898 Randomized or nonrandomized. 246 00:15:32,898 --> 00:15:36,101 We're going to have a whole lecture on randomization. 247 00:15:36,101 --> 00:15:38,971 Paul Wakim gives a great lecture on this. 248 00:15:38,971 --> 00:15:41,807 The idea is how am I allocating subjects. 249 00:15:41,807 --> 00:15:44,310 Is there a random element to it? 250 00:15:44,310 --> 00:15:49,315 Or can I figure out who's going to be going into which treatment group?