1 00:00:08,208 --> 00:00:10,043 So, what are some changes? 2 00:00:10,043 --> 00:00:15,148 If you have unequal numbers in each group, the ratio of cases to controls, 3 00:00:15,148 --> 00:00:19,886 you want to use this if you have, say, a certain lambda, a 4 00:00:19,886 --> 00:00:22,822 set of patients randomized to the treatment arm 5 00:00:22,822 --> 00:00:25,725 for every patient randomized to the placebo arm. 6 00:00:25,725 --> 00:00:29,029 You can also use this in a case-control setting. 7 00:00:29,029 --> 00:00:33,400 So, the idea here is that you're going to have different variances. 8 00:00:34,200 --> 00:00:38,905 You're going to kind of use this ratio, calculate the sample size for one group, 9 00:00:38,905 --> 00:00:42,976 and then use that to calculate the sample size for the other group. 10 00:00:45,512 --> 00:00:47,747 The shortcut way of doing this, 11 00:00:47,747 --> 00:00:51,818 let's say I'm going to use equal variance sample size formula. 12 00:00:51,818 --> 00:00:55,121 So, I assume both groups have the same variance. 13 00:00:55,121 --> 00:01:00,326 I want to look at the total sample size, calculate that, then increase it 14 00:01:00,326 --> 00:01:01,795 by this factor k. 15 00:01:01,795 --> 00:01:04,364 So, let's say I'm doing 2-to-1 randomization. 16 00:01:04,364 --> 00:01:06,599 I calculate the total sample size. 17 00:01:06,599 --> 00:01:13,239 If I had two equally sized study arms, it's 26, so it's going to have 13 per arm. 18 00:01:14,007 --> 00:01:16,676 I'd say, okay, 26 times 2 plus 19 00:01:16,676 --> 00:01:21,281 1, because that's my ratio, squared, divided by 4 times the 2. 20 00:01:21,281 --> 00:01:26,286 And I find out that I need 30 people in my study instead. 21 00:01:26,286 --> 00:01:30,890 So, I would have 20 in one group, 10 in the other. 22 00:01:30,890 --> 00:01:32,792 Why might I do this? 23 00:01:32,792 --> 00:01:35,095 Well, sometimes I have a device. 24 00:01:35,095 --> 00:01:38,164 Let's say, I have a new artificial knee. 25 00:01:38,164 --> 00:01:42,802 And I want to know how many knees I need to make. 26 00:01:42,802 --> 00:01:45,872 And I want to do an unequal randomization. 27 00:01:46,406 --> 00:01:49,576 This might be a case where I'll use that. 28 00:01:49,576 --> 00:01:52,412 Sometimes I have a fixed number of cases. 29 00:01:52,412 --> 00:01:54,080 I've already made the knees. 30 00:01:54,080 --> 00:01:57,083 I'd only have so many of them to use. 31 00:01:57,083 --> 00:02:01,087 So, if my sample size calculation said I needed 13 per arm, 32 00:02:01,087 --> 00:02:07,127 but I only have 11 of the knees, I want the same precision that I'm going to calculate 33 00:02:07,127 --> 00:02:12,899 how many people will be in my control arm and back calculate that using this formula. 34 00:02:12,899 --> 00:02:16,903 When you're doing survival analysis and other types of analyses, 35 00:02:16,903 --> 00:02:23,042 you may have a cohort of say those exposed and unexposed people that we talked about. 36 00:02:23,042 --> 00:02:26,479 You have this relative risk that you're interested in. 37 00:02:26,479 --> 00:02:29,549 I know the prevalence and the unexposed population. 38 00:02:29,549 --> 00:02:32,986 I need to figure out the number of events. 39 00:02:32,986 --> 00:02:39,492 So, the risk of an event in the exposed group divided by the risk of the event 40 00:02:39,492 --> 00:02:41,761 in the unexposed group, and back 41 00:02:41,761 --> 00:02:46,366 calculate the number of events I think I'm going to be seeing. 42 00:02:47,367 --> 00:02:52,038 So, I have the number of events in my unexposed group, calculate 43 00:02:52,038 --> 00:02:57,110 the number of events in the exposed group, then calculate how many people 44 00:02:57,110 --> 00:03:01,014 I need to see the events. Remember stability issues. 45 00:03:01,014 --> 00:03:03,750 In logistic regression, there's this big discussion, 46 00:03:03,750 --> 00:03:09,589 and a whole bunch of simulations were done that you need at least 10 subjects 47 00:03:09,589 --> 00:03:11,157 for every variable investigated. 48 00:03:11,157 --> 00:03:16,996 You look at principal components analysis and a lot of other methods that get used. 49 00:03:17,630 --> 00:03:19,999 You see everything from 10 to 100. 50 00:03:19,999 --> 00:03:21,868 So, all sorts of crazy. 51 00:03:21,868 --> 00:03:25,438 Some people don't say you have to use balanced designs. 52 00:03:25,438 --> 00:03:28,141 Other people say you have to use them. 53 00:03:28,141 --> 00:03:31,511 I'm just going to say, if you have balanced designs, 54 00:03:31,511 --> 00:03:34,914 you have the same number of people in each study 55 00:03:34,914 --> 00:03:38,985 arm, it's easier to figure out your power and your sample size. 56 00:03:40,520 --> 00:03:41,955 It is easier to 57 00:03:41,955 --> 00:03:46,626 handle, but you know, I have a job to work on complicated designs. 58 00:03:46,626 --> 00:03:48,428 So, you can do simulations. 59 00:03:48,428 --> 00:03:53,132 You can do pretty much anything you want as long as you can 60 00:03:53,132 --> 00:03:57,971 do the work to make sure it's going to be rigorous and valid. 61 00:03:57,971 --> 00:03:59,973 So, what about multiple comparisons? 62 00:03:59,973 --> 00:04:04,277 If you have four groups, and you want to do all two-way 63 00:04:04,277 --> 00:04:07,146 comparisons of means, you have six different tests. 64 00:04:07,981 --> 00:04:10,617 But sometimes you'd say, "Well, I have four groups. 65 00:04:10,617 --> 00:04:13,253 I have one placebo arm and three different doses. 66 00:04:13,253 --> 00:04:16,756 And I actually just want to compare each dose to the placebo. 67 00:04:16,756 --> 00:04:21,427 I'm not going to compare the doses to each other." So, you have to figure out 68 00:04:21,427 --> 00:04:24,330 exactly how many tests you really are going to run. 69 00:04:24,330 --> 00:04:27,533 The simple way people do it is this Bonferroni test, right? 70 00:04:27,567 --> 00:04:31,437 They divide their alpha by the number of tests. It's common. 71 00:04:31,437 --> 00:04:34,240 There's a long literature, but it's super conservative. 72 00:04:34,240 --> 00:04:39,045 And in some of your high-throughput laboratory tests, it can be worked out. 73 00:04:39,045 --> 00:04:43,716 Ed Corin, I think it was, worked this out, and maybe Lisa 74 00:04:43,716 --> 00:04:45,852 or somebody else in that branch. 75 00:04:45,852 --> 00:04:50,423 It said, like, there were cases that it was physically impossible for them 76 00:04:50,423 --> 00:04:52,525 to have any statistically significant results 77 00:04:52,892 --> 00:04:54,994 after they accounted for multiple comparisons. 78 00:04:54,994 --> 00:04:57,297 That's not an experiment worth running. 79 00:04:57,297 --> 00:05:02,035 This comes up a lot and it's like microarray, proteomics, and other omics 80 00:05:02,035 --> 00:05:03,069 types of experiments. 81 00:05:03,069 --> 00:05:04,837 Many times people say, "Just 82 00:05:04,837 --> 00:05:09,375 set your alpha and your beta stricter." I tend to agree with this. 83 00:05:09,375 --> 00:05:12,178 You know, just set a really strict Type 84 00:05:12,178 --> 00:05:17,083 I error level that you're going to adhere to and test everything against that. 85 00:05:17,083 --> 00:05:21,654 If that's what you want to think about for false positives, false negatives, 86 00:05:21,654 --> 00:05:26,492 there is also like familywise error rates, and all these other things. 87 00:05:26,492 --> 00:05:31,130 But I think just set strict upfront and plan for that. 88 00:05:31,130 --> 00:05:34,567 So, what are some of these rejected statements? 89 00:05:34,567 --> 00:05:35,768 Again, this St. 90 00:05:35,768 --> 00:05:37,804 George's Hospital -- you know, 91 00:05:37,804 --> 00:05:42,642 I'm looking at this link, I think this is the correct link. 92 00:05:42,642 --> 00:05:44,243 But I don't remember 93 00:05:44,243 --> 00:05:49,916 if I double checked it this morning because they keep moving some stuff around. 94 00:05:50,249 --> 00:05:56,556 Anyway, this entire -- you can put the name into your favorite search engine and find it. 95 00:05:56,556 --> 00:06:00,660 St. George's Hospital Medical School has one of the best statistics 96 00:06:00,660 --> 00:06:03,996 guys for research grant applicants that I've ever seen. 97 00:06:03,996 --> 00:06:09,936 Like many times, I was like, "Can I just copy and reference it and be done?" 98 00:06:09,936 --> 00:06:14,607 I think it's very applicable across any country that you'd go to. 99 00:06:14,607 --> 00:06:19,612 So, one of my favorite parts, because I see it way too often. 100 00:06:19,612 --> 00:06:25,351 It's people who go, "Me, too." It's like, "No, you must justify your sample size 101 00:06:25,351 --> 00:06:29,756 in every single application." So, this is an example, an exact quote, "A 102 00:06:29,756 --> 00:06:32,458 previous study in this area recruited 150 subjects 103 00:06:32,458 --> 00:06:36,562 and found highly significant results, p value equals 0.014, and therefore similar 104 00:06:36,562 --> 00:06:40,633 sample size should be sufficient here." Well, they could have been lucky. 105 00:06:40,633 --> 00:06:43,002 It could have been random sampling variation. 106 00:06:43,002 --> 00:06:46,072 It could have been a lot of other things. 107 00:06:46,773 --> 00:06:50,810 You need to give an estimate of what you're expecting for the variance, 108 00:06:50,810 --> 00:06:55,148 for the difference, what your Type I error is going to be, et cetera. 109 00:06:55,148 --> 00:06:57,650 You have to give all those little elements. 110 00:06:57,650 --> 00:07:01,988 It's good to know that some study who had similar elements found something significant. 111 00:07:01,988 --> 00:07:07,293 But honestly, if a lot of studies have already found this, why are you doing your study? 112 00:07:09,362 --> 00:07:10,530 No prior information. 113 00:07:10,530 --> 00:07:15,568 Now, this is always trouble because sometimes you do have no prior information. 114 00:07:15,568 --> 00:07:20,640 And what you're trying to do is run the little baby first pilot. 115 00:07:20,640 --> 00:07:25,311 But the quote in the application -- for substantial sample size application, 116 00:07:25,311 --> 00:07:29,982 I should say sample sizes aren't provided because there's no prior information 117 00:07:29,982 --> 00:07:32,718 on which to base them. All right. 118 00:07:32,718 --> 00:07:35,455 We have to know how many people, 119 00:07:35,455 --> 00:07:39,358 how many animals, whatever you are enrolling in this study. 120 00:07:39,358 --> 00:07:43,629 Find something that's published previously, and try to extrapolate from that. 121 00:07:44,063 --> 00:07:45,398 Conduct a small pre-study. 122 00:07:45,398 --> 00:07:49,135 But if this is your application to conduct the small pre-study, 123 00:07:49,135 --> 00:07:52,505 then you probably, in fact, don't need sample size calculations. 124 00:07:52,505 --> 00:07:55,208 Don't tell me the sample sizes aren't provided. 125 00:07:55,208 --> 00:07:59,245 Tell me like this is where you are, and that you're going 126 00:07:59,245 --> 00:08:03,282 to use this amount of information to then calculate appropriate sample sizes. 127 00:08:03,282 --> 00:08:07,019 That this is what you're gathering in order to do it. 128 00:08:08,154 --> 00:08:08,888 But if 129 00:08:08,888 --> 00:08:14,560 I can easily find a bunch of information that could have informed your sample sizes, 130 00:08:14,560 --> 00:08:19,265 I'm probably not going to have a very happy review for you. 131 00:08:19,265 --> 00:08:22,101 Variance? No prior information on standard deviations? 132 00:08:22,101 --> 00:08:25,104 Well, again, this might be where you talk 133 00:08:25,104 --> 00:08:29,609 about detecting the size of difference in terms of the standard deviations. 134 00:08:29,609 --> 00:08:33,379 This is where those effect sizes that Cohen and others 135 00:08:33,379 --> 00:08:37,149 and their different effect size values for different regression models. 136 00:08:38,050 --> 00:08:40,720 But this is where that can be useful. 137 00:08:40,720 --> 00:08:42,388 Again, it's your starting step, 138 00:08:42,388 --> 00:08:47,593 but it's not going to be what you use for a pivotal or final study. 139 00:08:47,593 --> 00:08:49,762 Now, the number of available patients, 140 00:08:49,762 --> 00:08:54,433 Wendy Weber and others are going to talk a little bit more about this, 141 00:08:54,433 --> 00:08:59,105 but you have to make sure you have patients to be in the study. 142 00:08:59,639 --> 00:09:00,339 Few things. 143 00:09:00,339 --> 00:09:04,977 You will not get all patients to be in your study, number one. 144 00:09:04,977 --> 00:09:11,050 In fact, you're lucky if like 5 percent agreed to be in your study in most cases. 145 00:09:11,050 --> 00:09:12,852 Number two, feasible is important. 146 00:09:12,852 --> 00:09:15,721 It does not tell me about your power. 147 00:09:15,721 --> 00:09:19,992 So, this is, again, a direct quote that they got at St. 148 00:09:19,992 --> 00:09:22,495 George's, "The clinic sees around 50 patients 149 00:09:22,495 --> 00:09:27,500 a year, of whom 10 percent may refuse to take part in the study." 150 00:09:28,868 --> 00:09:30,002 I don't think that's 151 00:09:30,002 --> 00:09:34,040 realistic that only 10 percent will refuse, but we keep moving on. 152 00:09:34,040 --> 00:09:38,778 "Therefore over the two years of the study, the sample size will be 90 153 00:09:38,778 --> 00:09:42,815 patients." Let's say a miracle occurs, and they get their 90 patients. 154 00:09:42,815 --> 00:09:46,519 I have no idea what the power is for this study. 155 00:09:46,519 --> 00:09:50,222 I know nothing about the differences that they want to detect. 156 00:09:50,590 --> 00:09:54,193 I don't even know what they want to measure. 157 00:09:54,193 --> 00:09:56,596 I don't know about the variance. 158 00:09:56,596 --> 00:09:57,797 I know nothing. 159 00:09:57,797 --> 00:10:03,002 It's okay to give this information and then say, "And with 90 patients, 160 00:10:03,002 --> 00:10:07,006 our power will be X based on the following assumptions." 161 00:10:07,006 --> 00:10:12,178 That would have been okay, but that sentence never appeared in the application. 162 00:10:12,178 --> 00:10:14,981 So, you do have to balance feasibility 163 00:10:14,981 --> 00:10:19,785 and power, but sample size isn't decided just on available patient numbers. 164 00:10:22,755 --> 00:10:26,058 So, what are some of our resources and conclusions? 165 00:10:26,058 --> 00:10:27,893 What impacts your sample size? 166 00:10:27,893 --> 00:10:29,729 The differences, the standard deviations 167 00:10:29,729 --> 00:10:35,234 or the variance, your Type I error that you're willing to tolerate, your Type II 168 00:10:35,234 --> 00:10:40,740 error that you're willing to tolerate, number of arms or samples, one or two-sided test. 169 00:10:40,740 --> 00:10:41,841 Are you randomizing? 170 00:10:41,841 --> 00:10:45,511 If you are, what type of randomization are you using? 171 00:10:45,511 --> 00:10:47,346 What are the analysis plans? 172 00:10:47,346 --> 00:10:52,118 If you don't have an estimate of the variance, make sample size or power 173 00:10:52,118 --> 00:10:53,085 tables, make graphs. 174 00:10:53,085 --> 00:10:57,723 You should in fact always -- even when you think you have good estimates, 175 00:10:57,723 --> 00:11:02,962 they may not be that good, you're always going to want to make tables and graphs. 176 00:11:02,962 --> 00:11:06,599 Show a wide variety of possible standard deviations, of possible differences. 177 00:11:06,599 --> 00:11:10,870 And protect yourself with a high sample size, if it's at all possible. 178 00:11:12,438 --> 00:11:14,473 So, when I was reviewing grants 179 00:11:14,473 --> 00:11:19,545 for the NIH, the top 10 statistics questions I had to ask back to investigators. 180 00:11:19,545 --> 00:11:23,616 The exact mechanism they were going to use to randomize their patients. 181 00:11:23,616 --> 00:11:26,652 Don't tell me you're just going to randomize them. 182 00:11:26,652 --> 00:11:31,424 Dr. Wakim went through a whole bunch of different ways you can randomize people. 183 00:11:31,424 --> 00:11:32,758 Tell me more information. 184 00:11:32,758 --> 00:11:35,961 Tell me when, et cetera. 185 00:11:35,961 --> 00:11:38,464 They would say they're going to stratify. 186 00:11:38,464 --> 00:11:41,667 I'd be like, "Why are you going to stratify? 187 00:11:41,667 --> 00:11:46,272 What are you going to stratify on?" The EMA, the European Medicines Agency, 188 00:11:46,272 --> 00:11:49,475 actually provides different information on something called dynamic allocation. 189 00:11:49,475 --> 00:11:53,045 So, if you're going to use different types of allocation 190 00:11:53,045 --> 00:11:57,683 schemes, they may or may not be acceptable to different regulatory bodies worldwide. 191 00:11:57,683 --> 00:12:01,954 Or they may be acceptable in pediatrics, but not acceptable in adults, 192 00:12:01,954 --> 00:12:07,359 acceptable on oncology, not acceptable in, you know, cardiovascular disease. 193 00:12:07,359 --> 00:12:10,029 Blinded or masked personnel. 194 00:12:10,029 --> 00:12:13,999 Who is doing your endpoint assessment? 195 00:12:13,999 --> 00:12:15,968 Who is masked? 196 00:12:15,968 --> 00:12:19,271 To what are they masked? 197 00:12:19,271 --> 00:12:28,214 Are you compromising your blinded study with something else that you were doing? 198 00:12:28,214 --> 00:12:36,455 Many times, I had to ask people to list every single hypothesis. 199 00:12:36,455 --> 00:12:38,457 The hypothesis test, 200 00:12:39,859 --> 00:12:41,060 the specific analysis, 201 00:12:41,060 --> 00:12:45,798 and the specific sample size or power justification for those hypothesis tests. 202 00:12:45,798 --> 00:12:49,335 That should be very clear in your grant application. 203 00:12:49,335 --> 00:12:52,505 Somebody shouldn't have to ask you for this. 204 00:12:52,505 --> 00:12:58,043 It should also be clear in the statistical analysis plans that go with it. 205 00:12:58,043 --> 00:13:02,014 And it needs to match everything else in your protocols. 206 00:13:02,014 --> 00:13:07,186 I would need to ask how or if they were adjusting for multiple comparisons. 207 00:13:08,020 --> 00:13:11,090 Some fields get very, very picky about multiple comparisons. 208 00:13:11,090 --> 00:13:14,493 Others say, "Well, if you're honest and you report everything, 209 00:13:14,493 --> 00:13:17,596 you can say I'm not adjusting for multiple comparisons." 210 00:13:17,596 --> 00:13:21,667 Typically, most of your regulators and pivotal trials will make you do 211 00:13:21,667 --> 00:13:26,438 what's called a statistical hierarchy or some way to control the Type I error. 212 00:13:26,438 --> 00:13:30,543 So, it may or may not be technically adjusting for multiple comparisons, 213 00:13:30,543 --> 00:13:36,081 but you are kind of doing a conservation, they call it, of the Type I error. 214 00:13:36,081 --> 00:13:39,084 I would ask if they're testing for effect modification. 215 00:13:40,753 --> 00:13:43,088 Are you doing any interim analyses? 216 00:13:43,088 --> 00:13:46,559 And if so, what exactly are you analyzing? When? 217 00:13:46,559 --> 00:13:48,460 What's the error spending model? 218 00:13:48,460 --> 00:13:50,796 And what are the stopping rules? 219 00:13:50,796 --> 00:13:55,801 So, these error spending models are, again, to conserve your Type I error. 220 00:13:55,801 --> 00:14:01,207 And I would ask if this was accounted for in the sample size calculations. 221 00:14:01,207 --> 00:14:07,012 Why all of those ways that you do interim analyses impact your sample size calculation? 222 00:14:07,012 --> 00:14:09,381 What is your expected dropout? 223 00:14:09,381 --> 00:14:12,985 You know, they would tell me about their sample size, 224 00:14:12,985 --> 00:14:18,757 but not the expected dropout, and did they plus up in order to account for this. 225 00:14:18,757 --> 00:14:23,095 How do you handle the dropouts and missing data in your analyses? 226 00:14:23,095 --> 00:14:27,600 Again, those methods may also impact your sample size and your power. 227 00:14:27,600 --> 00:14:32,471 And then I always had to ask about repeated measures and longitudinal data, 228 00:14:32,504 --> 00:14:36,108 almost always. Are you using a linear mixed model? 229 00:14:36,108 --> 00:14:39,311 Are you trying to use repeated measures ANOVA? 230 00:14:39,311 --> 00:14:42,114 Repeated measures ANOVA is an awful method. 231 00:14:42,114 --> 00:14:45,718 It's very antiquated. It has a lot of assumptions. 232 00:14:45,718 --> 00:14:51,323 The newer methods have a lot fewer assumptions, they were a lot more robust. 233 00:14:51,323 --> 00:14:54,526 Maybe they're going to use generalized estimating equations. 234 00:14:54,526 --> 00:14:57,363 But again, what is your actual method? 235 00:14:57,363 --> 00:14:59,765 How are you collecting the data? 236 00:14:59,765 --> 00:15:01,333 What's the study design? 237 00:15:01,333 --> 00:15:05,738 What are the analyses in order to actually test that data? 238 00:15:05,738 --> 00:15:11,744 All of that goes into figuring out your sample size and the analyses and conclusions 239 00:15:11,744 --> 00:15:13,345 that you can draw.