Good evening, and welcome to tonight's lecture on Sample Size and Power.
My name is Dr.
Laura Lee Johnson.
I'm an associate director at the United States Food and Drug Administration.
So, the disclaimer, of course, nothing that I say
the FDA wants to be construed as representing their views or policies.
So, why do we care about sample size and power?
Think about this a little bit.
Now last week, you all heard a couple of lectures from Paul Wakim.
This is actually a slide that he showed in another set of lectures that we gave.
And I think it's a very nice way to kind of set the mood for this lecture.
If you forget by the time we go through all the formulas in the next 90 minutes,
power is basically the probability of getting a statistically significant result
when, in fact, there's a clinically meaningful difference.
So, this is kind of unknown to us, right?
And part of what we're going to be discussing in that -- over the next several weeks
is, how do you know that something's clinically meaningful?
But this in essence, this idea of power, we want to have high power, right?
We want to have a high probability of seeing a statistically significant result
in those hypotheses tests from last week if, in fact, there
is this clinically meaningful difference.
So, by definition, those studies that have low power are less
likely to produce statistically significant results,
even when that clinically meaningful effect exists.
And that's the reason your animal care committees and the human
subjects committees are going to say, "Well, is it ethical to do research
if you don't
have that much power?" So, this becomes not just a, "I
want to have significant findings question." It's also an ethical question.
Is it worthwhile to put people at risk or their data or their privacy even at risk
if you're not going to be able to tell that, as one of my colleagues
said, "There's a there there?" Now, the flip of this in how power
and sample size and statistical significance tied together is that the lack
of statistical significance doesn't prove that there is no treatment effect,
because it could be a consequence of small sample size or really its low power.
So, it's important to have enough power and an adequate sample size.
So, our objectives
for this lecture, to calculate changes in sample size based on the changes in
the difference of interest, and the variance, or the number of study arms.
So, we'll talk about each of those.
We'll understand
the intuition behind power calculations, hopefully, by the end of tonight's lecture.
Be able to recognize some of the common sample size
formulas for the tests that were talked about last week.
And learn some tips for getting through the IRB.
Your general takeaway message is you need to get input from a statistician.
It's not just because I am one, but because the ramifications of
not continually talking to your statisticians and epidemiologists can be pretty big.
You can waste a lot of money and a lot of time.
All calculations, after I say calculations are important,
you kind of have to take them with a few grains of salt.
Most of the basic formulas that you will find are, like, the lowest level possible.
But in reality, we have these kind of what we call in the U.S.
"fudge factors." So, realistically, people are going to drop out.
Maybe the variance of that endpoint is going to be larger than you were anticipating.
Maybe accrual will be slower than you were anticipating.
A lot of things can happen.
And so, we have to take that into account when we're
trying to figure out what our actual sample size should be.
The other trick is, you will always, always, always round up on sample size, never down.
Why is that?
Because if your calculation tells you that you need 10.01 human beings
in your study, there is no one one-hundredth of a human being.
You need 11 people in your study, or your power goes down.
We'll talk a little bit more about that as we progress through the lecture.
But also, always remember that analysis follows the design.
So, few other take home messages.
You really need to give thought and have a broad discussion
about what difference is scientifically important and in what units.
Are we talking inches or centimeters? Why?
It can actually -- you know, the sample size calculations are not invariant
as a mathematical term, but the units matter, and also the measurement error.
Remember I mentioned at one point, do I use, like, a little plastic ruler?
Or how other method could I use to measure my waist?
Think about how you're going to be measuring, because that information
has to go into your sample size calculation.
Are you taking a single measure of systolic blood pressure,
or are you averaging three together?
This information is going to matter when you're calculating power and sample size.
How variable are the measurements?
Again, my favorite plastic ruler does not measure my waist well.
So, in sample size, you need to think of a few elements.
One is, what is the difference or the effect to be detected?
This might be a ratio. It might be an actual difference.
This element needs to be known.
You need to think about the variance in the outcome.
You need to understand what significance level you want.
So, is it a one-tailed or a two-tailed test is also an important part
of all of these equations. What power do you want?
What level of Type II error are you willing to allow?
Do you want equal randomization to different
study arms or unequal randomization to all the study arms?
Are you thinking about doing a superiority, equivalence, or non-inferiority trial?
All of these have different sample size calculations
or different numbers that you need to put in for the calculations.
Other elements that come up will be a follow-up period.
How long a participant is followed?
One of our examples will talk a little bit about that.
We'll also talk a little bit about censoring.
This is when a participant has been in the trial for a while, you have all their data,
but now that participant is no longer being followed.
So, you have what's called incomplete follow-up which is common.
And sometimes we also do administrative censoring.
So, let's say I'm trying to follow people and find out when they die.
Well, it's the end of my study, and they're still alive.
So, at the end of study I may administratively censor them saying, "I
knew they were alive up until this time point."
We'll talk more about this in the survival analysis lecture.
So, we're going to start off tonight talking about power,
go through some basic sample size information, a whole bunch of examples.
But there are going to be more examples in the textbook.
I'm going to
talk about some of the few changes you can make to the basic formula.
That's also covered more in depth in the textbook.
You also will do a little bit of multiple comparisons.
You've already talked about that sum, and other people will talk about it more.
So, I'll try to race through that to get to what St.
George's College has put out in their information to researchers. As
these are really poor sample size statements, please never send them to our IRB again.
So, again, power depends on your sample size.
A lot of people ask me why we don't have power formulas.
Well, they are there, but they're algebraically intensive.
It's actually easier to teach the sample size formula.
You can work backwards to figure out a power formula if you need it.
But either way, you know, the math will work.
But again, this idea about power is it's the probability
of rejecting the null hypothesis if the alternative hypothesis is true.
Typically, the more subjects you have, the higher power you have.
So, what's power affected by?
I already mentioned the sample size.
But variation in that outcome is huge.
If I can lower my variance, I increase my power, all other things being equal.
If I increase the significance level, so if I say, "Oh, instead of a Type
I error of 0.05, I could have 0.1," my power will go up.
And if I'm interested in different effects, it's always easier to detect a bigger effect.
If I increase the difference I'm willing to detect, my power will go up.
But again, it has to be reasonable.
If you tell me that you're going to lower someone's average systolic blood pressure
by 50 points, by 50 units, I'm going to say you are crazy, and that can't be done.
So, you have to be careful that your difference is actually reasonable.
One-tailed versus two-tailed tests.
For the same alpha,
your power is greater in a one-tailed test than a comparable two-tailed test.
Because remember, in a two-tailed test, I have to split it onto two different sides.
So, here are a few examples.
Let's say that I have a two-arm study.
I have a total sample size of 32.
So, I have 16 people in each study arm.
So, this is a small little study.
Some people may think it's like an R21 or an R03 if you get NIH's extramural funding.
81 percent power, my delta
is equal to 2, so the difference I want to detect is 2.
I set my standard deviation at 2.
The Type I error, I set at 0.05.
And I want to do a two-sided test.
So, at this sample size with all this information, I have 81 percent power.
Let's say I was wrong on the standard deviation.
Instead it was 1, not 2.
The good news is my power has gone up to 99.9 percent.
Problem is normally we are in this second situation
where we thought it was going to be 2, it turns out it was 3.
2 and 3 sounds so close together, but not in power land.
Now, my power has dropped to 47 percent.
We'll talk about why in a few minutes.
The significance level.
I was going to plan my study for 0.05.
Now, they're saying, "No, you have all these things going on.
We think you should test that 0.01." Your power, instead of
being 81 percent, is 69 percent when you do that.
You cannot change your alpha level without changing your sample size
if you want to maintain your power.
But let's say instead you are going to power all of these interactions at 0.05,
then somebody said, "No." You can test these interactions at 0.1, not 0.05.
Well, your power has now gone up, 81 percent to 94 percent.
That's pretty cool.
Let's say the difference to be detected, you wanted to detect
a difference of two hours of sleep with your new intervention.
You have this behavioral intervention that you think people will sleep longer.
You cut down one hour of sleep.
Your power to detect one hour of sleep is 29 percent.
Not such a smart thing.
You kind of want to figure out, do you care about two hours or one hour?
Let's say instead of two hours, you go for three hours.
These people are getting no sleep, and you just want to see, does it show anything?
Well, your power has gone up to 99 percent.
But does your intervention really cause people to sleep three more hours a night?
Probably not.
Let's say like I admit I did once.
You thought the sample size of 32 was per arm and not total for your study.
So, you actually had 64 people in the study.
The good news is your power is 98 percent.
The bad news is they just spent a lot of extra money they didn't need to spend.
The problem, however, is we always find ourselves in this other situation.
Sample size is supposed to be 32.
We're running out of money.
We're having trouble recruiting, and we ended up with only 28 people in the study.
So, our power that was 81 percent -- and typically,
the reason I chose 81 percent is because 80 percent is usually considered
kind of a minimum for a study, minimum level of power, although that varies.
We'll talk a little bit about that.
But what happens here is my power is now 75 percent.
So, I don't have 80 percent of a chance to see statistical significance
when I have something that's clinically meaningful.
I only have a 75 percent chance.
Maybe that's okay, but maybe not.
Then we have these two-tailed versus one-tailed test.
If I switch to a one-tailed test, a boost up a little bit to 88 percent power.
So, that gets us to what should your power actually be.
So, again, that Phase III minimum tends to be 80 percent.
There are other people that will say, "Your Type I error should equal your Type II
error." A lot of these large, huge definitive trials might have 99.9 percent power.
I have other folks that say, "Well, I'm just doing a proof of concept,
and they have 60 to 80 percent power." In some of the omics studies that I looked at,
they have really high power, because Type II error is such a problem for them.
So, what is your power formula?
It all depends on your study design.
It's just algebraically intensive,
and you should probably use a computer program to figure out your power.
Or make a statistician run a bunch of simulations for you
to figure out your range of power.
So, this is kind of our basic sample size 101, where everything starts.
Flipping from power to sample size,
same elements are in play.
Changes in the difference of interest are going to have huge impacts
on your sample size, and so are changes in the variance.
Some examples here.
If you have a 20-point difference that you designed for, maybe you need 25
patients per group.
If I change nothing else but this point difference, if I drop it
by half, to 10, I now need 100 patients per group.
If it drops again to a 5-point difference, I now need 400 patients per group.
All the types of things that feed into just your basic study
design, a basic randomized two-arm study changes in the difference
to be detected, your alpha or that Type I error, your beta, or one minus
beta, your power, the standard deviation, the number of samples you're taking.
Is it cluster randomized or non-cluster randomized studies?
Is it a one-sided or two-sided test?
All of these can have huge impacts on your sample size calculation.
So, your basic formula for the total sample of a two-arm study
that's just simple, basic individual randomized study, nothing special is that
2N, where N is the sample size for a single arm,
would be 4 times -- this is as critical value Z.
These are on that bell curve.
Your computer will tell you what this value is.
Or if you have a really old stats textbook, there's a picture of the bell curve
in the back that tells you. 1 minus alpha over 2.
Why is this alpha?
This is your Type I error that you are getting to allow.
Divided by 2 if it's a two-sided test.
So, for
a 0.05 test, that's two-sided, this value is like 1.96.
Z of 1 minus beta.
This 1 minus beta is that power, all right?
So, if it's 80 percent power, I think that's 1.645.
Again, it's in your textbook, we actually wrote some of the numbers down.
You take these two Z critical values, and you square them.
You multiply it by the variance,
divided by the difference you're entrusted and squared.
Now, all the different formulas for sample size for all the different study
designs and regressions, et cetera, they all look different.
But these basic elements will be in every single one of them.
So, what do you need to think about before taking all this information
to your statistician?
What do you even need to talk to them about
in addition to the basic background for your projects?
First off, do you have a randomized or non-randomized study?
Non-randomized studies require a lot larger sample size,
because remember, we've got all those potential confounders we've got to look at.
Similarly, the effect modifiers.
We have to account for that in the sample size planning.
Occasionally in surveys -- so, Barbara Stussman may talk a little bit
more about this in a few weeks, absolute sample size may be of interest.
So, if you're doing a survey, sometimes we take a percent of population approach.
We may say you have the given threshold in your confidence interval around,
your percent needs to be above a certain threshold.
So, you're trying to plan around that.
So, you've got to keep in mind what is the study's primary outcome.
That's your basis for your sample size calculation.
That said, secondary or other outcome variables, if you think they're important,
make sure you have sufficient sample size to be able to answer those key questions.
Increase the real sample size for your study to reflect things
like loss to follow up, expected response rates, lack of compliance, et cetera.
You can't do this, you know, just off the cuff though.
You have to write into your human subjects applications and your grant applications
how you are justifying those kind of plus-ups, right?
And again, always round up.
So, let's say that you're going to do your sample size calculation,
and you're looking through books and you're looking online,
which you're going to probably find are sample size calculations
for two groups, a continuous outcome, and look at the mean difference.
We're going to talk about that mainly tonight, because the similar ideas
hold for different types of outcomes and study designs.
But do understand a lot of those formulas will not be applicable to your research,
because you're answering a different question.
So, to your lovely person helping you with your sample size
calculation, you need to tell them what are all the variables of interest
that you want to measure.
What's the type of data? Is it continuous? Is it categorical?
Is it ordinal? Bring those some examples, if you can.
What is the desired power?
Where are you in this continuum of research?
And what is the power that's going to be expected in your field?
What is the desired significance level?
Along those same lines.
What's the effect or difference that's clinically important?
Very rarely is this a single number.
A lot of times, what you're looking for is you're going to say, "Okay.
What seems to be feasible given the research that's done?
What seemed to be consensus reports?"
Maybe you have animal data, you're trying to extrapolate it to human beings.
Maybe you have nothing at all.
And we'll talk about those. But what seems to matter?
This is the reason
I don't always like the effect size elements where they just look at the ratio
of the variance
and the difference, because -- or rather, the standard deviation and the difference.
The problem when you use that for your effect size is you haven't figured out like,
what's the actual variance?
And does that effect size actually translate to a difference that's relevant?
So, think about those standard deviations for your continuous outcomes.
But if at all possible, try to have real data on both of those bullets.
And decide if you're doing a one or two-sided test.
Realistically, you're going
to have to have a really good justification to do a one-sided test.
What is your data structure?
Are we talking paired data?
Are you doing pre-post on folks? Are you taking repeated measures?
Are you going to take this blood pressure measurement 16 times over a two-year period?
Are you looking at groups of equal sizes?
Or are you going to do a 2-to-1 randomization?
Or maybe it's a case control study, and you're going to get five controls for every case.
Do you have hierarchical or nested data?
Is that inside individually randomized folks, or are you doing a cluster randomized trial
where you're going to have people inside of schools, inside of regions?
Are you looking at biomarkers?
Or are you trying to actually do and develop a biomarker?
Because there's a lot of stuff that can be involved in the laboratory
assay part, and very different equations, if you're in fact trying to validate
a biomarker versus using it as essentially the endpoint in your trial.
And how much validation information do you have about any of these endpoints?
How do you know how they perform in your population?
Your study design is going to really matter.
I've talked a little bit about the randomization aspects.
You need to talk to them about how you're going to randomize your trial.
But again, there are all different formulations, whether it's equivalence.
So, again, how then do you define the equivalence bound?
If it's non-inferiority, what's that non-inferiority bound versus superiority?
If you have a non-randomized intervention study,
what other variables are you measuring that will need to go into the model?
So, prevalence or observational studies.
Are you trying to figure out positive and negative predictive values?
Sensitivity and specificity?
All of these have different calculations.