What does the Power of a study tell you?
The “Power” of a study is the probability of finding a significant effect when it actually exists. In other words, finding that the difference between 2 treatment groups is significant at the 5% (p < 0.05) level 80% of the time. So if I were to run an experiment 100 times, I would find a significant difference between my 2 treatment groups 80 times out of the 100 times the experiment was run.
What value should the power of the test be? Well, similar to a p-value being set to 0.05, the power (β) of a test is often set to 80%. Please note that although many researchers use these values, you can set these to different values.
So , let’s say in our research projects, we are aiming for 80% power but may come a little shy, what can we do? More often than not, to increase the power of our test we would be necessary to increase the sample size, hence why the topics go hand in hand.
Factors that affect Sample Size
When determining sample size or the number of replicates for an experiment, there are 4 factors that will play a major role in determining either sample size and thereby the power of a test. These are:
- variance of your outcome variable
- size of the difference between (treatment) means
- significance level of the test
- power of the test
How do these affect sample size?
- as variance increases, sample size will increase
- as the size of the difference between means decreases or gets smaller, sample size will increase
- as the significance level of the test (α) decreases, sample size will increase
- as the power of the test (1 – β) increases, sample size will increase
Sample size calculations and/or power test calculations are dependent on the experimental design. Many of these can be calculated by hand using a variety of formulae, but there are also a myriad of programs available to help you out.
SAS has a PROCedure available along with a standalone program called SAS Power and Sample Size (available with the SAS installation, but this is a separate component of SAS) called PROC POWER. This procedure provides syntax for the following experimental designs:
- Logistic regression
- Multiple linear regression
- Fisher’s z test and t test of partial correlation
- Single binomial proportion
- One sample t test
- One way ANOVA
- McNemar’s test for paired proportions
- Paired t test
- Two independent proportions
- Two sample t test
- Two survival curves
- Two independent groups
During this session I would like to highlight the standalone package called SAS Power and Sample Size and work through a couple of examples to show you how to use it.
Example #1
We are designing an independent samples t-test to see whether males drink more milk than females. We need a number of pieces of information to determine the power of the test. We need the following:
- average milk consumed by males (used to calculate the difference between groups)
- average milk consumed by females (used to calculate the difference between groups)
- standard deviation of milk consumed (measure of variation)
- sample size
Average milk consumed by males = 8 glasses/day
Average milk consumed by females = 5 glasses/day
Standard deviation = 2
Sample size 5 males and 5 females
Power = ________
Example #2
We are now designing an ANOVA with 2×3 factorial design. We want to examine whether male and female dodo birds have different beek lengths when housed in a small, medium or large yard. We were able to obtain means and a standard deviation from a study conducted in Atlantis centuries ago and now we need to know how many dodo birds we need in order to obtain a power of 80%.
Based on the paper from Atlantis we have the following means:
Males in small yard: 12
Females in small yard: 8
Males in medium yard: 21
Females in medium yard: 21
Males in large yard: 46
Females in large yard: 39
Overall standard deviation reported in the paper was: 4
SASsy Fridays 