[A, SfS] Chapter 6: Hypothesis Testing: 6.7: Computing the Power of a Hypothesis Test
Computing the Power of a Hypothesis Test
Computing the power of a hypothesis test
For a specified effect size in a one-sided hypothesis test, what is the power of the test?
#\phantom{0}#
Power
The power #\pi# of a hypothesis test when #\alpha# is specified is the probability of concluding in favor of #H_1# when #H_1# is actually true.
It is therefore the sensitivity of the test, i.e., its long-term success rate at detecting departures from the null hypothesis. It is also the complement of the probability of a Type II error. All of the tests we have described have been proven to be uniformly most-powerful tests, i.e., no test can have a higher power in each of the given settings than the one presented.
The power of a parametric test can only be computed for specific values of the parameter. Typically, a researcher has in mind a specific effect size #\eta# which would be regarded as meaningful in a particular setting. Scientists will often have a desired minimum power that they would like to achieve for detecting a given effect size, such as #80\%#.
For example, when testing #H_0: \mu \leq \mu_0# against #H_1: \mu > \mu_0# at a significance level #\alpha# when #\sigma^2# is known, a researcher might be particularly interested in values of #\mu# that satisfy #\mu - \mu_0 \geq \eta# for some effect size #\eta > 0#.
Thus the researcher would want to know the power of the test when #\mu = \mu_0 + \eta#. The power would get larger if #\mu > \mu_0 + \eta#, so this would give a lower bound on the power.
This test concludes in favor of #H_1# if \[Z = \cfrac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} \geq z_{\alpha}\] i.e., if \[\bar{X} \geq z_{\alpha}\cfrac{\sigma}{\sqrt{n}} + \mu_0\]
If #\mu = \mu_0 + \eta#, then: \[\bar{X} \sim N\Big(\mu_0 + \eta,\cfrac{\sigma^2}{n}\Big)\] And the power of the test is the corresponding probability: \[P\Big(\bar{X} \geq z_{\alpha}\cfrac{\sigma}{\sqrt{n}} + \mu_0\Big)\]
This translates (in this example only) to: \[\text{Power} =P\Big(Z \geq z_{\alpha}-\cfrac{\eta\sqrt{n}}{\sigma}\Big)\] where #Z \sim N(0,1)#.
#\text{}#
The below figure shows the normal density curve describing the distribution of the sample mean when #\mu = \mu_0# (on the left) and the same curve when #\mu = \mu_0 + \eta# (on the right), where in both places the standard error is #\sigma/\sqrt{n}#. The power of the test is the shaded area under the second density curve after #z_{\alpha}#.
A power curve shows how the power increases (toward a maximum of 1) as #\eta# increases:
We can see that, in this case, we achieve a power of #80\%# when the effect size is approximately #4#. A larger sample size would mean that we would achieve #80\%# power for a smaller effect size, because the curve would rise much more steeply. This would also happen if we have a smaller standard deviation, but we have no control over that. We can also increase the power for a given effect size by choosing a larger significance level, since this makes it easier to reject the null hypothesis.
We can develop a similar procedure for testing #H_0: \mu \geq \mu_0# against #H_1 : \mu < \mu_0# at significance level #\alpha# when #\sigma^2# is known. Likewise, we can extend this to the other types of hypothesis tests we have described.
#\text{}#
Example:
If we test #H_0: \mu \leq 100# against #H_1: \mu > 100# at level #\alpha = 0.04#, where we are investigating a variable #X# which has a #N\Big(\mu,8^2\Big)# distribution on the population using a sample of size #n = 16#, what is the power of the test if the effect size is #\eta = 4#?
Solution:
We reject #H_0# if \[\cfrac{\bar{X} - 100}{8/\sqrt{16}} = \cfrac{\bar{X} - 100}{2} \geq z_{0.04} = 1.75\] i.e., if \[\bar{X} \geq 2(1.75) + 100 = 103.5\]
If the effect size is #4# then the true mean is #104#.
In this case, #P(\bar{X} \geq 103.5)# is computed in #\mathrm{R}# using
> pnorm(103.5,104,2,low=F)
to be #0.5987#. So the power of the test is about #60\%#.
Example:
If we test #H_0: \mu \geq 50# against #H_1: \mu < 50# at level #\alpha = 0.01#, where we are investigating a variable #X# which has a #N(\mu,18^2)# distribution on the population using a sample of size #n = 36#, what is the power of the test if the effect size is #\eta = 5#?
Solution:
Now we have a one-sided test in the opposite direction, so we make some adjustments to the procedure.
In this setting we reject #H_0# if \[\cfrac{\bar{X} - 50}{18/\sqrt{36}} = \cfrac{\bar{X} - 50}{3} \leq -z_{0.01} = -2.326\] i.e., if \[\bar{X} \leq 3(-2.326) + 50 = 43.022\]
If the effect size is 5 then the true mean is #50 - 5 = 45#.
In this case, #P(\bar{X} \leq 43.022)# is computed in #\mathrm{R}# using
> pnorm(43.022,45,3)
to be #0.2548#. So the power of the test is #25.48\%#.
Example:
Now suppose we test #H_0: p \geq 0.4# against #H_1: p < 0.4# at level #\alpha = 0.05#, where we are investigating a binary variable #X# which has a #B(1,p)# distribution on the population using a large sample of size #n = 96#.
What is the power of the test if the effect size is #\eta = 0.03#?
Solution:
Again we have a one-sided test in the left tail, but now the parameter is #p#.
In this setting we reject #H_0# if \[\cfrac{\hat{p} - 0.4}{\sqrt{\cfrac{0.4(1-0.4)}{96}}} = \cfrac{\hat{p} - 0.4}{0.05} \leq -z_{0.05} = -1.645\] i.e., if \[\hat{p} \leq 0.05(-1.645) + 0.4 = 0.31775\]
If the effect size is #0.03# then the true proportion is #0.4 - 0.03 = 0.37#. But this would mean that the variance of #\hat{p}# changes to \[\cfrac{0.37(1 - 0.37)}{96} \approx 0.002428\]
In this case, #P(\hat{p} \leq 0.31775)# is computed in #\mathrm{R}# using
> pnorm(0.31775,0.37,sqrt(0.002428))
to be #0.1445#. So the power of the test is #14.45\%#.
Or visit omptest.org if jou are taking an OMPT exam.
