Deck 7: Making Sense of Statistical Significance: Effect Size and Statistical Power

Question 1

Some IQ tests have a standard deviation of 16 points. If an experimental procedure produced an increase of 3.2 IQ points, the effect size would represent a __________ effect size.

A)small
B)medium
C)large
D)extra large

Answer 1

small

Question 2

Which of the following is NOT a correct statement about effect size of a study finding:

A)It provides much information about statistical significance.
B)It is a standardized measure of lack of overlap between populations.
C)It increases with greater differences between means.
D)It can be converted to a standardized effect size.

Answer 2

It provides much information about statistical significance.

Question 3

Our general knowledge of what is a small or a large effect size can help the researcher:

A)evaluate the overall importance of a result
B)make better sense of statistical results
C)compare results with effect sizes found in other studies
D)all of the above

Answer 3

all of the above

Question 4

Reviews of a collection of studies on a particular topic that use meta-analyses represent an alternative to traditional __________ articles. These traditional articles describe and evaluate each study and then attempt to draw some overall conclusion.

A)general educational method
B)computer-assisted research
C)engagement goal setting
D)narrative literature review

Question 5

The effect size conventions proposed by Cohen are useful to researchers for:

A)predicting the value of the measured variable to use for the experimental condition
B)evaluating research results to determine if they are statistically significant
C)predicting the effect of a study on various populations
D)determining the power of a planned study

Question 6

In what way is effect size most comparable to a Z score?

A)It can range from-1 to +1
B)It provides a direct indication of statistical significance
C)It provides a standard for comparison for results across studies, even studies using different measures
D)All of the above

Question 7

Cohen has proposed some effect-size conventions based on the effects observed in psychology research in general because:

A)researchers frequently need to decide whether the effect size that they have found allows them to reject the null hypothesis
B)it is usually difficult to know how big an effect to expect from a given experiment
C)Cohen originally developed the relevant scales
D)they are more accurate than figuring a minimum meaningful difference

Question 8

In a research study with an effect size of .20, the populations of individuals have an overlap of about:

A)25%
B)45%
C)65%
D)85%

Question 9

The extent to which an experimental procedure separates the two populations of individual scores in a research study is the:

A)effect size
B)standard deviation of the mean
C)sampling distribution of the mean
D)statistical significance

Question 10

A standard verbal memory test is known to have a standard deviation of 10 points. If an experimental procedure produced an increase of 8 points, the effect size would represent a __________ effect size.

A)small
B)medium
C)large
D)unable to determine without additional information

Question 11

Standard rules about what to consider a small, medium, and large effect size based on what is typical in behavioral and social science research are known as:

A)effect size conventions
B)mean difference conventions
C)Cohen's conventions
D)both A and C

Question 12

(Population 1M - Population 2M)/Population SD is a formula for figuring:

A)power
B)effect size
C)the variance of the distribution of means
D)the combined standard deviation

Question 13

One important advantage of using effect sizes is that:

A)they tell us all the relevant parameters of a study's group of scores
B)they directly tell us the statistical significance of the research hypothesis
C)they are basically the same as the level of statistical significance in an experiment, but they allow us to create meta-studies
D)they are a standardized score, making study-to-study comparisons easier

Question 14

Effect size is a measure of:

A)the difference between individual members of a sample
B)the extent to which two populations overlap
C)the extent to which two populations do not overlap
D)the statistical significance of a research study

Question 15

According to Cohen's effect size conventions for mean differences, the following represent small, medium, and large effects, respectively:

A) .5, .8, 1.0
B) .4, .75, .90
C) .2, .3, .5
D) .2, .5, .8

Question 16

A statistical method for combining effect sizes from different studies is known as:

A)combination analysis
B)comparison analysis
C)multivariate analysis
D)meta-analysis

Question 17

A biologist is interested in the effects of cross-species interactions on aggression. He combines the results of many studies to determine an overall average effect size. What statistical procedure should he use?

A)Z test
B)t test
C)confidence intervals
D)meta-analysis

Question 18

According to Cohen's conventions, for research that compares means, a large effect size in which only about 53% of the populations of individuals overlap would be:

A) .5
B) .6
C) .7
D) .8

Question 19

Jacob Cohen introduced to behavioral and social scientists some very important statistical tools that included all of the following EXCEPT:

A)linear algebra
B)power analysis
C)effect size
D)sophisticated uses of regression analysis

Question 20

In order to calculate the standardized effect size, one should divide the raw score effect size for each study by its:

A)respective population standard deviation
B)respective population mean
C)respective estimate of population variance
D)sample size (N)

Question 21

Sample size affects power because the larger the sample size:

A)the larger the standard deviation of the distribution of means
B)the smaller the standard deviation of the distribution of means
C)the larger the sample's mean Z score
D)the smaller the sample's mean Z score

Question 22

A researcher may not be able to change the effect size of a planned study to increase power. Another aspect of a planned study that the researcher can usually change to increase power is:

A)the sample size
B)the beta level
C)the population parameters
D)the sample mean

Question 23

One way to increase the power of a planned study by altering the amount of variance within each of the two distributions is to:

A)use the variance of the experimental group instead of the variance from the comparison distribution
B)plan to exclude any scores in the sample that are not outliers
C)use more precise measures
D)reduce the number of items on any questionnaires employed

Question 24

Even if the research hypothesis is true, an experiment may not produce significant results because:

A)sampling error and other uncontrolled variables may cause too little variance in the population
B)the sample that happens to be selected from the population being studied may not be extreme enough to provide a clear case for rejecting the null hypothesis
C)the original population is very extreme with regard to its mean and standard deviation, leading the researcher to prove the null hypothesis
D)just as there is a probability of rejecting the null hypothesis, there is always a chance that it must be accepted

Question 25

Which of the following is not a direct influence on power?

A)significance level
B)one-versus two-tailed tests
C)type of hypothesis-testing procedure
D)type of statistical software package

Question 26

In the situation where the null hypothesis is true, the population distribution from which the sample was taken is:

A)the same as the known population's distribution
B)to the right of the original population distribution
C)to the left of the original population distribution
D)either to the right or left of the original distribution, depending on whether the mean of the sample is higher or lower than the mean of the original population

Question 27

Standard power tables are useful for:

A)directly determining the power of an experiment
B)determining the predicted score (but not the variance)for the group exposed to the experimental manipulation
C)determining the predicted effect size of a proposed experiment
D)determining the probability of falsely accepting the research hypothesis

Question 28

Using a two-tailed test makes it __________ to get significance on any one tail. Thus, keeping everything else the same, power __________ with a two-tailed test than with a one-tailed test.

A)easier; more
B)harder; less
C)easier; less
D)harder; more

Question 29

Having a very small amount of overlap between the distribution of population 2 and the predicted distribution of population 1:

A)reduces the power
B)increases the power
C)does not affect the power
D)indicates that the two cannot be effectively compared

Question 30

In the meta-analysis described in your text (by Eppley, Abrams, and Shear)on the effect of relaxation techniques on anxiety levels, the authors found that:

A)all relaxation techniques yielded similar effects
B)there were clear differences in the effect sizes associated with different relaxation and meditation procedures
C)progressive relaxation was the most effective technique
D)relaxation techniques do not affect anxiety levels significantly

Question 31

The __________ of a research study is the probability that it will produce a statistically significant result if the research hypothesis is true.

A)effect size
B)cutoff score
C)statistical power
D)critical region

Question 32

The effect size and the number of participants are two important determinants of:

A)the minimum significant result
B)reliability
C)power
D)alpha

Question 33

It is useful to understand statistical power for which of the following reasons?

A)Determining the number of participants to use in an experiment
B)Making sense of findings in research articles
C)Understanding the implications of a study that is not statistically significant
D)All of the above

Question 34

In theory, one could increase the power of planned study by:

A)increasing the effect size by increasing the predicted difference between population means
B)increasing effect size by decreasing the population standard deviation
C)using a one-tailed test
D)all of the above

Question 35

What effect will using a one-tailed test over a two-tailed test have on power (presuming the true population difference is in the expected direction)?

A)it will increase power
B)it will have no effect on power
C)it will decrease power
D)power cannot be calculated if a one-tailed test is used

Question 36

Effect size is one of the two major factors that contribute to power. Another factor is:

A)the sample's standard deviation
B)the minimum meaningful difference
C)the sample size
D)the mean of the known population

Question 37

In actual practice, the usual reason for determining power before conducting a study is to:

A)eliminate the possibility that a mistake may occur
B)ensure that regardless of whether the research hypothesis is true, the experiment will yield a significant result
C)determine the number of participants needed to have a reasonable chance of getting a significant result if the research hypothesis is true
D)recognize the likelihood that the experiment will need to be repeated

Question 38

When a study has only a small chance of being significant even if the research hypothesis is true, the study is said to have:

A)low power
B)low probability
C)low market value
D)low sample size

Question 39

If the research hypothesis is true, but the study has a low level of power:

A)there is a high probability that the study will have a significant result
B)the probability of getting a significant result is low
C)the null hypothesis will almost certainly be rejected
D)the significance level selected is probably too lenient (for example, .10 instead of .05)

Question 40

If statistical power for a given research study is .40, one can say that: "Assuming the researcher's prediction is correct, the researcher has a __________ chance of attaining statistically significant results."

A)20%
B)40%
C)45%
D)80%

Question 41

One way of understanding __________ is in terms of the degree of separation between populations due to the experimental variable.

Question 42

The __________ of a study is the difference in the population means divided by the population SD.

Question 43

According to Cohen, for an experiment to be worth conducting, it should have a level of power of about:

A)1%
B)5%
C)80%
D)near 100% (about 98% or higher)

Question 44

If the results of a study are not statistically significant and the sample size is large, then:

A)the result is very important
B)the result proves the null hypothesis
C)the research hypothesis is probably false
D)the result proves the research hypothesis

Question 45

The __________ is a statistic that combines two of the main influences on power, the expected difference between populations and the population standard deviation.

Question 46

For a particular research study, simply knowing about __________ does not provide much information about the size of the effect.

Question 47

When judging a study's results, there are two important questions. They are:

A)How large is the power and how competent are the researchers?
B)How stringent is the significance level and how small is the effect size?
C)Is the result statistically significant and is the effect size large enough for the results to be meaningful?
D)Is the study replicable and can we draw conclusions despite not having attained statistical significance?

Question 48

Knowing the effect size of a study lets you compare results with effect sizes found in other studies, but only when the other studies have equal sample sizes. (True or False)

Question 49

If a study with only a few participants manages to be statistically significant, it probably had:

A)an interesting finding
B)a small effect size
C)a large effect size
D)a large sample size

Question 50

What is the most likely explanation for why a study with a very small effect size came out significant?

A)the study had a large sample size
B)the study had a large population standard deviation
C)the researcher used an insensitive hypothesis-testing procedure
D)the researcher used a two-tailed test

Question 51

One way of decreasing the standard deviation of the distribution of means in a planned study, thereby increasing power, is to have a larger number of participants. Another would be to:

A)use a more stringent alpha level
B)use the sample variance in place of the variance of the population
C)change the minimum meaningful difference.
D)use a less diverse population

Question 52

In order to convert a raw score effect size to a standardized effect size, one should:

Question 53

Effect size is a standardized measure of difference (lack of overlap)between populations. (True or False)

Question 54

Effect size is an important tool for making sense of research results because:

A)it tells us the degree to which a result is statistically significant
B)it tells us the magnitude of the statistical effect
C)it limits variance and thus increases power
D)it is always smaller than the variance

Question 55

Practical significance is a combination of statistical significance and:

A)effect size
B)the level of measurement (whether it is equal interval or ordinal)
C)the population parameters
D)the amount over or under that level that the sample scored

Question 56

The effect size of a study is a good indicator of its practical importance because, unlike statistical significance, it is not affected by __________.

Question 57

What does it mean to say "A between-groups effect size of .6 was achieved for the treatment group as compared to the no-treatment group"?

A)The group getting the treatment was .6 standard deviations higher on the outcome.
B)The group getting the treatment was .3 standard deviations higher on the outcome.
C)Both groups improved, but the no-treatment group showed a greater increase in scores.
D)All of the above.

Question 58

In statistics, we cannot state that the research hypothesis is ever definitely false. However, if one fails to reject the null hypothesis in a study with a high level of power, this allows us to:

A)suspect that the research hypothesis may still be true
B)conclude that the research hypothesis is most likely false
C)make no statements about the research hypothesis
D)reject the notion that the effect size has anything to do with statistical significance

Question 59

Effect Size = (Population 1M - Population 2M)/__________.

Question 60

Changing the procedures of a study so that it decreases the __________ of the population increases the effect size.

Question 61

Power is affected by sample size, effect size, whether a one- or two-tailed test is used, __________ , and type of hypothesis-testing procedure used.

Question 62

A researcher would use a __________ to combine results of several different studies, based on their effect sizes.

Question 63

If the research hypothesis is true, an experiment will automatically yield a significant result. (True or False)

Question 64

Using a two-tailed test makes it harder to get significance on any one tail. Thus, power is less with a two-tailed test than with a one-tailed test. (True or False)

Question 65

The word significant in statically significant means the researcher can be fairly confident that there is some real effect. It does not, however, mean that the effect is significant in a __________ sense.

Question 66

Aside from sample size, other influences on statistical power include __________ , __________ , and __________.

Question 67

More extreme significance levels (.001, .0001)mean greater statistical power. (True or False)

Question 68

According to Cohen, a good general rule for when it is worthwhile to conduct a study is when power is at least __________.

Question 69

Cohen's effect-size conventions for a small, medium, and large effect are __________ , __________ , and __________ , respectively.

Question 70

If the difference between population means is 10 and the population SD is 5, according to Cohen's conventions, there is a(n)__________ effect size.

Question 71

In studies using a very large number of participants, it is common to get statistically significant results that have very small __________.

Question 72

Increasing sample size __________ statistical power.

Question 73

A nonsignificant result from a study with low power is truly inconclusive. However, a nonsignificant result from a study with high power does suggest either that the research hypothesis is false or that there is less of an effect than was predicated when figuring power. (True or False)