Deck 19: Synthesizing Research Results

Question 1

Which are considered shortcomings of narrative literature reviews? (circle all that apply)

A) failure to comprehensively review the existing knowledge base
B) lack of clearly stated rules for inclusion and exclusion of studies
C) lack of useful information and therefore should never be conducted
D) failure to use statistical metrics to combine findings across studies objectively

Answer 1

A,B,D

Question 2

In this type of search procedure to obtain primary studies for a research synthesis, an important early and relevant primary study is identified and used as the starting port for all chronologically later studies that have cited it:

A) ascendancy searching
B) descendancy searching
C) systematic searching
D) relevancy searching

Answer 2

B

Question 3

The following tends to be true of published vs unpublished studies:

A) published studies and unpublished studies tend to show the same effect sizes
B) published studies tend to show larger effect sizes than unpublished studies
C) published studies tend to show smaller effect sizes than unpublished studies
D) published studies tend to show larger effect sizes if conducted by students

Answer 3

B

Question 4

In terms of inclusion and exclusion criteria for primary studies to be included in a meta-analysis, the textbook recommends:

A) exclude studies that are methodologically poor, as these effects will contaminate the summary effect
B) exclude studies that are methodologically poor, and switch to vote counting
C) include studies that are methodologically poor, but perform study-level moderation analysis of rigorously vs. poorly designed studies
D) include studies that are methodologically poor, but perform study-level mediation analysis of rigorously vs. poorly designed studies

Question 5

According to conventions recommended by Cohen to describe the strength of d, the following is considered a medium effect:

A) d = 0.20
B) d = 0.50
C) d = 0.80
D) d = 1.00

Question 6

A primary study evaluated the effectiveness of a therapy designed to treat graduate students suffering from clown phobia. On the dependent measure of attraction toward clowns, the mean score for the treatment group is 5, whereas the mean score for the control group is 4. The pooled standard deviation is 2. Cohen's d is:

A) 5
B) 4
C) 2
D) 0.5

Question 7

Which of the following are considered metrics of effect size? (circle all that apply)

A) correlation
B) odds ratio
C) T test
D) Cohen's d

Question 8

If two variables each contain dichotomous response options, the following correlation should be used to examine the strength of their relationship:

A) Pearson correlation
B) point-biserial corelation
C) Fisher's r
D) Phi

Question 9

The "file drawer problem" is the tendency for unpublished studies to:

A) be ignored in a meta-analysis
B) lack appropriate information to calculate the effect size
C) be methodologically flawed
D) yield a larger effect size

Question 10

In a meta-analysis, the effect sizes of 20 primary studies are shown to be distributed heterogeneously. The following model is recommended to calculate the summary effect.

A) unweighted model
B) fixed-effect model
C) mixed model
D) random-effects model

Question 11

A meta-analysis evaluated the relationship between attitudes and behavior toward exercising. To illustrate the practical importance using the summary effect, a binomial effect size display (BESD) was constructed. What is the magnitude of the summary effect?

A) r = 0.20
B) r = 0.40
C) r = 0.60
D) r = 0.80

Question 12

Which of the following is true of the vote counting method? (circle all that apply)

A) it is methodologically superior to meta-analysis
B) the number of statistically significant studies is counted
C) the number of non- statistically significant studies is counted
D) the decision about whether a theoretical link is tenable is made on the basis of number of significant vs. non-significant studies

Question 13

Which of the following limitations with narrative literature reviews are meta-analyses intended to address or avoid?

A) often, narrative reviews do not provide a comprehensive knowledge base of relevant literature
B) there are not clearly defined criteria for what studies are included or excluded from a narrative review
C) narrative reviews do not take into account differences in methodologies or theoretical approaches to the same hypothesis or research question
D) A and B
E) B and C
F) all of the above

Question 14

At the outset of a meta-analysis (i.e., when determining what studies will be included in the analyses), what are the primary considerations a researcher must address? (circle all that apply)

A) decide whether to include moderators in the analysis
B) specify tentative rules for inclusion and exclusion in analyses
C) locate every study that fits the (tentative) inclusion criteria
D) determine whether to use a fixed-effects or random-effects model

Question 15

In locating all applicable studies for a meta-analysis, descendancy searching refers to ____(a)_____, and ascendancy searching refers to _____(b)_____.

A) searching articles in descending order, with most recent articles first (i.e., the descendants); searching articles in ascending order, with earliest articles first (i.e., the ancestors)
B) finding seminal articles that are applicable to the meta-analytic hypothesis, and then finding all subsequent articles that cite this seminal article (i.e., the descendants); finding very recent major studies relevant to the meta-analysis and then using the references of this recent article to find earlier seminal studies (i.e., the ancestors)
C) finding very recent major studies relevant to the meta-analysis (i.e., the descendants) and then using the references of this recent article to find earlier seminal studies; finding seminal articles that are applicable to the meta-analytic hypothesis (i.e., the ancestors), and then finding all subsequent articles that cite this seminal article
D) searching articles in ascending chronological order (i.e., starting with the ancestors); searching articles in descending order (i.e., starting with the descendants)

Question 16

Which of the following statistics would be considered an effect size for use in meta-analyses?

A) an odds-ratio
B) the p-value for a study
C) the standardized mean difference between two experimental groups
D) A and B
E) A and C
F) all of the above

Question 17

Which of the following statements is NOT true of fixed-effects meta-analysis models?

A) studies with larger sample sizes are weighted heavier because they are thought to reflect closer approximation to the "true score"
B) fixed-effects models can be generalized to studies that are not included in the meta-analysis
C) the distribution of effect sizes is thought to be homogeneous
D) differences in individual effect sizes are thought to reflect measurement error

Question 18

In a random-effects model, the summary effect size will be _____(a)_____ influenced by one extremely large sample; in a fixed-effects model, the summary effect size will be _____(b)_____ influenced by one extremely large effect size.

A) greatly; slightly
B) greatly; not at all
C) not at all; slightly
D) slightly; greatly

Question 19

In meta-analyses, which of the following examples would be considered methodological moderators? (circle all that apply)

A) whether the dependent variable was self-reported or via observational measure
B) whether the independent variable was operationalized differently across studies
C) whether socio-economic status influenced the relationship between A and B
D) whether the obtained relationship between A and B was causal or correlational

Question 20

A researcher wanted to provide a visualization of the practical significance of an obtained summary effect size. What technique could the researcher use to do so?

A) a bar graph
B) a binomial effect size display (BESD)
C) a bivariate summary effect diagram (BSED)
D) a bidirectional depiction of effect size (BDES)

Question 21

What are some of the reasons for why the correlation coefficient is often considered to be the preferred effect size to use in a meta-analysis? (circle all that apply)

A) it is easy to interpret
B) it is easy to compare (and aggregate) effect sizes from studies that used any combination of dichotomous and continuous measurements of variables
C) studies are more likely to report correlations than they are to report group differences
D) correlations do not take sample size into account while other effect size estimates do

Question 22

Researchers conducted a meta-analysis on the relationship between Z and X, and found a summary effect size of -0.08, p > 0.05, even though all of the individual effect sizes were highly significant. What could be potential reasons for this observed summary effect size?

A) they may have included too few primary studies in their meta-analysis, and thus their meta-analysis is underpowered
B) the distribution of effect sizes if very heterogeneous, with positive and negative individual effect sizes cancelling each other out
C) there may be an important moderator that is masking the true relationship between Z and X
D) A and B
E) B and C
F) all of the above

Question 23

In meta-analysis, weighting procedures are applied to synthesize the effects of primary studies for the purpose of calculating a summary effect. The three weighting procedures are unweighted model, fixed-effect model, and random-effects model. Conceptually describe how each model is calculated, and the assumptions made by each.

Question 24

A meta-analysis synthesized studies examining an independent variable (drug treatment vs. control) on levels of mental health (high vs. low). The summary effect is determined to be r = 0.30. To illustrate the practical importance of the effect, (a) create a binomial effect size display (BESD), (b) clearly label the display, and (c) interpret the display in one or sentences.

Question 25

What is the "file drawer problem?" Why is it a problem in meta-analysis? What approach has been recommended to determine the extent of the file drawer problem in a meta-analysis?

Question 26

After converting everything to the appropriate statistics, researchers obtained a summary effect size of r = 0.16. You are worried that this effect size is fairly small, and may not be interpreted as well as it could be. To help demonstrate the importance of this summary effect size, you create a BESD for the effect size. (a) Given r = 0.16, construct a BESD for this summary effect. (b) Interpret, and explain the implications of two of the cells in the BESD, to illustrate how this BESD can help emphasize the practical importance of even a small effect size.

Question 27

How might vote counting be impacted by the file drawer problem? Specifically, (a) describe what vote counting and the file drawer problem refer to, then (b) explain how and why the file drawer problem would or would not pose an issue for the results of a "vote counting."

Question 28

Crano, Brewer, and Lac suggest that random-effects models are preferable for most meta-analyses. Provide three reasons for why this is so. In other words, describe three advantages of random-effects models over other meta-analysis models, benefits of random-effects models in general, or any combination of advantages.

Question 29

Define each the following methods of research synthesis: narrative literature review, meta-analysis, and vote counting. Then, discuss the advantages and disadvantages of each.

Question 30

You are interested in conducting a meta-analysis on whether there is a consistent relationship that shows children's aggression and violence are related to watching violence on television. (a) Would you use a(n) unweighted, fixed-effects, or random-effects model to obtain a summary effect size? Explain and justify your choice with at least two reason for choosing your preferred model. (b) Provide three example moderators that might influence this relationship, one methodological moderator, one demographic moderator, and one theoretical moderator. (c) Finally, interpret what it would mean if you obtained the following results (in the table below).
Specifically, what does the summary effect size tell us, and what might be some of the explanations for the observed summary effect size?

Question 31

Describe the steps in conducting a meta-analysis. What requirements need to be met?

Question 32

What is the "file-drawer problem" and how can it influence the results of a meta-analysis?

Question 33

Describe the difference between fixed-effect models and random-effects models.