Deck 8: Criterion-Related Validation

Question 1

In a ________ criterion-related validation study, scores on a criterion of interest are collected prior to obtaining test scores.

A) predictive
B) postdictive
C) conclusive
D) concurrent

Answer 1

B

Question 2

________ criterion-related validation designs typically provide the greatest freedom in obtaining samples.

A) Predictive
B) Postdictive
C) Conclusive
D) Concurrent

Answer 2

A

Question 3

________ criterion-related validation designs require considerably more time before drawing a conclusion about the experimental measure.

A) Predictive
B) Postdictive
C) Conclusive
D) Concurrent

Answer 3

A

Question 4

Using Cohen's (1988) guidelines to evaluate the magnitude of correlation-based effect sizes, an obtained criterion-related validity coefficient of .52 would be considered ________.

A) essentially non-existent
B) small
C) moderate
D) large

Question 5

To improve prediction over random selection, the validity coefficient must be at least of ________ magnitude.

A) essentially non-existent
B) small
C) moderate
D) large

Question 6

_______ percent of the variance in a criterion would remain unexplained if the criterion-related validity of a test was found to be .30.

A) 9
B) 30
C) 70
D) 91

Question 7

A small organization which has too few employees to conduct a full-scale criterion-related validation study may wish to employ the use of ________ validity.

A) generalized
B) postdictive
C) synthetic
D) Retrospective

Question 8

Construct irrelevant variance is closely tied to ________.

A) attenuation due to unreliability
B) restriction of range
C) criterion contamination
D) inadequate sample size

Question 9

Roth et al. (2017) found that the use of job incumbent samples to estimate the amount of variability in predictor scores in the population results in an ________ of the true criterion-related validity.

A) underestimation
B) accurate estimation
C) overestimation

Question 10

The correction of artifacts impacting the predictor in a criterion-related validity study would likely ________ the estimate of criterion-related validity.

A) artificially increase
B) artificially decrease
C) improve the accuracy of
D) not impact

Question 11

Explain why the choice of a criterion is crucial to criterion-related validation.

Question 12

(a) Identify one advantage of utilizing a concurrent or postdictive criterion-related validation designs compared to predictive criterion-related studies.
(b) Identify two disadvantages of utilizing a concurrent or postdictive criterion-related validation study compared to predictive criterion-related studies.

Question 13

Consider a scenario in which a highly selective school wanted to improve its selection of prospective students. A consultant developed a new predictor with an internal consistency reliability of .75. The consultant then correlated the new measure with a criterion of student success, and found that the resulting correlation was .39. Prior to reporting the results, the consultant considered whether corrections for attenuation due to unreliability should be performed in the predictor, the criterion, neither, or both. Provide a recommendation explaining how the consultant should proceed.

Question 14

(a) Explain why restriction in range in a predictor can be an issue for demonstrating criterion-related validity.
(b) What is the biggest challenge in mathematically correcting for the problem of restriction in range?

Question 15

Criterion-related validation studies require sufficiently large sample sizes.
(a) What common research concerns might lead you to doubt the results of a criterion-related validation study that utilized a small sample?
(b) What minimal size sample was suggested by Schmidt, Hunter and Urry (1976) for conducting criterion-related validation studies?