Deck 6: Estimating Reliability

Question 1

The degree to which participant learning over time can impact a reliability estimate reflects variance in ________.

A) scoring
B) true score equivalence
C) test administration
D) true score fluctuation

Answer 1

D

Question 2

Error variance due to ________ may result in overestimating a reliability coefficient.

A) scoring
B) memory effects
C) guessing
D) test administration

Answer 2

B

Question 3

Internal consistency reliability estimates attempt to address issues commonly encountered in ________ reliability.

A) parallel forms
B) test-retest
C) inter-rater

Answer 3

A

Question 4

________ is the average of all possible split-half reliability estimates.

A) Inter-rater reliability
B) Inter-rater agreement
C) Spearman-Brown
D) Coefficient alpha

Question 5

________ refers to the degree to which raters provide the same ratings across observed events.

A) Inter-rater reliability
B) Inter-rater agreement
C) Spearman-Brown
D) Coefficient alpha

Question 6

If we are interested in capturing the relationship between scores provided by two raters, we would compute ________ .

A) Inter-rater reliability
B) Inter-rater agreement
C) Spearman-Brown
D) Coefficient alpha

Question 7

Cohen's kappa can be computed when two raters provide input on a variable at the ________ level of measurement.

A) nominal
B) ordinal
C) interval
D) ratio

Question 8

If we used a test's standard error of measurement (SEM) to compute a 95% confidence interval around a person's test score, we would be 95% certain we have captured the ________ .

A) amount of error associated with the test
B) score the test-taker would receive on a second administration of the test
C) test-taker's true score
D) variability across test scores

Question 9

While most reliability estimates are computed using a bivariate correlation coefficient, the computation of ________ is more complex.

A) parallel forms reliability
B) split half reliability
C) Coefficient alpha
D) inter-rater reliability

Question 10

Explain how each of the following sources of measurement error may influence the reliability estimate:
(a) guessing
(b) true score fluctuation, and
(c) test administration procedures

Question 11

Identify two concerns with computing a split-half reliability estimate. How might these concerns be addressed?

Question 12

In a pilot study of a new measure, a researcher found that the test's Coefficient alpha was just less than desired. Identify two ways the researcher could modify the test to improve the test's reliability.

Question 13

Do measures of internal consistency reliability provide evidence of the unidimensionality of a measure? Explain.

Question 14

Explain why the best practice would be to report more than one type of reliability estimate for a newly developed psychological measure.