Deck 6: Reliability and Validity in Measurement and Research

Question 1

If a test consists of questions that reflect the kinds of material of interest to the researcher, then the test is said to have good ____________ validity.

A) criterion
B) construct
C) convergent
D) content

Answer 1

D

Question 2

To evaluate whether a set of scientific observations has been replicated, one can compare the __________ of the different sets of observations.

A) validity
B) test-retest reliability
C) effect size
D) internal consistency

Answer 2

C

Question 3

Caroline administers a scale to a group of students and then asks them to complete the scale again a few days later. Which type of reliability is Caroline trying to assess?

A) test-retest
B) internal-consistency
C) inter-judge
D) convergent

Answer 3

A

Question 4

To minimize the potential that scores on an instrument may be artificially inflated after people take the same test a second time, researchers sometimes assess __________ reliability rather than test-retest reliability.

A) internal-consistency
B) split-half
C) alternative-form
D) item-to-item

Question 5

A researcher constructs a five-item measure of attitudes toward national health insurance. The average intercorrelations among the items is r_ii = .40. Using the Spearman-Brown formula, he finds that R^SB = .77. What is the reliability of the scale as a whole?

A) .77
B) .50
C) .40
D) Cannot be determined without knowing the sample size.

Question 6

Which of the following is used to describe the judge-to-judge reliability?

A) the Pearson r correlation between each judge
B) the mean of all the Pearson r correlations between all pairs of judges
C) Cronbach's alpha coefficient
D) effect size

Question 7

An instrument that gives essentially the same results each time it is used, regardless of who administers it, is said to be high in

A) accuracy.
B) face validity.
C) predictive validity.
D) reliability.

Question 8

The Spearman-Brown formula is used to determine the __________ reliability of an instrument.

A) item-to-item
B) internal-consistency
C) test-retest
D) alternative-form

Question 9

What is face validity?

A) whether the instrument seems on the surface (or "face") to be measuring something relevant.
B) whether the instrument can generalize a causal relationship across persons and settings
C) the degree to which a measure correlates with other outcome measures
D) the "ability to discriminate"

Question 10

Jim is frustrated with a class because the questions on the most recent exam did not reflect the material that was discussed in class or in the text. The source of his frustration was this exam's apparent lack of

A) content validity.
B) construct validity.
C) divergent validity.
D) convergent validity.

Question 11

Which of the following refers to the consistency or stability of a measurement?

A) reliability
B) bias
C) validity
D) fluctuation

Question 12

One must question an exam's __________ if it seems to measure more of one's ability to guess accurately than what one has learned in the class.

A) reliability
B) accuracy
C) validity
D) bias

Question 13

David is afraid that the results from his experiment may not be able to be replicated in real life situations. David is concerned that his experiment lacks which type of validity?

A) external
B) predictive
C) construct
D) internal

Question 14

Margaret calculates the average inter-item correlations for her scale and then uses the Spearman-Brown formula for her final calculations. Margaret is calculating the __________ reliability of the instrument.

A) alternative-form
B) internal-consistency
C) test-retest
D) item-to-item

Question 15

The degree of relatedness of the individual items on a test is the focus of __________ reliability.

A) test-retest
B) judge-to-judge
C) internal-consistency
D) alternative-form

Question 16

__________ errors tend to cancel out over the long run whereas __________ do not.

A) Random; systematic
B) Systematic; random
C) Bias; systematic
D) Bias; random

Question 17

Which type of reliability is concerned with the temporal stability of an instrument?

A) split-half
B) test-retest
C) internal-consistency
D) Spearman-Brown

Question 18

Which of the following refers to how well a measure or research design does what it purports to do?

A) validity
B) bias
C) reliability
D) error

Question 19

Which of the following refers to the ability to generalize a causal relationship across variations in persons, settings, treatments, and outcomes?

A) internal validity
B) statistical conclusion validity
C) external validity
D) construct validity

Question 20

Dennis consistently sets his watch 10 minutes ahead of the actual time. This is an example of

A) random error.
B) systematic error.
C) reliability.
D) validity.

Question 21

When one's research purpose is to make causal generalizations, one must be concerned with the __________ validity of how the variables are conceptualized.

A) face
B) construct
C) internal
D) content

Question 22

What is the difference between validity and reliability? Why are they both of concern to researchers?

Question 23

What is the purpose of test-retest reliability and alternate-form reliability? In what types of situations would one use each type of reliability?

Question 24

Evaluating whether an instrument correlates highly with similar measures as well as whether the instrument does not correlate well with dissimilar measures is to determine which type of validity?

A) content
B) criterion
C) convergent
D) construct

Question 25

After analyzing his data, a researcher concludes that there was no relationship between the independent and dependent variables in his study. However, he is concerned with whether or not his conclusions may be erroneous. This researcher is concerned with which type of validity?

A) internal
B) statistical-conclusion
C) convergent
D) construct

Question 26

To assess the validity of a college admissions test, a researcher will correlate the test scores with the students' first semester grade-point average. Which type of validity is the researcher investigating?

A) concurrent
B) discriminant
C) convergent
D) predictive

Question 27

When students are upset because they perceive an exam as being "unfair," they are most likely concerned with the test's __________ validity.

A) content
B) criterion
C) construct
D) predictive

Question 28

Which type of validity refers to whether a measure correlates highly with different measures of the same trait or behavior?

A) convergent
B) divergent
C) concurrent
D) content

Question 29

The ability to rule out plausible rival hypotheses in an experiment refers to which type of validity?

A) internal
B) construct
C) external
D) statistical conclusion

Question 30

Using a future criterion to evaluate the criterion validity of an instrument is called

A) predictive validity.
B) concurrent validity.
C) discriminant validity.
D) content validity.

Question 31

Convergent and discriminant validity are aspects of which type of validity?

A) content
B) construct
C) concurrent
D) criterion

Question 32

What is internal-consistency reliability? Why would one use the Spearman-Brown formula to assess this type of reliability?

Question 33

Sam is developing an instrument that would identify those students who may be at risk for flunking out of school. After administering this test to a group of students, he compares the results of the test with their current classroom performance. Sam is most likely trying to evaluate the instrument's ____________ validity

A) discriminant
B) concurrent
C) content
D) predictive

Question 34

After the completion of an experiment, a researcher realizes that there is another explanation for the results other than the original hypothesis. The researcher realizes that this experiment is deficient with respect to which type of validity?

A) construct
B) external
C) internal
D) statistical conclusion

Question 35

The degree to which a measure correlates with other outcome measures is called

A) discriminant validity.
B) construct validity.
C) criterion validity.
D) content validity.

Question 36

How can the Spearman-Brown formula be useful in determining how long an instrument should be or how many judges one should use in a study?

Question 37

Construct validity of an instrument is established by examining an instrument's __________ and __________ validity.

A) concurrent; predictive
B) concurrent; divergent
C) convergent; predictive
D) convergent; divergent

Question 38

Which type of validity refers to whether a measure distinguishes between related but conceptually distinct behaviors or traits?

A) predictive
B) construct
C) convergent
D) discriminant

Question 39

What is the difference between random and systematic error? Which error represents more of a threat to the nature of the conclusions that can be drawn from a study? Why?

Question 40

The "ability to discriminate" is a characteristic of which type of validity?

A) criterion
B) convergent
C) construct
D) content

Question 41

Which two types of validity are concerned with the strength or weakness of one's causal inferences? Why?

Question 42

Which two types of validity are concerned with whether or not one's causal inferences are correct? Why?

Question 43

What is construct validity in test development? What type of evidence does one use in the construct validation of an instrument?

Question 44

What is statistical-conclusion validity?

Question 45

What is meant by external validity? How might one try to demonstrate the external validity of a study?

Question 46

What is internal validity? Why is it important in trying to identify causal relationships?

Question 47

What is content validity? How might one try to ensure that an instrument has good content validity?

Question 48

What is criterion validity? What role do concurrent and predictive validity have in determining criterion validity?

Question 49

How can effect sizes be used to evaluate whether the results from a study have been replicated?

Question 50

Why would experimenters be concerned with construct validity when they try to make causal generalizations?