Deck 4: Measurement Validity

Question 1

"Measurement construct validity" includes which of the following types of validity? (circle all that apply)

A) face validity
B) content validity
C) convergent validity
D) criterion validity

Answer 1

A,B,C,D

Question 2

Two researchers wanted to determine if their new scale of people's aversion to technology was a valid measure of the construct. On the surface, the items seem to be related to distaste for technological advances. Additionally, in an initial cross-sectional study of the scale, they found that people who scored high on aversion to technology were unlikely to own smartphones, tablet computers, or an electronic navigation device (technology aversion and ownership of devices were measured at the same time). Moreover, they found that people's technology aversion score was unrelated to scores on preference for American versus imported cars, or attitudes about the current president. In this example, which of the following types of validity have been addressed? (circle all that apply)

A) convergent validity
B) concurrent validity
C) discriminant validity
D) predictive validity

Answer 2

B,C

Question 3

__________ is concerned with the extent to which items in a scale adequately reflect the breadth of the construct that is supposedly being measured.

A) content validity
B) criterion validity
C) face validity
D) convergent validity

Answer 3

A

Question 4

When discussing the validity of a scale in terms of its relationship to a theoretically related outcome, _____(a)_____ refers to the type of validity in which scale and outcome are assessed simultaneously, while _____(b)_____ refers to the type of validity in which the outcome is assessed some time after the scale.

A) predictive validity; concurrent validity
B) predictive validity; convergent validity
C) concurrent validity; predictive validity
D) convergent validity; concurrent validity

Question 5

It is not only important to examine the validity of the content of a scale, it is also important to examine how that scale is related to a target outcome or criterion that it should theoretically be related to. Specifically, this type of validity encompasses which of the following:

A) convergent validity
B) concurrent validity
C) outcome validity
D) A and B only
E) B and C only
F) all of the above

Question 6

In a study of political preferences, Delilah assessed Democrats' and Republicans' attitudes toward the current US President (a Republican), as well as attitudes about various hypothetical governmental policies. This example reflects what type of construct validation?

A) divergent validity
B) concurrent validity
C) convergent validity
D) predictive validity

Question 7

The multitrait-multimethod matrix approach is one technique for determining the validity of a target construct. Specifically, the MTMMM addresses what types of validity in its model?

A) discriminant validity
B) divergent validity
C) convergent validity
D) A and C only
E) B and C only
F) all of the above

Question 8

When using self-report measures, researchers risk having participants' responses be systematically influenced by factors that have nothing to do with the constructs of interest (or the constructs that the items are intended to measure). These "threats to measurement validity" include which of the following:

A) language difficulties
B) maturation
C) mood
D) A and B
E) A and C
F) all of the above

Question 9

Delroy Paulhus and colleagues (e.g., Paulhus & Reid, 1991) have conducted extensive research on the concept of self-deception (i.e., lying to oneself), including a scale called the Self-Deceptive Denial scale (as part of the larger BIDR inventory). These items assess the extent to which people are willing to lie to themselves by answering negatively on the scale, even though that the items on the scale are thought to reflect internal states that almost everyone has experienced at least once (e.g., I can't think of anyone I hate deeply; seeing an attractive person of the opposite sex makes me think about sex; I have never done anything I am ashamed of). By actually studying self-denial, Paulhus and colleagues are exploiting what threat to measurement validity?

A) language difficulty
B) mood
C) acquiescence
D) social desirability

Question 10

When developing a scale that assesses teenagers' preference for sweets, researchers wrote these three items, on a 1-10 scale (1 = extremely disagree, 10 = extremely agree):
I) Chocolate cake is my favorite dessert.
Ii) I would rather revel in confectionery than savor mincemeat.
Iii) I am definitely not a big vanilla aficionado. (reverse coded)
However, when they examined teens' responses, they noticed that overall, there were consistent response patterns for each individual item across all participants, none of which were expected to occur. What issues (i.e., threats to measurement validity) would they be most vulnerable to with their current scale? (circle all that apply)

A) acquiescence
B) social desirability
C) mood
D) language

Question 11

Once a scale has been determined to be valid, it can be assumed to be valid in future administrations and across various settings.

Question 12

A scale that is found to have high validity will inevitably also have high replicability.

Question 13

Face validity differs from other forms of validity in that:

A) it does not deal with the content of specific scale items
B) it is the most difficult form of validity to achieve
C) it cannot be established in a rigorous, systematic manner
D) items low in face validity can still be of use to the researcher(s)
E) it addresses whether a scale or set of items correlates with another closely related measure

Question 14

If a hypothetical scale meant to assess knowledge of zoology contained questions about animals, plants, and inanimate objects, the scale would be said to have low _______ validity.

A) convergent
B) predictive
C) content
D) concurrent
E) None of the above

Question 15

Generating a large pool of items, consulting a panel of experts, and conducting a review of the literature, are all ways to improve:

A) content validity
B) predictive validity
C) convergent validity
D) discriminant validity
E) criterion validity

Question 16

Concurrent and predictive validity:

A) are synonymous
B) are two subtypes of criterion validity
C) are both desirable, however concurrent validity is deemed the more rigorous of the two
D) are both desirable, however predictive validity is deemed the more rigorous of the two
E) are both necessary to establish discriminant validity

Question 17

Briefly define the following components of Campbell and Fiske's (1959) Multi-trait Multi- Method Matrix:

A) Heterotrait-heteromethod values
B) Monotrait-heteromethod values
C) Reliability diagonals
D) Heterotrait-monomethod values

Question 18

In the MTMMM, monotrait-heteromethod triangles:

A) assess convergent validity
B) Iidicate the reliability of multiple measures
C) represent the relationship between different methods of measurement of the same trait or construct
D) show the correlations between multiple traits measured using the same method
E) show the correlations between multiple traits and multiple methods of measurement

Question 19

Which of the following is NOT a requirement set forth by Campbell and Fiske (1959) for establishing construct validity using the MTMMM?

A) Establishing convergent validity across measurement methods
B) Statistically significant monotrait-heteromethod values
C) Higher values in monotrait-heteromethod triangles than in heterotrait-monomethod triangles
D) Patterns of trait correlations are the same in monomethod and heteromethod blocks
E) Heterotrait-heteromethod values exceed heterotrait-monomethod values

Question 20

Imagine that Susan is a high school senior completing a survey for your research study. Presume that earlier that morning, Susan received a rejection letter from one of her top colleges. In addition, your survey involves the topic of racism in everyday life. Susan is also an international student, and English is not her first language. You look at Susan's completed survey and notice that all of her item responses lie close to the midpoint of your 9-point scale. Given this information, which of the following threats to validity may be at play?

A) Acquiescence
B) Extreme response sets - (other students may also respond in similar fashion)
C) Mood - (individual differences in mood don't lead to systematic bias, not the same as weather example)
D) Social desirability - (survey is on a sensitive topic that other respondents may also respond favorably on)
E) Language difficulty - (individual language difficulty is different from difficult language used on the survey, not experienced by all respondents)

Question 21

(a) Describe what is meant by heterotrait-monomethod, monotrait-heteromethod, and heterotrait-heteromethod in the multitrait-multimethod matrix approach. (b) Explain what these concepts supposed to assess.

Question 22

Mercury and Andromeda wanted to develop a valid measure of people's attitudes toward ancient mythology, and wrote the following items:
i. Ancient mythology is important because it provides insight into modern problems.
ii. I enjoy reading about ancient civilizations and the mythologies surrounding them.
iii. I believe in ghosts.
Provide two specific suggestions for how they could test the validity of their three-item scale. In your response, be sure to explain the type of validity you are testing, the possible outcomes of those tests, and how you would determine if the validity of the scale is satisfactory.

Question 23

People's responses on a scale can be influenced by systematic biases that are completely irrelevant to the construct that the scale was intended to measure. Specifically, imaging a researcher is asking for help in developing a scale assessing people's enjoyment of noxious or foul odors. Describe two threats to measurement validity that might influence responses to this scale that have nothing to do with people's penchant for stinky smells. In your response, be sure to (a) provide an example item for the scale that you will use as a reference, (b) identify and describe each validity threat, and (c) discuss how each validity threat would manifest in relation to your example item (i.e., explain how each threat would specifically bias responses, and in what direction).

Question 24

Discuss the meaning of scale validation as a "continuous process."

Question 25

Of all the various kinds of validity (predictive, convergent, face, content, discriminant, etc.) are there some forms of validity that are more important than others? Are they all equally important?

Question 26

What are the five threats to measurement validity presented in this chapter? Are there other possible threats to validity that you can think of?

Question 27

During a lecture about designing your first research study, your professor (who is also your research advisor) told the class that in general, using a previously validated scale (when there is one available) is preferable to creating all of your own items. However, in a different lecture, your advisor mentioned that just because a scale is validated does not mean it will always be valid. Based on what you know about validity, which statement is correct? In what situations is the first statement better advice? In what situations is the second statement better advice?

Question 28

Your advisor asked you to test the construct validity of a new 50-item scale that the two of you developed, called the SMRT intelligence test (meant to measure people's "intelligence"). At first glance, it seems to at least possess some face validity, in that the items appear to ask about different things that you think "smart people would be likely to score high on," particularly emotional intelligence, mathematical ability, and vocabulary. In a previous pilot study using this 50-item SMRT test, high scores on the SMRT correlated with heightened board game-playing skills. Given this information, what is your assessment of the different aspects of construct validity for the SMRT intelligence test? What additional validity tests could you do to strengthen the argument that the SMRT is a valid measure of "intelligence" - how would you know (i.e., what results would you need to see from those tests)?

Question 29

You conducted a laboratory experiment investigating the effects of a perspective taking intervention on participants' attitudes toward outgroup members (e.g., if participants were Democrats, outgroup members would be Republicans). You created a new measure of prejudice because you could not find a previous validated prejudice scale. Your results indicated that at Time 1, prejudice scores were similar for the treatment and control groups, while at Time 2, prejudice scores were slightly, but significantly, lower in the control group, rather than in the treatment group. How could that be - could there be something wrong with the way you measured prejudice? Specifically, how might our knowledge of measurement validity explain these results? Discuss the different aspects of measurement validity and how they could have impacted your results.