Exam 4: Measurement Validity

The multitrait-multimethod matrix approach is one technique for determining the validity of a target construct. Specifically, the MTMMM addresses what types of validity in its model?

In the MTMMM, monotrait-heteromethod triangles:

Face validity differs from other forms of validity in that:

Imagine that Susan is a high school senior completing a survey for your research study. Presume that earlier that morning, Susan received a rejection letter from one of her top colleges. In addition, your survey involves the topic of racism in everyday life. Susan is also an international student, and English is not her first language. You look at Susan's completed survey and notice that all of her item responses lie close to the midpoint of your 9-point scale. Given this information, which of the following threats to validity may be at play?

Briefly define the following components of Campbell and Fiske's (1959) Multi-trait Multi- Method Matrix:

__________ is concerned with the extent to which items in a scale adequately reflect the breadth of the construct that is supposedly being measured.

Mercury and Andromeda wanted to develop a valid measure of people's attitudes toward ancient mythology, and wrote the following items: i. Ancient mythology is important because it provides insight into modern problems. ii. I enjoy reading about ancient civilizations and the mythologies surrounding them. iii. I believe in ghosts. Provide two specific suggestions for how they could test the validity of their three-item scale. In your response, be sure to explain the type of validity you are testing, the possible outcomes of those tests, and how you would determine if the validity of the scale is satisfactory.

Generating a large pool of items, consulting a panel of experts, and conducting a review of the literature, are all ways to improve:

Once a scale has been determined to be valid, it can be assumed to be valid in future administrations and across various settings.

Your advisor asked you to test the construct validity of a new 50-item scale that the two of you developed, called the SMRT intelligence test (meant to measure people's "intelligence"). At first glance, it seems to at least possess some face validity, in that the items appear to ask about different things that you think "smart people would be likely to score high on," particularly emotional intelligence, mathematical ability, and vocabulary. In a previous pilot study using this 50-item SMRT test, high scores on the SMRT correlated with heightened board game-playing skills. Given this information, what is your assessment of the different aspects of construct validity for the SMRT intelligence test? What additional validity tests could you do to strengthen the argument that the SMRT is a valid measure of "intelligence" - how would you know (i.e., what results would you need to see from those tests)?

When using self-report measures, researchers risk having participants' responses be systematically influenced by factors that have nothing to do with the constructs of interest (or the constructs that the items are intended to measure). These "threats to measurement validity" include which of the following:

Which of the following is NOT a requirement set forth by Campbell and Fiske (1959) for establishing construct validity using the MTMMM?

You conducted a laboratory experiment investigating the effects of a perspective taking intervention on participants' attitudes toward outgroup members (e.g., if participants were Democrats, outgroup members would be Republicans). You created a new measure of prejudice because you could not find a previous validated prejudice scale. Your results indicated that at Time 1, prejudice scores were similar for the treatment and control groups, while at Time 2, prejudice scores were slightly, but significantly, lower in the control group, rather than in the treatment group. How could that be - could there be something wrong with the way you measured prejudice? Specifically, how might our knowledge of measurement validity explain these results? Discuss the different aspects of measurement validity and how they could have impacted your results.

Delroy Paulhus and colleagues (e.g., Paulhus & Reid, 1991) have conducted extensive research on the concept of self-deception (i.e., lying to oneself), including a scale called the Self-Deceptive Denial scale (as part of the larger BIDR inventory). These items assess the extent to which people are willing to lie to themselves by answering negatively on the scale, even though that the items on the scale are thought to reflect internal states that almost everyone has experienced at least once (e.g., I can't think of anyone I hate deeply; seeing an attractive person of the opposite sex makes me think about sex; I have never done anything I am ashamed of). By actually studying self-denial, Paulhus and colleagues are exploiting what threat to measurement validity?

"Measurement construct validity" includes which of the following types of validity? (circle all that apply)

Two researchers wanted to determine if their new scale of people's aversion to technology was a valid measure of the construct. On the surface, the items seem to be related to distaste for technological advances. Additionally, in an initial cross-sectional study of the scale, they found that people who scored high on aversion to technology were unlikely to own smartphones, tablet computers, or an electronic navigation device (technology aversion and ownership of devices were measured at the same time). Moreover, they found that people's technology aversion score was unrelated to scores on preference for American versus imported cars, or attitudes about the current president. In this example, which of the following types of validity have been addressed? (circle all that apply)

What are the five threats to measurement validity presented in this chapter? Are there other possible threats to validity that you can think of?

When developing a scale that assesses teenagers' preference for sweets, researchers wrote these three items, on a 1-10 scale (1 = extremely disagree, 10 = extremely agree): i. Chocolate cake is my favorite dessert. ii. I would rather revel in confectionery than savor mincemeat. iii. I am definitely not a big vanilla aficionado. (reverse coded) However, when they examined teens' responses, they noticed that overall, there were consistent response patterns for each individual item across all participants, none of which were expected to occur. What issues (i.e., threats to measurement validity) would they be most vulnerable to with their current scale? (circle all that apply)

It is not only important to examine the validity of the content of a scale, it is also important to examine how that scale is related to a target outcome or criterion that it should theoretically be related to. Specifically, this type of validity encompasses which of the following:

If a hypothetical scale meant to assess knowledge of zoology contained questions about animals, plants, and inanimate objects, the scale would be said to have low _______ validity.