Deck 11: Validity of Research Results in Quantitative, Qualitative, and Mixed Research

Question 1

Identify and explain the types of evidence that are needed to reach a causal conclusion.

Answer 1

not answered

Question 2

When scores improve simply due to having taken a pretest, this threat to validity is called:

A) History effect
B) Multiple-treatment interference
C) Testing effect
D) Maturation effect

Answer 2

C

Question 3

Explain the threats to external validity and when they might exist in a research study.

Answer 3

not answered

Question 4

Identify and explain the types of validity or legitimation used in mixed research.

Question 5

Which of the following statements is true?

A) A statistical relationship is sufficient evidence to infer causality.
B) Temporal order of the cause and effect is not important in inferring causality.
C) A statistical relation of X and Y is insufficient evidence for inferring causality.
D) Temporal order of the cause and effect variables and a statistical relationship are all that is needed to infer causality.

Question 6

Explain the role of operationalization of constructs in research.

Question 7

Explain the meaning of statistical conclusion validity, construct validity, internal validity, and external validity and their importance in the research process.

Question 8

Explain the meaning of confounding variables.

Question 9

When an extraneous variable systematically varies with the independent variable and influences the dependent variable, it is called:

A) Another dependent variable
B) A confounding variable
C) A moderating variable
D) An unreliable variable

Question 10

A researcher does a one-group pretest-posttest study where she examines the effects of a reading program on children's reading comprehension level. She finds that the students' scores at the end of the school year are higher than at the beginning of the school year. She attributes the changes to the reading program. Her colleague argues you would expect that the children would improve anyway from the beginning of the year to the end of the year. Her colleague is arguing that the results are potentially due to:

A) History effect
B) Multiple-treatment interference
C) Testing effect
D) Maturation effect

Question 11

A state passes a law requiring mandatory seatbelt use and claims it has reduced traffic fatalities. At the same time that the seatbelt law was changed, speed limits were also reduced from 70 MPH to 55 MPH by the federal government. Traffic fatalities decreased during the first year of the seatbelt law. The change in the speed limits represents what kind of threat to internal validity?

A) Maturation effect
B) Instrumentation effect
C) Testing effect
D) History effect

Question 12

A researcher does a study of a mathematics program in a one-group pretest-posttest design. She gives the group of second graders a multiple-choice pretest. Unknown to her, the children had never taken a multiple-choice test. The children scored higher on the same test given at posttest. She argues that the children improved because of the mathematics program. That the children had never taken a multiple-choice test before and showed improvement at posttest suggests that one possible threat to her conclusion is a:

A) History effect
B) Multiple-treatment interference
C) Testing effect
D) Selection effect

Question 13

Identify and explain the types of validity or trustworthiness used in qualitative research.

Question 14

Explain the threats to internal validity and be able to identify when they might exist in a research study.

Question 15

When an unplanned event occurs during the course of the study that might influence the dependent measure, this is a called:

A) History effect
B) Maturation effect
C) Testing effect
D) Instrumentation effect

Question 16

A researcher conducts an experiment on memory strategies where the research assistant unknowingly runs control participants only in the afternoon and the experimental group only in the morning. Time of day in this case very well might be a(n):

A) Independent variable
B) Dependent variable
C) Mediating variable
D) Confounding variable

Question 17

To infer causality, three conditions must be met. One of them is that:

A) X and Y occur at the same time.
B) X is not related to Y.
C) Confounding variables can be ruled out as causes.
D) Manipulating one variable does not influence another variable.

Question 18

An extraneous variable is:

A) Any variable other than the independent variable that might influence the dependent variable.
B) A variable that was left out of the study because it was irrelevant.
C) A dependent variable that is redundant with the other dependent variables.
D) Any variable other than the dependent variable that might influence the independent variable.

Question 19

When physical or mental changes occur in individuals, that influences their scores on the dependent variable(s). This is called:

A) History effect
B) Maturation effect
C) Testing effect
D) Instrumentation effect

Question 20

When the researcher can draw clear causal conclusions from what went on in a study, this is called:

A) Internal validity
B) Internal reliability
C) Internal consistency reliability
D) External validity

Question 21

The ability to generalize the results of a study to a population is greatest when:

A) Random assignment is used
B) Random selection is used
C) Convenience sampling is used
D) Purposive sampling is used

Question 22

When there is differential maturation in two groups, this is also called:

A) Mortality effect
B) Selection-history effect
C) Selection-regression artifacts effect
D) Selection-maturation effect

Question 23

It is sometimes forgotten that the ability to generalize to the target population requires that:

A) The characteristics of the accessible population match the target population
B) The accessible population be 100 or greater
C) Random assignment took place
D) The study sample be a convenience sample

Question 24

When the participants in the different treatment conditions have different characteristics that vary along with the independent variable, this is called:

A) Maturation
B) Regression artifacts
C) History
D) Differential selection

Question 25

Reactivity can influence:

A) The ecological validity of a study
B) The internal validity of a study if it varies across comparison groups
C) Both ecological and internal validity of a study
D) Neither ecological nor internal validity

Question 26

When the researcher is unable to provide clear evidence that the independent variable preceded the effect, we have the threat to internal validity known as:

A) Ambiguous temporal precedence
B) Proper time order
C) Improper causal precedence
D) Linear temporal precedence

Question 27

If the way a variable is measured changes during the course of a study, this threat to validity is called:

A) Testing effect
B) Instrumentation effect
C) Maturation effect
D) Multiple-treatment interference

Question 28

A researcher is carrying out a study of the effects of teaching different approaches to argumentation on writing persuasive essays. He pretests the participants by having them write three persuasive essays prior to the study, and three essays after the study. He has two individuals grade all the essays. The two individuals were given no training prior to the study. The essays from the students in the study actually looked worse rather than better after the study. Based on the information presented above, what threat to internal validity could explain the contradictory finding?

A) Selection
B) Maturation effect
C) Instrumentation
D) History effect

Question 29

A personnel committee is evaluating professors. Professor X has the highest student ratings in the department. One of the committee members suggests that Professor X should receive the highest merit raise in the department. Another committee member raises an objection. She states, "Professor X scores are high, compared to other professors, because over half of the students drop out of Professor X's courses, not because Professor X is an excellent professor." What threat to the validity of the conclusion that Professor X is an excellent professor is the committee member describing?

A) Maturation effect
B) History effect
C) Testing effect
D) Differential attrition effect

Question 30

The ability to generalize results across settings is called:

A) Multiple treatment interference
B) Population validity
C) Ecological validity
D) Internal validity

Question 31

The ability to generalize the results of a study to individuals not in the study is called:

A) Internal validity
B) Temporal validity
C) Ecological validity
D) Population validity

Question 32

The ability to generalize the results across different but related dependent variables is called:

A) Temporal validity
B) Outcome validity
C) Treatment variation validity
D) Ecological validity

Question 33

A group of researchers conduct a study where children from particular classrooms are assigned to treatment or control conditions. After the study, the researcher finds out that the students in the control group are higher achievers than those in the experimental group. He found no treatment effect. The failure to find a treatment effect might be due to:

A) A treatment effect
B) A testing effect
C) A differential selection effect
D) A maturation effect

Question 34

When the target population has several subpopulations within, it is possible that:

A) The overall results might not hold for subpopulations
B) The overall results will always hold for all subpopulations
C) Generalizing across the subpopulations is easily accomplished
D) The overall results will apply to all subpopulations

Question 35

When the different threats to internal validity act differently in different comparison groups, this is known as:

A) Additive and/or interactive effects
B) A testing effect
C) A maturation effect
D) A regression artifacts effect

Question 36

When there is differential drop out between groups in a study, this problem is called:

A) Maturation effect
B) Differential attrition
C) History effect
D) Testing effect

Question 37

A researcher examines a program looking at the effects of mentoring on poor readers' reading achievement. He looks at two different schools. One serves as the control and the other the experimental group. Both schools had reading achievement that was around the 50th percentile. During the time that the mentoring program is in place in the experimental group, a statewide reading initiative is started in randomly selected schools. The experimental, but not the control school is involved in the statewide reading initiative. At the end of the year, the experimental group does better than the control. From the information presented above, a likely threat to the internal validity of the study is:

A) Selection by mortality interaction
B) Mortality
C) Selection-history effect
D) Selection-maturation effect

Question 38

The tendency of scores that are well below the mean to move up toward the mean on retesting, and the tendency of scores that are well above the mean to move down toward the mean on retesting is called:

A) Selection effect
B) Maturation effect
C) Instrumentation effect
D) Regression artifact effect

Question 39

A school district examines a program that uses mentors to help very poor readers improve their reading performance. The children in the program are at the 4th percentile at pretest. At posttest they are around the 20th percentile. While it is possible that the program made the difference, another reason for the change in scores could be:

A) History
B) Regression artifact
C) Multiple-treatment interference
D) Differential selection

Question 40

The ability to generalize the results across variations of the treatment is called:

A) Temporal validity
B) Outcome validity
C) Treatment variation validity
D) Ecological validity

Question 41

Reflexivity in qualitative research refers to:

A) Using multiple sources of data
B) Reflection after data collection about hypotheses
C) Self-reflection by the researcher on his or her biases and predispositions
D) Only self-reflection by the researcher about what literature to include in the literature review

Question 42

Multiple theoretical perspectives refer to:

A) Using multiple investigators to determine the accuracy of a researcher's account of an event
B) Using multiple data sources to confirm a theory
C) Using multiple theories or perspectives to interpret and explain the data
D) Using extended fieldwork to test whether one theory fits the data

Question 43

Anna is writing up her qualitative research study. She took in depth notes on what took place with the participants as well as what she saw and heard when working with the participants. She did this to improve the study's:

A) Interpretive validity
B) Participant validity
C) Theoretical validity
D) Descriptive validity

Question 44

One way that a qualitative researcher can help reduce researcher bias is to use:

A) Negative case sampling
B) Random assignment to groups
C) Statistical regression
D) Snowball sampling

Question 45

Roger is concerned with generalizing the results of his study to and across different groups of people, settings, times, and treatments. His concerns revolve around:

A) Internal validity
B) External validity
C) Causation
D) Temporal precedence

Question 46

To most accurately measure a construct, it is recommended that:

A) Researchers use only one measure
B) Researchers use effective items to measure the construct
C) Researchers not study constructs over time
D) Researchers avoid multiple operationalism

Question 47

A major problem for research validity in qualitative research is:

A) The researcher does not include quantitative measures
B) There is no way to test the internal and external validity of qualitative studies
C) There is no way to validate descriptions of settings
D) Researcher bias can go unrecognized and can invalidate conclusions

Question 48

Theoretical validity refers to:

A) Accuracy and defensibility of the theoretical explanations given
B) Accuracy of the descriptions of the data
C) Accuracy of the researcher's description of the meanings ascribed to participants
D) Accuracy of the low inference descriptors used

Question 49

In qualitative research, descriptive validity refers to:

A) The mean, standard deviation, and the range
B) The validity of theory that the researcher uses
C) The validity of tests used in the study
D) Factual accuracy of accounts reported by researcher

Question 50

One common way to ensure descriptive validity in qualitative research is to:

A) Use multiple theoretical perspectives
B) Compute means and standard deviations
C) Use multiple investigators
D) Using pattern matching

Question 51

One way to promote theoretical validity in qualitative research is to:

A) Use low inference descriptors
B) Avoid participant observation
C) Use extended fieldwork
D) Use data from interviews only

Question 52

Interpretive validity relates to:

A) Accuracy of researcher in portraying the research participants' thoughts and meanings
B) Factual accuracy of the account reported by a researcher
C) Validity of the theory applied to the data
D) Degree to which the results can be generalized to other populations

Question 53

Richard has made predictions about the data he will collect for his research study. After his data are collected he compares them to his predictions. He does this to assess which type of validity?

A) External validity
B) Internal validity
C) Population validity
D) Theoretical validity

Question 54

Sarah is conducting an ethnographic study of fifth grade classes using an interdisciplinary curriculum. After she collects her data, she goes back and asks each teacher to help her out by checking the accuracy of the descriptions she has written of the classrooms. This technique is called:

A) Participant observation
B) Participant feedback
C) Participant validity
D) Theoretical validity

Question 55

In his study, Patrick obtained feedback from his participants and used many of their descriptive words in writing up the study for publication. These strategies provide evidence of:

A) Interpretive validity
B) Descriptive validity
C) Reflexive validity
D) Theoretical validity

Question 56

An alteration of performance that occurs when an individual is aware that he or she is participating in a study is called:

A) Multiple treatment interference
B) Reactivity
C) Camera effect
D) Experimenter effects

Question 57

A researcher places a video camera in a classroom to tape lessons that are part of one of the treatment conditions. The video camera is likely to increase the probability of:

A) Multiple treatment interference
B) Selection effects
C) Reactivity
D) Maturation

Question 58

Pattern matching refers to:

A) Looking at the match between the actual data and a theoretically predicted set of results
B) Looking at the match between the descriptions of researchers and the descriptions of participants in a qualitative study
C) The researcher matching what participants do by imitating it
D) The researcher finding the match for his or her results in another study

Question 59

The extent to which a study can be generalized across time is called:

A) Incremental validity
B) Ecological validity
C) Population validity
D) Temporal validity

Question 60

To increase the theoretical validity of his study, a qualitative researcher should use:

A) Low inference descriptors
B) Peer review
C) Data from observations only
D) Data from interviews only

Question 61

Which of the following does not represent a threat to internal validity?

A) Random assignment
B) History
C) Differential attrition
D) Regression artifact

Question 62

External validity in qualitative research:

A) Is always irrelevant in qualitative research
B) Is confirmed through random sampling in qualitative research
C) Is confirmed through random assignment in qualitative research
D) Can be obtained by replication in qualitative research, but it is often of little interest to the qualitative researcher

Question 63

Fred took the SAT to meet an entrance requirement for a college. On his first attempt he scored far below average. He decided to study for the test and take it again. This time he scored higher and closer to the average of all people taking the test. This example illustrates the possibility of _______________ as a rival plausible explanation.

A) History
B) Maturation
C) Regression artifact
D) Differential selection

Question 64

Dr. Sanjay is studying whether his racism awareness program increases his research participants' awareness of issues of race and ethnicity in his city. On Monday, he gives an attitude survey to his participants to gauge their awareness. On Tuesday through Thursday, he puts them through his program. On Wednesday, a local predominantly black church is firebombed and the KKK claims responsibility. On Friday, Dr. Sanjay gives his awareness survey again and finds awareness of racial issues has increased dramatically. This example illustrates the possibility of ______________ as a plausible rival explanation.

A) History
B) Maturation
C) Regression artifact
D) Differential selection

Question 65

When we talk about inferring that the independent and dependent variables are related in the population of interest and the strength of the relationship is accurate, we are talking about:

A) Statistical conclusion validity
B) Internal validity
C) Construct validity
D) External validity

Question 66

The confounding effects referred to collectively as "maturation" include all of the following plausible rival hypotheses except:

A) Aging
B) Instruction
C) Fatigue
D) Boredom

Question 67

A school counselor was concerned with the rising level of violence at the school in which she was employed so she decided to try an experimental program to reduce the level of violence. She identified two groups of violent children at her school and randomly assigned them to two groups. Both groups of children are taught effective methods of dealing with other children and with frustration, but one group is reinforced every time they use any of these techniques and isolated when they are violent. The other group is isolated and punished in the form of having to do additional duties such as cleaning the bathrooms whenever they act violently. Some of the participants in both groups fail to complete the study but most of the ones that drop out are in the punishment group. The study found that both groups reduced their violent behavior by equal amounts. Which is the best conclusion to draw from this study?

A) There is no difference in the effectiveness of reinforcement versus punishment
B) The study is confounded by differential attrition and this might explain the failure to find any group differences
C) The study is confounded by a history bias and this might explain the failure to find any group differences
D) Reinforcement would have been demonstrated to be more effective if the study had lasted longer

Question 68

The extent to which a researcher understands, uses, and presents participants' but maintains her objective outsider view is known as:

A) Paradigmatic legitimation.
B) Inside-outside legitimation.
C) Philosophical legitimation.
D) Sequential legitimation.

Question 69

Mr. Jones has demonstrated that his experimental reading program is very effective with 4th and 5th grade children with a learning disability. Ms. Thornton is using his experimental reading program but does not use it exactly like the instructions specify. Although Ms. Thornton has varied the way in which she is implementing the reading program she also finds it to be very effective with her students with learning disabilities. This is a demonstration that the reading program has:

A) Population validity
B) Temporal validity
C) Ecological validity
D) Treatment variation validity

Question 70

Internal validity in qualitative research refers to:

A) The validity of cause and effect conclusions
B) The validity of the research results for other settings
C) The validity of the research results for other populations
D) The validity of the theories that guided the research

Question 71

If a research study revealed that the independent and dependent variables covary and they really do covary, the study has:

A) Been demonstrated to be worthwhile
B) Revealed a causal relationship
C) Statistical conclusion validity
D) Revealed a spurious relationship

Question 72

Dr. Kenji wants to determine whether his new phonics program helps children's reading comprehension. He gives a fifth grade class the program and also studies a second grade class that does not go through the program. He finds that the fifth graders read with better comprehension. The confounding variable in this research design is:

A) The grade the students are in
B) The phonics program
C) The reading comprehension scores
D) Not giving one group the program

Question 73

If the results of your study can be generalized to other people, in other settings, using different treatment variations and outcome measures, and the results have held up over time you have the ultimate in:

A) Statistical conclusion validity
B) Internal validity
C) External validity
D) Construct validity

Question 74

Dr. Faust investigated the overall effectiveness of an experimental program to teach reading to children from cultures other than the U.S.. He found that the experimental program was much more effective than the standard reading program typically administered to these children. In this study, Dr. Faust investigated:

A) Causal explanation
B) Ambiguous temporal precedence
C) Statistical conclusion validity
D) Causal description

Question 75

William has conducted a qualitative study. He has internally validated his findings by ruling out rival explanations for the results. This is referred to as the:

A) Researcher-as-laborer metaphor
B) Researcher-as-teacher metaphor
C) Researcher-as-legislator metaphor
D) Researcher-as-detective metaphor

Question 76

Qualitative researchers can determine whether their results would hold in other settings and with other populations by using:

A) Replication logic
B) Low inference descriptors
C) Data multiple sources of data
D) Multiple theoretical perspectives

Question 77

A qualitative researcher is often concerned with identifying the immediate, intentional, particular, complex, and local causes of attitudes, actions, and events. This is known as:

A) Nomothetic causation
B) Idiographic causation
C) Replication causation
D) Naturalistic causation

Question 78

Dr. Martin's research team was able to generalize the results of their study because they saw the same results in several other studies. These researchers were using:

A) Theoretical transfer
B) Replication logic
C) Low inference descriptors
D) Participant feedback

Question 79

Mary wanted to demonstrate that her qualitative research results held in other settings and with other populations. In order to do this she should:

A) Use low inference descriptors
B) Use multiple sources of data
C) Use multiple theoretical perspectives
D) Use naturalistic generalization

Question 80

Dr. Johnson conducted a research study in which he investigated the effect of three different methods of teaching mathematics to three different classes of fifth grade students. However, one of the classes had smarter students than the other two and as a result this class demonstrated the most improvement in their mathematics scores. In spite of this problem, he concluded that the method used to teach mathematics to this class of students was the most effective. In this experiment, the intellectual ability of the students in the various classes was:

A) A confounding extraneous variable
B) An extraneous variable that did not confound the results
C) An independent variable
D) A dependent variable