Deck 6: Causation and Experimental Design

Question 1

The Hawthorne Effect suggests that if the control group believes it is being left out of some special treatment, it will perform worse than expected on posttests.

Answer 1

False

Question 2

In an experiment on obedience to authority, one subject tips another off that the shocks that he thinks he is administering to another subject are not real. This is an example of:

A) John Henry Effect
B) Contamination
C) Placebo effect
D) Hawthorne Effect
E) History effect

Answer 2

B

Question 3

Treatment misidentification can be limited by a double-blind procedure.

Answer 3

True

Question 4

In experiments, subjects are ideally assigned to control and experimental groups using simple random sampling.

Question 5

With experiments, researchers are concerned with whether the relationship between the treatment and the outcome variable holds true for certain subgroups.

Question 6

Use the following to answer questions 16-18:
In each set, match a concept from Group A to a definition or example from Group B.
Experimental Designs
Group A
1. True Experiment
2. Non-Equivalent Control Group
3. Before-and-After
4. Ex Post Facto Control Group
5. Solomon Four Group

Group B
a. Randomized assignment into control and experimental groups
b. Lacks a control group
c. Contains two sets of control groups and two sets of experimental groups
d. Experimental and control groups are designated after experimental treatment
e. Lacks a posttest
f. Control group is created not by randomization, but to be as similar as possible to the experimental group

Question 7

Before-and-after experimental designs lack control groups.

Question 8

According to the requirement of time order in establishing a causal effect, the dependent variable must come before the independent variable.

Question 9

Association, time order, and nonspuriousness are required to establish a causal effect.

Question 10

Compensatory rivalry exists between comparison groups in an experiment when:

A) The control group expends extra effort to equal the experimental group
B) There is a greater level of attrition in the control group
C) Members of the treatment group compete with one another
D) Members of the treatment group expend greater effort
E) The treatment group is pre-tested, but the control group is not

Question 11

Use the following to answer questions 16-18:
In each set, match a concept from Group A to a definition or example from Group B.
Criterion of Causality
Group A
1. Association
2. Context
3. Mechanism
4. Nonspuriousness
5. Time Order

Group B
a. Identification of other variables that allow for a relationship between independent and dependent variables
b. The process that creates the connection between the independent and dependent variables
c. The degree to which the relationship can be modeled in other experiments
d. The relationship between independent and dependent variables is not due to a third variable
e. Observed correlation between the dependent and independent variables
f. The temporal priority of the independent variable

Question 12

A nonspurious relationship between variables is a false relationship.

Question 13

The ex post facto control group design is a quasi-experimental design.

Question 14

It is never ethical to deceive subjects about an experiment.

Question 15

In an experiment, the control group receives the experimental treatment.

Question 16

Experiments have relatively low external validity.

Question 17

Subjects who can be recruited for a laboratory experiment, randomly assigned to a group, and kept under carefully controlled conditions for the duration of the study are the most likely to be representative of the larger population.

Question 18

Use the following to answer questions 16-18:
In each set, match a concept from Group A to a definition or example from Group B.
Sources of Internal Invalidity in Experiments
Match the measurement procedure in Group A to the definition in Group B.
Group A
1. Pretesting
2. Selection bias
3. The History Effect
4. The Hawthorne Effect
5. Demoralization
6. The John Henry Effect
7. Regression
8. Mortality

Group B
a. Things can happen outside the experiment that change subjects' scores
b. Competition between control and experimental groups can change scores
c. Subjects' scores can change because they feel special for being chosen for an experiment
d. Subjects will drop out of the experimental or control group at rates that change the nature of the comparison between groups
e. Measurement before experimental treatment may change subjects' scores on posttest
f. Inferences about group processes are drawn from individual level data
g. Subjects in the control group may feel deprived of treatment and perform worse on posttests than expected
h. Extreme scores used to select subjects will move back toward the average in subsequent testing
i. Subjects in the control and experimental group are initially different

Question 19

It is never ethical to randomly assign treatment to subjects in experiment just because the effect of the treatment is unknown.

Question 20

Aggregate matching is when individual cases in the treatment group are matched with similar individuals in the comparison group.

Question 21

Which of the following is not an example of endogenous change, in which internal validity is compromised because of natural developments in the subjects, independent of experimental treatment?

A) Testing
B) The History Effect
C) Maturation
D) Regression
E) All of the above are types of endogenous change

Question 22

To establish time order, which of the following must come first?

A) Covariation
B) Association
C) The dependent variable
D) The pretest
E) The independent variable

Question 23

Rachel finds that higher salaries produce greater productivity in work environments with a great deal of creative flexibility, but have no effect in routine, unskilled work. What element of causality has she identified?

A) Association
B) Time Order
C) Nonspuriousness
D) Context
E) Mechanism

Question 24

Which of the following are requirements for identifying a causal effect?

A) Association, time order, and nonspuriousness
B) Time order, nonspuriousness, and mechanism
C) Nonspuriousness, mechanism, and context
D) Association, mechanism, and context
E) Context, time order, and association

Question 25

A researcher is concerned that gender may affect how subjects respond to an experimental stimulus. So subjects are paired with another subject who has the same gender, and one member from each pair is assigned by a flip of a coin to the experimental group and one to the control group. This method of assigning subjects to groups is called:

A) Matching
B) Quotas
C) Stratified randomization
D) Random assignment
E) Nonprobability sampling

Question 26

The reason that ex post facto control group designs are not considered quasi-experimental is:

A) They lack control groups
B) They lack random assignment for control groups
C) Control groups are specified after treatment has occurred
D) They lack pretests
E) Subjects are allowed to self-select group membership

Question 27

A university conducts a series of surveys in which subjects are drawn from first year and graduating students and finds that graduating students demonstrate much higher rates of approval for the university (in terms of whether they think the classes were relevant, the instructors were competent, etc.). The university then claims that more experience with the university leads to higher approval. What source of internal invalidity has the university not considered?

A) Selection bias
B) Mortality
C) Noncomparable groups
D) Regression
E) Contamination

Question 28

Two types of before-and-after experimental designs in which multiple pretest and posttest observations are made of the same group are:

A) Solomon four group designs and panel designs
B) Panel designs and repeated measures panel designs
C) Repeated measure panel designs and time series designs
D) Time series designs and nonequivalent control group designs
E) Nonequivalent control group designs and Solomon four group designs

Question 29

The process through which the independent variable creates changes in a dependent variable is known as a(n):

A) Association
B) Context
C) Consideration
D) Covariation
E) Mechanism

Question 30

To say that variables have an association is to say that:

A) The relationship between them is genuine
B) They occur at the same point in time
C) They empirically vary together
D) The relationship between them is false
E) One is caused by the other

Question 31

Quasi-experimental designs are like true experiments, except:

A) They never have a control group.
B) They use multiple control groups.
C) They do not randomly assign subjects to groups.
D) They take measurements at more than one point in time.
E) They do not use pretests.

Question 32

In an experiment, measurement of the dependent variable after the experimental treatment is known as:

A) Post-treatment assessment
B) Pretest
C) Posttest
D) Control-test
E) Matching

Question 33

A true experiment allows the establishment of nonspuriousness through:

A) Using two comparison groups
B) Controlling when treatment occurs
C) Random assignment into two or more comparison groups
D) Documenting the conditions under which change occurs
E) All of the above

Question 34

After 30 years of teaching the same research methods course, Professor Zone's average exam scores increase because he doesn't want to spend the time arguing with students over borderline grades. This is an example of:

A) History effect
B) Instrumental decay
C) Mortality
D) Differential attrition
E) Contamination

Question 35

When a staff member explains to a subject why he or she has been deceived in an experiment, what is the staff doing?

A) Contamination
B) Debriefing
C) Self-fulfilling prophecy
D) Treatment misidentification
E) Process analysis

Question 36

When parents are registering their children for kindergarten, they are given the opportunity to enroll their children in an experimental music program that researchers hope will improve overall learning. What experimental design is being used?
Experiment
B) Non-equivalent control group
C) Before and after
D) Time Series
E) Solomon Four Group

Question 37

After they had failed the writing exam that was required of all students before graduation, fifteen students were placed in a class in which they received additional tutoring. After completing this class, all fifteen students retook the exam earned higher scores. Their success was held up as an example that the class curriculum really works. What source of internal invalidity has not been considered?

A) History Effect
B) Noncomparable Groups
C) Contamination
D) Regression
E) Placebo Effect

Question 38

A spurious relationship between two variables is:

A) False
B) Causal
C) Valid
D) Reliable
E) Compensatory

Question 39

The common feature of all before-and-after designs is:

A) Deliberate assignment is used to determine control and experimental groups
B) All cases receive the experimental treatment
C) Pretests are given to only half of the subjects
D) Group assignment occurs through matching
E) Multiple groups are used as experimental and control groups

Question 40

A procedure that assesses that treatment and measurement are consistent throughout an experiment is a(n):

A) Evaluation
B) Double-blind procedure
C) Process analysis
D) Evaluability assessment
E) Needs assessment

Question 41

Which of the following are particularly useful for studying the impact of new laws or social programs that affect large numbers of people?

A) Multiple group before-and-after design
B) Time series design
C) Repeated measures panel design
D) Ex post facto design
E) Test design

Question 42

Describe the difference between randomization and matching. What are the advantages and disadvantages of each?

Question 43

When a researcher matches individual cases with similar individuals in a comparison group, the researcher is using which of the following methods?

A) Before-and-after design
B) Ex post facto group design
C) Individual matching
D) Aggregate matching
E) Group matching

Question 44

A politician campaigns on a platform of eliminating welfare benefits. One ad used by politician says: "Receipt of welfare benefits leads to poor school performance: People who receive welfare benefits have, on average, lower grades than people who do not receive welfare benefits." Construct an argument to challenge this causal reasoning. First, give three examples that drive home the point that association does not establish causation. Then, construct a plausible alterative explanation for low grades that does not involve an effect of welfare benefits on grades, but which does allow for their association.

Question 45

Why aren't experiments very generalizable? What can be done to improve the external invalidity of experiments?

Question 46

Which of the following includes several pretest and posttest observations, allowing the researcher to study the process by which an intervention or treatment has an impact over time?

A) Multiple group before-and-after design
B) Time series design
C) Repeated measures panel design
D) Ex post facto design
E) Test design

Question 47

Why aren't context and mechanism as critical as time order, association, and nonspuriousness for establishing causation?

Question 48

In what situations is it ethnical to deceive participants in an experiment? Provide an example that would ethically allow deception and an example in which the ethics of deception are more ambiguous.

Question 49

While researchers changed the work environment to see how it affected worker productivity, they got surprising results. No matter what conditions the researchers changed (such as increasing the lighting), the workers seemed to work harder simply because they were part of a special experiment. This is the origins of a concern about internal invalidity in experiments known as:

A) The Hawthorne Effect
B) The double-blind experiment
C) Compensatory rivalry
D) Selection bias
E) The Solomon Four Group design

Question 50

Two studies are conducted of the relationship between SAT scores and subsequent earnings. In Study A, students are the units of analysis and it is found that college students who had higher SAT scores have higher earnings in subsequent jobs. Write a statement about this finding that would be reductionist. Explain why the statement is reductionist. In Study B, colleges are the units of analysis and it is found that colleges in which the average student SAT score was higher also have, on average, higher earnings among their graduates. Write a statement about this finding that exemplifies an ecological fallacy. Explain what makes this statement an ecological fallacy.

Question 51

In terms of validity, experiments are weakest in terms of establishing:

A) Time order
B) Internal validity
C) Random assignment
D) Generalizability
E) Association

Question 52

Discuss the multiple forms of internal validity that create problems in experiments. (There are more than ten!) Describe each in your own words and provide an example that demonstrates why each potentially limits validity.

Question 53

You want to conduct research that documents the effects of watching violent cartoons on violence during children's playtime. Set up the following types of experimental and quasi-experimental designs: true experiment, non-equivalent group, before-and-after, ex-post facto, and Solomon four group. Which design do you think would produce the most valid results? Explain why.
Answer Key

Question 54

Sherman and Berk's (1984) study looking at the impact of arrest in actual domestic violence incidents is an example of a:
Experiment
B) Field experiment
C) Quasi-experimental design
D) Posttest
E) Law Enforcement Adequacy Test

Question 55

In Sherman and Berk's (1984) study of the effect of arrest on future domestic violence, police officers determined which subjects would be arrested based on random selection, made while officers were responding to actual calls for domestic disturbances. Their study is a good example of:

A) A double-blind study
B) A quasi-experimental design
C) A field experiment
D) The use of matching in group assignment
E) A Solomon Four Group design

Question 56

For researchers to ensure the results of their experiment worked for other groups, they decided to make several comparisons involving the same independent and dependent variables but with different groups, also known as:

A) Multiple group before-and-after design
B) Time series design
C) Repeated measures panel design
D) Ex post facto design
E) Test design

Question 57

Is it possible to determine whether a treatment led to a change in the value of cases on the dependent variable in an experiment in which there was no pretest? Explain your answer.

Question 58

Compare randomization and random sampling. What are their similarities and differences? In what way does each help to improve the validity of research conclusions? Why does the use of one of these techniques preclude the use of the other?

Question 59

Describe each of the five criteria for identifying a causal effect and provide an example of each. Which are best demonstrated through experiments? Why? Which do experiments not demonstrate well? How can experimental designs be altered to compensate? Is anything lost with quasi-experimental designs?