Deck 10: Quasi-Experimental Approaches to Outcome Evaluation

Question 1

To show that a program had an effect on an outcome variable, it is necessary to

A) inform all stakeholders beforehand.
B) dismiss Type II errors.
C) explain why some people did not benefit from the program.
D) eliminate explanations of change not related to the program.

Answer 1

D

Question 2

The ways in which evaluators expand on the pretest/posttest design in order to control for threats to internal validity include the following EXCEPT for

A) increasing the size of the sample to reduce Type I and Type II errors.
B) observing the participants at additional times before and after the intervention.
C) observing additional people who have not received the program.
D) using a variety of variables, some expected to be affected by the program, some not.

Answer 2

A

Question 3

The primary reason to increase the number of observation periods is to

A) examine non-program effects related to history and maturation.
B) reduce the chances of a Type I error.
C) increase external validity.
D) simplify the statistical analyses necessary.

Answer 3

A

Question 4

A time series evaluation design leads to

A) an unavoidable loss of statistical power and an increase in Type II errors.
B) requires repeated observations of the same entity at many points in time.
C) includes the development of measures of alternative outcome variables.
D) is usually needed for programs administered to small groups who are part of a large organization.

Question 5

In the graph below, all preintervention observations are less favorable than the post interventions observations;

A) clearly the intervention had an impact.
B) regression to the mean is the most likely reason for the change.
C) the possible effects of maturation or history should be considered carefully.
D) partial correlations should be calculated before drawing any conclusions.

Question 6

A pattern like the one in the graph below could occur because

A) staff members' enthusiasm led to a short term improvement in outcomes.
B) program participants do not accept the intervention.
C) maturation effects may be strong.
D) the evaluator became more accurate in assessing success during the preintervention observations (i.e., instrumentation problems).

Question 7

A particularly strong time series design is available when

A) an outcome variable is based on self-report.
B) the intervention can be removed and reinstated later.
C) the samples are small.
D) local history has an effect on the outcome variable.

Question 8

Smoothing a graph of observations made over a time series

A) is best done visually by passing a smooth line through the data points.
B) is essentially dishonest and unethical since means or medians are substituted for the real data.
C) can be done by substituting the median of 3 adjacent data points for the original points.
D) can make any pattern of data look like an effect of a program.

Question 9

When a group similar to the program group is available, a good quasi-experimental evaluation design is a

A) time series design.
B) non-equivalent, control group design.
C) partial correlational analysis.
D) pretest/posttest design.

Question 10

The major weakness of the nonequivalent, control group design concerns

A) instrumentation effects.
B) testing effects.
C) low statistical power.
D) selection effects.

Question 11

If the program participants in a non-equivalent control group design are chosen for a program because they are extremely needy, it is likely that a

A) regression effect will be a viable alternate interpretation.
B) instrumentation effect will be a viable alternate interpretation.
C) attrition effect will be a viable alternate interpretation.
D) testing effect will be a viable alternate interpretation.

Question 12

If one selects the poorest readers from an elementary school class using a standard reading test and then places them in an ineffective reading program, retesting will likely show an improvement in scores due to

A) attrition.
B) regression to the mean.
C) regression to the mean and maturation.
D) instrumentation and history.

Question 13

An uncontrolled threat to internal validity could

A) make a program appear effective.
B) make a program appear ineffective.
C) make a program appear either effective or ineffective.
D) raise the chance of a Type I error.

Question 14

The regression discontinuity design is

A) a specific form of a time series design.
B) meant to make use of the threat of regression to the mean.
C) a specific form of a non-equivalent control group design.
D) impossible to use in evaluation settings.

Question 15

The regression discontinuity design can be analyzed using

A) a t test.
B) medians of three adjacent points.
C) change scores.
D) multiple regression.

Question 16

A most crucial concern in selecting additional dependent variables to compare to an outcome variable of interest is to be sure that these additional variables will

A) not be affected by the program.
B) be affected by the program.
C) increase the external validity of the evaluation design.
D) not reduce the probability of a Type I error.

Question 17

An especially strong quasi-experimental design combines the features of

A) posttest only design with regression discontinuity.
B) time series with a posttest only design.
C) a time series design with a non-equivalent control group design.
D) regression discontinuity with regression artifacts.

Question 18

Using two time series with interventions delivered at different times

A) is especially affected by maturation influences.
B) is not affected by history.
C) can be used with only one group.
D) includes a replication.

Question 19

Comparison groups are best chosen in order to control for

A) specific threats to internal validity relevant to a particular evaluation setting.
B) all possible threats to internal validity.
C) unauthorized tampering with the administration of the outcome variables by worried staff members.
D) sample size variations.

Question 20

Work or school absenteeism is a particularly good outcome variable because

A) it cannot be faked.
B) only good reasons would keep employees or students at home.
C) it is an important variable to businesses and schools and is routinely recorded.
D) no threats to internal validity can influence absenteeism.

Question 21

A time series design used when an intervention has occurred during a short time period (e.g. one week out of ten observation periods spread out over ten weeks) is called a(n)

A) pretest/posttest time series.
B) interrupted time series.
C) intervention time series.
D) regression time series.

Question 22

Nonequivalent control group designs

A) nearly always ensure high internal validity.
B) are best used if a selection-by-maturation interaction needs to be controlled.
C) are seldom used in program evaluation.
D) utilize at least two groups which are not randomly formed.

Question 23

What would be some appropriate comparison groups for a quasi-experimental evaluation of a change in a probation law? A change in the requirements for a college major? A review course for the Graduate Record Examination?

Question 24

Suppose that a hospital made an effort to improve the meals served to patients. How could a control construct design be used to learn if patients liked the meals better than before the change?

Question 25

The selective control design refers to combining various quasi-experimental designs to control the threats to internal validity that are thought to be the most viable to a particular evaluation. Suppose that a college instituted a program to help retain freshmen so that a greater proportion re-enrolled for the sophomore year than in the past. The obvious comparison is between the re-enrollment rates of the current freshmen and the previous year's freshmen. What threats to internal validity weaken the conclusions that one could draw from such a comparison? Expand on this basic design, seeking to control at least some of the threats to the internal validity of the design.

Question 26

Suppose that the North Dakota state legislature raised the high school requirements for mathematics in an effort to assure their citizens had a better chance to gain admission to more selective colleges. Further suppose that the legislature of South Dakota adopted the identical requirement two years later. Explain which quasi-experimental design you would use to evaluate the outcomes of the changes in standards. What negative side effects might be observed among high school students?