Deck 11: Using Experiments to Evaluate Programs

Question 1

Although some quasi-experimental designs are fairly effective in separating the effect of the program from viable alternative interpretations,

A) they are not feasible in agency settings.
B) they are more disruptive of agency procedures than are experiments.
C) a lingering doubt may remain about internal validity.
D) staff will greatly object to quasi-experimental methods.

Answer 1

C

Question 2

Whenever preexisting, intact groups are used for an evaluation,

A) one can treat the groups as if they were equivalent.
B) one can be sure that the participants were randomly assigned to the groups.
C) the evaluation is readily interpretable.
D) the groups usually differ on many variables.

Answer 2

D

Question 3

When an evaluator recognizes a potential threat to the internal validity of an evaluation,

A) it may be impossible to control for it short of randomization.
B) correlational designs are needed.
C) using just one group would permit the threat to be ignored.
D) a pretest/posttest design can be used.

Answer 3

A

Question 4

A randomized experiment

A) cannot be a good evaluation design.
B) relieves the evaluator of worrying about most of the threats to internal validity.
C) has high external validity.
D) lowers Type I error compared to a quasi-experimental design.

Question 5

Pretests are recommended even when the evaluation design consists of randomly formed groups because

A) correlations between amount of service and change scores improve interpretability.
B) programs are not implemented as designed.
C) pretests will permit some interpretations even if randomization procedures are compromised.
D) non-equivalent control groups need to be equated using a pretest.

Question 6

The use of pretests in an experimental evaluation may introduce the internal validity threat of

A) regression to the mean.
B) history.
C) maturation.
D) testing.

Question 7

Service providers sometimes object to random assignment to groups because they try to match particular social services to particular individuals,

A) however, it is impossible to match services to program participants.
B) however, there is considerable variation in the services provided to people with similar needs even in the absence of an evaluation.
C) therefore, evaluators often permit program staff to form the comparison groups.
D) moreover, services are usually planned to meet each participant's unique needs.

Question 8

The use of pretests with an experimental evaluation

A) permits one to control for random individual pretest differences, thereby lowering Type II error.
B) gives the appearance of credibility to the evaluation but is of no practical use.
C) is a violation of experimental design.
D) will not be understood by stakeholders.

Question 9

A good approach to interpretation when a program fails to achieve its objectives is

A) to hold tight to previously stated objectives and call a spade, a spade.
B) to recognize that the objectives may have been too ambitious and to focus on some phase of the program that worked.
C) to treat well-conceived programs as experiments and search for what can be learned to improve the next version of the program.
D) to point out that repeated failures probably indicates low staff motivation.

Question 10

When determining the costs of a valid, experimental evaluation of a new medical treatment, innovators often overlook the costs of

A) making a Type II error.
B) data collection.
C) monitoring the program and control groups participants.
D) not setting the probability of a Type I error accurately.

Question 11

An evaluation that _____________ is a waste of resources.

A) does not focus on causes
B) cannot be interpreted
C) uses correlation coefficients
D) does not support the program

Question 12

Usually a new program cannot be implemented in all parts of an organization simultaneously. This pattern

A) is unfortunate for evaluation.
B) can be utilized by an evaluator to create control and experimental groups.
C) creates threats to internal validity that are impossible to control.
D) violates ethical standards since this is unfair to those who receive a program later.

Question 13

One can expect the most objections to a proposal for an experimental evaluation of a program when

A) a new program is introduced.
B) resources are scarce.
C) there is controversy about the effectiveness of the program.
D) the program has been in place and accepted for years.

Question 14

When an ambitious program evaluation of a public policy (such as welfare reform) fails to find that the new policy had a positive impact,

A) the effort in implementing and evaluating the program was a waste of time and money.
B) one should assume that the evaluator did a poor job.
C) one can expect that the quality of the debate over the policy will be more enlightened.
D) the policy will be changed within months of the evaluation report.

Question 15

Once stakeholders agree that random assignment to groups will be permitted, it is crucial that

A) the evaluator periodically monitor the outcome variables.
B) stakeholders be permitted to make the actual group assignments.
C) different rates of attrition among groups not be permitted to introduce bias.
D) regression to the mean be carefully examined.

Question 16

Diffusion or imitation of the program by people in a non-treatment control group tends to

A) make an effective program look less effective than it is.
B) make an ineffective program look more effective than it is.
C) make a good program appear to have a negative impact.
D) nullify the effects of attrition from control groups.

Question 17

Attempts to compensate the control group tends to

A) make an effective program look less effective than it is.
B) make an ineffective program look more effective than it is.
C) make a good program appear to have a negative impact.
D) nullify the effects of regression to the mean.

Question 18

When control groups feel rivalry with the experimental group, it is possible that

A) attrition will increase among the members of the experimental group.
B) an effective program will look less effective than it is.
C) an ineffective program will look more effective than it is.
D) the performance of the experimental group will improve.

Question 19

If a control group experiences resentful demoralization because its members realize that they are not receiving desired program benefits, it is possible that

A) attrition will increase among the members of the experimental group.
B) an effective program will look less effective than it is.
C) the program will look more effective than it is.
D) the performance of the control groups will improve.

Question 20

When an outcome evaluation is designed as an experimental evaluation,

A) it is unnecessary to worry about implementation since a poorly implemented program will probably yield a poor outcome.
B) one can relax since the data can be clearly interpreted.
C) the data will "speak for themselves" when the analysis is done.
D) it is still essential to examine the degree of program implementation.

Question 21

One of the many differences between quasi-experiments and true experiments is that

A) threats to external validity do not have to be considered in true experiments.
B) outcomes observed reflect the degree of experimental control.
C) a true experiment does not require a pretest.
D) results from quasi-experiments are usually more interpretable than results from true experiments.

Question 22

One objection made about true experiments that is not valid is that

A) the control group may miss out on a valuable program.
B) random assignment creates some artificiality in the program setting.
C) they are more expensive than pretest/posttest designs.
D) evaluators are only interested in data, not people's lives.

Question 23

When services or medical treatments are offered before being carefully evaluated,

A) little harm is done because if a treatment is ineffective, people will quickly notice and discard it.
B) much harm can be done because some ineffective treatments have come to be considered standard treatments.
C) theoretically some harm could be done, but people seldom implement a harmful program.
D) much harm will occur because nearly all non-evaluated programs are harmful.

Question 24

Interpretations of outcomes of experimental designs are much more useful when

A) stakeholders focus on statistical significance.
B) t tests are used.
C) the amount of change a program could cause has been discussed before examining the findings.
D) correlations are calculated for program and control groups separately.

Question 25

Describe the organizational conditions that favor conducting an experimental evaluation.

Question 26

Describe the threats to internal validity that can hinder the interpretability of experimental evaluations.

Question 27

Describe the reasons that program staff often give when they object to conducting an evaluation using an experiment involving random assignment to groups.

Question 28

How might an evaluator seek to reduce attrition among the members of control groups?

Question 29

Distinguish the internal validity threat of history from the internal validity threat of local
history.