Deck 9: Outcome Evaluations With One Group

Question 1

Pretest-posttest designs permit the evaluator to learn whether

A) the program caused a change in the outcome variable.
B) the participant improved during the program.
C) improvement was great enough to reject threats to internal validity.
D) Type I errors could account for a change.

Answer 1

B

Question 2

A posttest-only evaluation design is often useful when

A) the stakeholders do not have advanced training in research methods.
B) there is a definite, objective criterion of successful outcome.
C) there is little time or resources for an evaluation.
D) program success is hard to define.

Answer 2

B

Question 3

Internal validity refers to whether

A) a statistically significant change occurred.
B) Type II errors occurred.
C) a research design provides sufficient control that one can learn whether the program caused a change in an outcome.
D) the findings can be generalized to other similar groups.

Answer 3

C

Question 4

Evaluation designs should be selected so that

A) the stakeholders' questions can be answered.
B) basic knowledge in the field can be gained.
C) the evaluation will merit publication.
D) the expectations of the major stakeholders will be supported.

Question 5

The use of small samples can lead unsophisticated evaluators to conclude falsely that

A) Type I errors were held to only 5%.
B) findings could be applied to additional populations.
C) a program had no impact.
D) statistically significant results were observed.

Question 6

Conclusion validity refers to

A) statistical significance.
B) high internal validity.
C) high external validity.
D) adequate clinical power.

Question 7

The question of how much improvement in a treatment group is enough

A) cannot be settled empirically.
B) should be determined after examining the results.
C) is based on internal validity considerations.
D) is an irrelevant issue.

Question 8

Correlating the amount of service received with the condition of the participants after receiving services from a program

A) can sometimes suggest that a service is not effective.
B) cannot be useful if internal validity is low.
C) simply requires correlating a change score with a measure of service.
D) is more valid in isolating causes than a true experiment.

Question 9

The difference between the mean of the outcome variable after a program and the mean of the variable measured before the program divided by the standard deviation of the pre-program scores is an index of

A) effect size.
B) correlation.
C) statistical significance.
D) generalization.

Question 10

The threat to internal validity (called "maturation") refers to

A) predictable changes in people that can be expected solely due to the passage of time.
B) events happening in the community that will change the behavior of the program participants.
C) the improved skill of a program staff as they become more experienced.
D) differences between people who want a program and those who do not want it.

Question 11

The threat to internal validity called "history" refers to

A) predictable changes in people that can be expected solely due to the passage of time.
B) events happening in the community that will change the behavior of the program participants.
C) the improved skill of the staff as they come to understand the history of the program.
D) improvement in the skills of the evaluator in measuring outcome variables.

Question 12

Local history is a form of threat to internal validity referring to

A) predictable changes in people that can be expected solely due to the passage of time.
B) events happening in the community that will change the behavior of the program participants.
C) events that are unique to a particular program group.
D) the changing nature of an evolving program.

Question 13

The fact that people who choose to, or who are chosen to be, in a program are different from those who are not in a program creates the threat to internal validity called

A) history.
B) regression to the mean.
C) selection.
D) testing.

Question 14

Another name for the threat to internal validity called "attrition" is

A) history.
B) participant loss.
C) instrumentation.
D) regression to the mean.

Question 15

If an evaluator judges the rate of success of a program on the basis of those who complete the program, she is likely to be failing to deal with the threat to internal validity called

A) attrition.
B) regression to the mean.
C) testing.
D) history.

Question 16

When initially extreme scores are followed by less extreme values (even when hypothesized), one wonders whether _____________ can account for the change.

A) maturation
B) selection
C) regression to the mean
D) testing

Question 17

Regression to the mean is a threat to the internal validity of the pretest/posttest design whenever

A) a long time transpires between the two observation periods.
B) program participants get older between the two observations.
C) people are chosen for a program because initial observations showed that they needed the program more than others did.
D) evaluators gain experience making the observations for the pretest and the posttest.

Question 18

The threat to internal validity called "testing"

A) refers to changes in behavior due to the observation techniques.
B) refers to the use of measurement procedures themselves.
C) refers to school-based program evaluations.
D) cannot be a threat to pretest/posttest designs.

Question 19

The threat to internal validity called "instrumentation" refers to

A) failures in the electronic equipment sometimes used to collect data.
B) problems that are beyond the evaluator's control.
C) changes in the meaning of the scores of the measures used.
D) that which cannot be a threat to pretest/posttest designs.

Question 20

Repeatedly dividing the sample on the basis of demographic descriptors to look for differences in the means of outcome variables for different subgroups of the sample

A) raises the chances of a Type I error.
B) raises the chances of hindsight error.
C) lowers the internal validity of a study.
D) increases the level of external validity.

Question 21

The pretest/posttest design using only the program's clients can help introduce the staff to the merits of program evaluation because

A) this is a very sophisticated research design.
B) the threats to internal validity are all controlled except for instrumentation and the interaction of selection by maturation.
C) a plan to compare the program group with people in other programs might be rejected because the staff feels too threatened.
D) it is easier to take the needs of all stakeholders into account using this design.

Question 22

Retrospective pretests were developed to be sensitive to

A) hostile participants in the evaluation.
B) expert opinion about what went wrong in the implementation of the program.
C) how people believe that their parents raised them.
D) changes created by the program that may have changed how participants understand themselves.

Question 23

Explain which threats to internal validity need to be considered in a comparison of the mean scores on a test administered on the first and last days of this course.

Question 24

Contrast the effects of attrition depending on whether the most successful participants leave prematurely (because they feel they no longer need the program) or the less successful participants leave prematurely (because they feel that they are not making progress).

Question 25

Illustrate how advances in medical technology can make it appear as though the incidence of a disease has changed over time.

Question 26

Consider a program to assist new parents to develop parenting skills. Why might their views of their own preprogram skill levels change as a result of their own development?

Question 27

Develop two hypothetical situations in which threats to internal validity would serve to create
an apparent positive program effect and two in which an apparent negative effect would be
created.