Deck 13: Evaluation Reports: Interpreting and Communicating Findings

Question 1

The introduction to an evaluation report usually contains

A) a summary of the findings.
B) the statistical analyses.
C) a description of the setting of the program.
D) an analysis of the conflicts among the stakeholders groups interested in the evaluation.

Answer 1

C

Question 2

The introduction should deal with the reasons for conducting an evaluation because

A) the evaluator needs to justify his/her work.
B) the kind of evaluation done and the implications drawn will follow from the reasons for conducting an evaluation.
C) few readers will be clear on the statistical analyses.
D) formative evaluations, although few in number, need to be clearly labeled.

Answer 2

B

Question 3

It is important to be sure that readers of an evaluation report recognize what is NOT dealt with in an evaluation

A) because many readers hold limited definitions of what a program evaluation is.
B) because readers may be very willing to apply whatever they find in an evaluation report.
C) and why stakeholders should seldom have much input into the study.
D) even though the concept of program evaluation has become widely understood in most organizations.

Answer 3

A

Question 4

Including a description of the program participants observed during program evaluations helps readers to

A) find ways to discredit the evaluation.
B) depersonalize the participants.
C) rationalize program failures so that funding is not withdrawn.
D) judge the degree the participants are the ones planned for.

Question 5

Methods of selecting a sample are described in reports because -----

A) internal validity may be threatened by some methods of finding a sample.
B) testing effects can confuse some participants.
C) correlational analyses are always more informative when the sample is homogeneous.
D) Type I errors can be affected by an improper approach to selecting a sample.

Question 6

An evaluation report could be more difficult to write than an article on basic research because

A) there are often more observations in an evaluation.
B) the readers of an evaluation report will have more statistical sophistication.
C) the readership of an evaluation report may be more heterogeneous.
D) the readership of an evaluation report will be more skilled in detecting threats to internal validity.

Question 7

In presenting evaluation results

A) it is sufficient to present statistical significance tests since the crucial issue is whether the results are reliable.
B) it is essential to compare group differences to the standard deviations of the variables.
C) the data will speak for themselves so detailed presentations need not be made.
D) the meaning of group differences can only be interpreted in the light of the importance of the variable observed.

Question 8

Small improvements in important variables-death rate, for example-are

A) worth finding and stressing in reports.
B) simply small effects and not worth stressing.
C) just as important as any other statistically significant finding.
D) to be judged on the basis of the t test or size of the correlation coefficient.

Question 9

Small improvements in outcome variables

A) should be omitted from evaluation reports in order to focus attention on large effects.
B) can be crucial if the variables are important or affect many people.
C) should not be overly stressed.
D) are important only if the results are statistically significant.

Question 10

The "adequacy" of a program refers to whether

A) the program produces statistically significant changes in outcome variables.
B) financial supporters of the program are satisfied with the program.
C) the program is sufficiently ambitious to serve the actual needs of the target population.
D) the program has been implemented as specified in the program's plans.

Question 11

A written evaluation report includes

A) an abstract much like those used with research reports.
B) elaborate footnotes detailing the precise statistical tests used.
C) detailed statistical results in the body of the report so that readers can evaluate the analysis.
D) a summary that provides readers with an overview of the evaluation and recommendations.

Question 12

The use of various versions of evaluation reports is prompted by the need to

A) hide some outcomes from certain stakeholders.
B) write for a variety of stakeholders.
C) convince all stakeholders of the value of the program.
D) thoroughly document program failure to avoid rationalization by the program staff.

Question 13

Mixing quantitative summaries of observations and qualitative descriptions of the program

A) will illuminate the meaning of the numbers and the descriptions.
B) is likely to confuse readers.
C) cannot be condoned by methodologically sound evaluators.
D) inevitably opens the door to sloppy evaluations.

Question 14

Large sample sizes tempt evaluators and writers of popular reports of evaluations (e.g. newspaper reporters) to

A) dismiss the work as overly expensive.
B) ignore the representativeness of the sample.
C) question whether the sample represents the people served.
D) conduct overly simple analyses.

Question 15

Internal evaluators often experience the problem of being unable

A) to carry out correct statistical analyses.
B) to observe samples large enough to detect even moderately-sized effects.
C) to present their finding to the right stakeholders.
D) to understand the program in the context of the larger organization.

Question 16

In presenting a written report, a conflict about who has the right to see the report first may be reduced by

A) releasing it to all stakeholders simultaneously.
B) making a distinction between a draft report and the final report.
C) allowing those who pay for the evaluation to have the report first.
D) making those who want to see the report request it specifically.

Question 17

A strong argument for letting the program manager and staff see a near-to-completed draft before other stakeholders do is to --

A) learn about mistakes in factual matters.
B) respect friendships that may have developed in the course of the evaluation.
C) permit them to develop rationalizations about program failures.
D) encourage recommendations to be put into effect before others learn about problems.

Question 18

Evaluation data, unlike basic research data, belong to

A) the evaluator.
B) the program participants who provided data.
C) the program staff members.
D) whomever financed the project.

Question 19

Disputes about release of evaluation findings are best

A) resolved by the funding stakeholder.
B) avoided by making explicit agreements about what can be done with the data before the evaluation is conducted.
C) handled by the evaluator's best judgment since the evaluator conducted the evaluation.
D) placed in the hands of the manager and staff of the program that was evaluated.

Question 20

Program participants

A) have no right to evaluation data.
B) may have evaluation data if they are personally paying for the service.
C) could be sent a summary of the evaluation, which includes no confidential information.
D) should be approved on a case-by-case basis before receiving any program evaluation results.

Question 21

Evaluators recognize that an oral presentation of findings

A) is essential in assuring that stakeholders understand the findings.
B) is of marginal importance in communicating the findings of the evaluation.
C) can replace a written report.
D) permits the evaluator to alter the conclusions depending on the reaction of the stakeholders.

Question 22

Plans for communicating the findings of evaluations

A) should be developed while the evaluation is being planned.
B) can be dealt with once the findings are available.
C) should be kept confidential during the analysis of the observations.
D) should be left in the hands of the stakeholder funding the evaluation.

Question 23

In forming plans to communicate evaluation findings, evaluators

A) should follow a standard procedure with all program evaluation projects.
B) remember that a confidential approach to communication is likely to create interest in the report and greater acceptance.
C) are most successful if they are flexible and develop a communication plan that matches the specific needs of the setting of the project.
D) stress the scientific nature of program evaluation even if that means that some potential users cannot understand the findings.

Question 24

When preparing evaluation reports, it is important to reflect on

A) the gullibility of most stakeholders involved in social services.
B) common misunderstandings of statistical analyses.
C) the likelihood that evaluations usually find large program effects.
D) the need to maintain good interpersonal relationships even if it means shading the findings to match the manager's views.

Question 25

Distinguish between the purposes of a written report and those of an oral presentation of the findings of a program evaluation.

Question 26

Contrast the different emphases that distinguish an evaluation report from an article in a research journal.

Question 27

Why is it important to send progress reports to stakeholders while the evaluation is underway?

Question 28

Contrast the issues that would be the focus of (a) program evaluation report available to program managers, versus (b) a report prepared for recipients of program services.

Question 29

Describe some ways that recommendations could be offered that increase the probability of their being totally rejected by the stakeholders.