Deck 14: How to Encourage Utilization

Question 1

If program staff members are worried about the findings of an evaluation,

A) they will resist adopting ideas for change contained in an evaluation report.
B) they are likely to grasp at any suggestion.
C) they will be eager to innovate.
D) one can assume that the staff has pride in its work.

Answer 1

A

Question 2

When evaluators sense a fear of evaluation, they would be wise to

A) avoid conducting the evaluation since all reports will be distorted by the timid staff.
B) encourage the management to state clearly that program improvement and innovation are wanted, even if some innovations may turn out to be ineffective.
C) suggest in strong terms that the staff should get its act together.
D) point out to the staff that fear of evaluation is a symptom of a staff which feels there are problems with its own program.

Answer 2

B

Question 3

If unsuccessful program innovations are not identified,

A) others are more likely to implement ineffective programs.
B) no one will forget them.
C) a Type I error has occurred.
D) implementation problems may have caused the failure.

Answer 3

A

Question 4

Evaluators are more likely to see evaluations used if

A) reports are simple, clearly labeling failures as failures.
B) suggestions are only tangentially related to the program.
C) staff feel respected.
D) staff are concerned about program termination if goals are not met.

Question 5

The conclusions of evaluation reports should seldom

A) deal with failure.
B) give support for a program.
C) give blanket approval or disapproval of a program.
D) be based on specific data.

Question 6

Sometimes an innovative program will fail to do better than currently used approaches,

A) but the new approach may still have certain positive features (e.g. lower cost) that suggest it be adopted anyway.
B) but the possibilities of this are extremely small.
C) and the evaluator can expect to see recommendations carried out with little hesitation.
D) but for the sake of the morale of the staff, this failure should be ignored or explained away.

Question 7

When an evaluator fails to document successful achievement of outcome goals, but the innovation seems to makes sense anyway, evaluators

A) can speak of the innovation having "common sense plausibility" and urge that further evaluations be conducted.
B) can question the integrity of those who firmly supported it.
C) should immediately raise the possibility of a Type I error.
D) should actively try to persuade program supporters to change their attitudes.

Question 8

If only the more costly elements of a program are eliminated because of negative findings,

A) one should question the value of the evaluation.
B) the staff probably has excessive power.
C) the evaluator did not do a thorough job of communicating the negative nature of the findings.
D) the evaluation served a useful purpose, especially if additional study is to be done.

Question 9

Evaluators need not be overly discouraged if the implications of an evaluation are not taken into consideration immediately because

A) organizational politics are strong.
B) you can't win 'em all.
C) evaluators should not be concerned about utilization anyway.
D) evaluation findings are only one form of information that managers must use.

Question 10

The finding of "no differences" between a comparison group and a treatment group could

A) be due to poor evaluation methodology or inadequate analyses.
B) be due to sabotage on the part of the program staff.
C) create an arrogant attitude among the program staff.
D) be what most innovators of new programs want to see.

Question 11

Implementing the results of evaluation studies is encouraged when

A) the staff holds the evaluator in fearful awe.
B) the written results are permitted to "speak for themselves."
C) the evaluation feeds back into program planning.
D) evaluators keep a proper "hands off" attitude toward the program.

Question 12

Given the limited development of the science of program evaluation, weaknesses in every study could lead to all results being questioned. It is indefensible to use this fact to

A) explain away all program weaknesses identified by an evaluation.
B) call for further evaluation.
C) work closely with stakeholders to interpret the results.
D) suggest improvement in further evaluations.

Question 13

Acceptance of valid, unfavorable feedback is enhanced when managers can view such feedback

A) like a baseball score-its only a game.
B) like medical tests-if the results show that the patient is ill, the doctor does not reject the test.
C) with regard to the source of the feedback-students or employees do not have the big picture.
D) like a horse race-there can be only one winner, but the losers may well still be fast horses.

Question 14

Evaluation findings do not provide definitive conclusions about the value of a type of program because

A) evaluators do not use valid research designs.
B) evaluators often fudge reports to favor a program.
C) conditions at other locations are often different from the setting in which an evaluation was conducted.
D) reports are written which are sensitive to stakeholder information needs.

Question 15

Strongly polarized attitudes toward a policy, such as limitations on handgun ownership, will result in people who

A) are able to find common ground with each other to draw reasoned conclusions from an evaluation of the policy.
B) are eager for the evaluation findings.
C) trust the evaluator seeking to remain neutral on the issue.
D) are likely to select those findings of the evaluation that support their views and reject opposing ideas.

Question 16

Evaluation conclusions are not likely to be applied and indeed should not be applied when

A) a fixation on outcome evaluation led to the use of methods that could not have produced interpretable findings.
B) evaluations focus on the extent of program implementation.
C) the costs of implementing the findings are carefully described in the evaluation report.
D) stakeholders have had input into the recommendations.

Question 17

Examining reports of completed evaluations in a systematic, quantitative manner

A) is useful only to keep the evaluator from making gross errors in his/her work.
B) is probably a waste of time since each evaluation is unique.
C) can provide the program planners with invaluable ideas that have been effective.
D) is not possible from a statistical point of view.

Question 18

Program managers often look for a simple, definitive conclusion from an evaluation (e.g. good/bad), but many evaluators prefer to deal with the findings as working hypotheses on which to base

A) funding decisions.
B) research articles.
C) staff salary increases.
D) program improvement efforts.

Question 19

The most important contribution that an evaluator can make to a program is to

A) encourage the development of a learning culture among the program staff and organization management.
B) get staff to care about the participants of the program.
C) perform an extremely rigorous evaluation that would draw attention to the organization sponsoring the program.
D) be sure that the staff respects the credentials of the evaluator.

Question 20

It is valid to label a program as ineffective when

A) a statistical comparison between program and comparison groups is non-significant.
B) statistical comparisons between randomly-assigned program participants and a control group are non-significant and statistical power is 95%.
C) the program lacks common sense plausibility but the statistical test is significant (i.e. p < .05).
D) statistical comparisons between randomly formed program and control groups are non- significant and statistical power is less than 5%.

Question 21

Internal evaluators have an advantage over external evaluators in that internal evaluators

A) are often friends of the program staff or administrators.
B) do not have a long-term commitment to the agency.
C) can repeatedly remind program managers of the evaluation and its findings.
D) have a perspective gained by working in many different agencies.

Question 22

What are the steps that evaluators can take on their own to reduce the fear of evaluation?

Question 23

What are the steps that evaluators can take to encourage others to take to reduce the fear of evaluation?

Question 24

Although some anxiety about evaluation is normal, a high level of evaluation apprehension can lead to ineffective reactions to evaluations that are not totally favorable. Describe some of the fears and show how they lead to problems between evaluators and staff members.

Question 25

Describe how "the evaluator's perspective" differs from that of the staff and program managers.