Deck 5: Ethics in Program Evaluation

Question 1

If an evaluator believes that a potential client has requested an evaluation that will not meet the agency's real needs, the evaluator should

A) simply do the best job she can.
B) attempt to redefine the form of evaluation to be done.
C) tell the client what is to be done.
D) seek to gain power by going to the client's supervisor or board of directors.

Answer 1

B

Question 2

Ethical issues in program evaluation seem more compelling than in basic research because

A) evaluators are less skilled than basic researchers.
B) program evaluations are often replicated, thus assuring that errors will be detected.
C) program evaluations are seldom replicated and the findings may be applied even if in error.
D) service program staff members are notoriously unethical.

Answer 2

C

Question 3

Program evaluators stop thinking about ethical issues

A) once a project is planned in an ethical manner.
B) after it is clear that the data have been gathered ethically.
C) after a report of an ethically conducted evaluation has been written.
D) at their peril because ethical standards can be violated at any stage or even after the report is finished.

Answer 3

D

Question 4

"Informed consent" refers to

A) the contract used to employ an evaluator.
B) obtaining the agreement from people to supply data for an evaluation after they have been adequately informed about what is expected of them.
C) the consent that program staff must give to the evaluator who has been requested to evaluate their program; without such consent, the evaluation cannot be done.
D) participation of people in the service program.

Question 5

Evaluators are especially concerned about the possibility of invalid conclusions because

A) evaluations are more likely to have an impact on people's lives than basic research reports do.
B) what they conclude cannot be applied.
C) they work in applied settings in which mistakes cannot be tolerated.
D) evaluation is a young field that has yet to prove its worth.

Question 6

If after beginning an evaluation the evaluator comes to feel that the evaluation plan cannot be carried out in a way that will yield a valid evaluation, the best alternative is to

A) do the best one can do, recognizing that no evaluation is perfect.
B) seek to renegotiate the evaluation plan so that it describes a plan that can be carried out well.
C) provide a favorable evaluation to the stakeholder paying for the project.
D) refuse to complete the project on the grounds that the findings will not be valid.

Question 7

In conducting an evaluation of an innovative program, the most important responsibility of the evaluator is to

A) conduct a valid evaluation.
B) consider the generalizability of any positive findings.
C) be sure that the most needy individuals get into the program.
D) be sure that the evaluation does not harm the participants.

Question 8

Evaluators prefer to have informed consent from all participants in an evaluation. When a full description of the program groups (for example, innovative program and control groups) threatens to undermine the internal validity of the evaluation, a good way to decide if full
disclosure is necessary is to

A) ask the major stakeholders what is the best procedure.
B) argue that a valid evaluation is more important than program participants having full information.
C) consider the cost to the participants of each group if they did not have such information.
D) insist that full disclosure is always required.

Question 9

Prior agreement from people who are to be sources of data in program evaluations can be called "informed consent" only when

A) an Informed Consent form has been signed.
B) the evaluator is present when the form is signed.
C) agreement has been given only after sufficient information has been provided to permit understanding the requirements of the evaluation.
D) the sponsor of the program fully understands the evaluation and what is expected of program participants.

Question 10

Problems that develop when an evaluator becomes an advocate for the program being evaluated include:

A) the loss of objectivity needed to conduct an evaluation.
B) the temptation to design evaluations that are too complex.
C) being forced by the staff to produce only favorable evaluations.
D) the development of hostility between evaluator and program staff.

Question 11

The term stakeholder refers to

A) a tool used in barbecue restaurants.
B) the evaluators of a program.
C) anyone affected by the program being evaluated.
D) all people who are not affected by the program evaluation.

Question 12

The evaluator needs to be concerned about serving the information needs of stakeholders because

A) these groups determine whether an evaluator will be paid.
B) stakeholders who feel that they have been ignored will seldom use the evaluation.
C) they must be mollified even if their input is not essential.
D) the evaluator will learn little from them.

Question 13

A problem with using measurement tools that do not validly measure the outcome of a program is that

A) the evaluation is not adequately scientific.
B) the stakeholders are less likely to understand the evaluation report.
C) a biased evaluation favoring the program is likely.
D) the actual outcome may be obscured.

Question 14

Using a general school achievement test when a measurement of a specific skill is needed

A) can only lead to a favorable, but invalid report.
B) will lead to very dissatisfied stakeholders.
C) is not likely to be sensitive to the effect of the program.
D) will increase the likelihood that the report will be accepted.

Question 15

Interviewing to gather data for an evaluation

A) is usually used in low-budget jobs.
B) indicates that the evaluator does not understand scientific measurement.
C) is seldom necessary for conducting a valid evaluation.
D) requires considerable training and skill.

Question 16

When considering evaluation designs, ethical evaluators seek to

A) match the research design and the informational needs of the stakeholders.
B) design the most scientifically valid outcome evaluation.
C) convince stakeholders that standardized tests are always better than other measurement procedures.
D) trace causal relationships between the program elements and the outcome even when examining a new program.

Question 17

The most scientifically sound evaluation design can fail to be useful if

A) the evaluator never checked to learn if the program elements were actually implemented.
B) interviews are relied upon for the data.
C) complex statistical analyses are not done.
D) the evaluation results are not published in scientifically respectable journals.

Question 18

It would be unusual for an evaluator to discover that

A) there were conflicts in the values of different stakeholder groups.
B) there were no conflicts in the values of different stakeholder groups.
C) the various stakeholder groups have difficulty articulating how the program is supposed to affect participants.
D) the various stakeholder groups draw the same interpretation from the results of an evaluation.

Question 19

A stakeholder group that is seldom consulted in planning an evaluation is the

A) management of the facility sponsoring the program.
B) staff of the program being evaluated.
C) leadership of the program being evaluated.
D) program participants or clients.

Question 20

Sources of different interpretations of the meaning of an evaluation that are largely beyond the control of an evaluator are disagreements over the

A) relative values placed on the outcomes of a program.
B) ways variables should be measured.
C) sources of evaluation data to be used.
D) ways to measure the cost of the program.

Question 21

Evaluation results can fail to detect the positive outcomes of an effective program when

A) Type I errors are large.
B) the measures or observation procedures used are not reliable.
C) the staff goes to extra effort during the data collection period.
D) evaluators overlook negative side effects.

Question 22

Evaluators are more concerned about Type II errors than are basic researchers because

A) Type II errors can harm the participants in basic research.
B) the probability of making a Type I error can be controlled.
C) concluding that an effective program does not work can deprive people of services.
D) the probability of a Type II error is easy to calculate and explain to stakeholders.

Question 23

A major criticism of the designs of published evaluations is that

A) Type I errors are too large.
B) too many participants are tested or observed.
C) too much time is wasted on examining the implementation of the programs.
D) the designs were too weak to detect even moderately large effects.

Question 24

Focusing solely on the stated goals of a program when designing an evaluation might mean that

A) undesirable side effects of the program might be overlooked.
B) undesirable side effects of the program might become the focus of the evaluation.
C) the program's critics might have too much input into the evaluation.
D) the program's sponsors would object to the evaluation design.

Question 25

Unexamined values held by an evaluator might distort the results of an evaluation because

A) the choice of variables to measure may reflect these values.
B) the sponsors of the program might recognize these assumed values.
C) evaluators usually adopt the values of less powerful groups in society.
D) statistical methods do not incorporate values.

Question 26

Detecting negative side effects of programs

A) are the result of Type I errors.
B) are more likely in programs involving children.
C) can lead to a better program if side effects are recognized and reduced.
D) will earn the evaluator accolades from the program's sponsors.

Question 27

Statistical Type I and Type II errors

A) refer to false positive and false negative conclusions, repectively.
B) need not be a concern of program evaluators.
C) refer to false negative and false positive conclusions, respectively.
D) are sometimes desirable if they lead to an evaluation conclusion that is favorable to the program.

Question 28

Type II errors -----

A) can be harmful if they lead to the elimination of beneficial services.
B) can be avoided by testing a few participants instead of all of the participants.
C) lead to conclusions that a program is effective when it is not.
D) are, fortunately, rare in program evaluation.

Question 29

Practicing evaluators report that the most common ethical problems involve

A) evaluators who do not make contact with stakeholders.
B) the refusal of clients to reveal their honest opinions.
C) stakeholders who are more interested in basic research than in practical information.
D) stakeholders who seek to influence the conclusions of evaluations.

Question 30

Contrast the interests of college students and faculty in creating plans for an evaluation of general education requirements in an arts and science college.

Question 31

How should an evaluator respond to a program manager who requests access to the original survey forms that his employees completed after employees had been assured of confidentiality by the evaluator?

Question 32

Why are Type II errors particularly likely in program evaluations?

Question 33

Outline the most important ethical principles for program evaluators.

Question 34

How might one present the idea of random assignment to a program group or to a control(wait-list) group? Assume that you are asked to prepare a paragraph or two for the staff members who are to obtain informed consent from the people seeking service from the agency sponsoring the evaluation?