Deck 3: Developing and Using a Theory of the Program

Question 1

The selection of indexes of program effectiveness is usually difficult because

A) program staff members refuse to specify what changes are to occur in their clients.
B) different stakeholders have different views about what the most important criteria of program success are.
C) program effectiveness cannot be measured.
D) it is unethical to measure program effectiveness when participants need the program's services.

Answer 1

B

Question 2

When there is disagreement about the choice of the criteria of program effectiveness, the best procedure for program evaluators would be to

A) take over the choice of criteria.
B) use the criteria preferred by the stakeholder who is funding the evaluation.
C) use the criteria preferred by the program staff.
D) arrange discussions among stakeholders to find common concerns and to gain agreement to measure a variety of criteria.

Answer 2

D

Question 3

The program goals are usually described to evaluators because

A) goals help evaluators to determine what variables to measure.
B) evaluators confine their attention to the goals that program sponsors can describe.
C) it is impossible to carry out any form of evaluation when an evaluator does have not an explicit statement of goals.
D) program goals will show evaluators which negative side-effects should be studied.

Answer 3

A

Question 4

The criteria used in a program evaluation may include an assessment of how completely the program has been implemented

A) whenever evaluators doubt the honesty of the program staff.
B) whenever severe side-effects are likely to lead the staff to reduce the strength of the program.
C) because practical difficulties can lead well-intentioned people to depart from program plans.
D) because stakeholder groups usually view each other with hostility and suspicion.

Question 5

In outcome evaluations, the achievement of intermediate goals is often examined because

A) it shows that evaluators understand the program.
B) staff members insist on measuring such criteria.
C) success in achieving intermediate outcomes can often make up for failure to achieve outcome goals in summative evaluations.
D) this aids evaluators to learn whether program activities do indeed contribute to changes in the participants even when the achievement of the ultimate outcomes is minimal.

Question 6

Evaluators can increase the practical value of their evaluations by

A) gathering information on criteria that are expected to be unaffected by the program.
B) keeping a narrow, precise focus for the evaluations.
C) avoiding all contact with the program staff.
D) including observations of criteria which must apply to all programs, such as avoiding discriminatory behavior and treating clients with respect.

Question 7

Social values

A) must be ignored by all evaluators.
B) do not enter into program planning.
C) are not held in common by members of a pluralistic society so they do not enter into the choice of possible services that could be funded publicly.
D) are central in the choice and implementation of programs; however, empirical data cannot dictate which values are to be adopted.

Question 8

An examination of the needs of program clients would be important in an evaluation of outcome because

A) a mismatch between the needs of clients and the services provided could be responsible for disappointing outcomes.
B) a target population may have additional needs that the program was not designed to meet.
C) stakeholders disagree on the needs that are most pressing.
D) the staff members may be better trained to meet needs different from those the program was funded to deal with.

Question 9

Evaluators who learn that a treatment group shows a higher average level on measures of desired outcomes than a comparison group at a p < .05 level have

A) completed the work necessary to show that the program is worthwhile and should be maintained.
B) made a serious error since statistical significance tests are never used in program evaluations.
C) not completed their work since field settings require an understanding of the extent of improvement.
D) focused on a criterion that is value-oriented and, consequently, inappropriate for program evaluation.

Question 10

In setting a specific mean level expected for a criterion of success among a treatment group in an outcome evaluation, the evaluation team

A) depends solely on the program staff since the staff has the best understanding of the program.
B) works with the staff, but recognizes that program staff members often overestimate the degree clients will change even in an excellent program.
C) should ignore the program staff members since their level of self-interest will so greatly affect what they suggest that their input is worse than useless.
D) should use objective information available in the research literature. If such information is not available, the team should decline to conduct an evaluation.

Question 11

Evaluators discuss the conceptual, or theoretical, basis for the program because

A) theory is more interesting and complex than the service program being evaluated.
B) most program staff members and agency administrators have thoroughly considered the conceptual basis of their service and are eager to have it used in an evaluation.
C) they have been trained to do so, but such discussions add little to the quality of the program.
D) program theory helps evaluators to learn what might be important to observe and measure.

Question 12

Theories that are relevant to the development and operation of a service program

A) are the concern of program developers, not evaluators.
B) add little to carrying out an effective program evaluation.
C) can assist evaluators to carry out more useful evaluations.
D) are of no practical use, but can help get an evaluation published in a scientific journal.

Question 13

The degree to which a program is accepted and valued by potential users

A) is a proper concern to program evaluators and stakeholders.
B) is irrelevant since people seldom like what is good for them.
C) should form the primary focus of an evaluation to the exclusion of other criteria especially when evaluation funds are limited.
D) is the best guide to whether a program would be effective.

Question 14

Concern over whether the program funds are spent on the activities described in the approved program plan

A) is part of evaluations of program implementation.
B) indicates obsessive-compulsive thinking since the target population benefits regardless of what services are provided.
C) should concern accountants; program evaluators focus on softer criteria.
D) would not be of interest to governmental bodies responsible for supporting the program.

Question 15

The objectivity of program evaluators is most likely to be jeopardized by

A) meeting program staff in program settings.
B) seeking information about similar programs.
C) depending on only one stakeholder group for information on the program.
D) getting input from representatives of all major stakeholder groups.

Question 16

Evaluation criteria should be chosen

A) to ensure the evaluation has the most scientific credibility regardless of how long it would take to complete.
B) to reflect the practical limitations of budget and time available for an evaluation.
C) by external evaluators with excellent reputations.
D) by those who are responsible for funding the evaluation.

Question 17

When agreement has been reached regarding the criteria that are to be used to indicate program success,

A) it is helpful that some means be used to document that agreement has been reached.
B) evaluators know that work will progress smoothly the rest of the way.
C) stakeholders seldom question the choice of criteria at a later time, even if the evaluation results are unfavorable.
D) it is never appropriate to add or subtract any criteria during the evaluation.

Question 18

The selection of criteria and standards by which to judge the effectiveness of a program

A) is not all that important since good programs will look good and ineffective ones will look bad regardless of the criteria.
B) can be put off until after the data are being analyzed as long as many different criteria have been measured during data gathering.
C) is easy to carry out since all except the most disorganized program already have their criteria and standards clearly described in program materials.
D) can determine whether the evaluation turns out to be favorable or an unfavorable.

Question 19

An evaluation of a reading program that focuses on the reading achievement of fourth graders (in units of grade level equivalents) is using an

A) implementation goal.
B) intermediate goal.
C) outcome goal.
D) efficiency goal.

Question 20

Outcome measures with low reliability (for example, .40),

A) can never be used by program evaluators.
B) may be used as indicators of program success, but not as measures of individual success.
C) are probably not especially central to the goals of the program.
D) may be used as criteria, but only for the evaluators' purposes; they should not be mentioned in reports.

Question 21

Illustrate how an ill-considered choice of evaluation criteria can distort program activities to the extent that the value of a program is reduced. You might think of a school lunch program, or a youth sports program.

Question 22

Illustrate how an evaluator might mistakenly use criteria that can be measured with high reliability but that does not represent the program's purpose very well.

Question 23

Why are statistically significant differences between treatment and non-treatment groups not sufficient to show that a program is worth offering?

Question 24

Illustrate how a program could produce favorable outcomes yet be seen as unfair to some stakeholders.

Question 25

Why are negative side effects often ignored in program evaluations? Imagine the side effects of introducing a modern medical clinic into a rural village in an undeveloped nation.