Deck 4: Developing Measures of Implementation and Outcomes

Question 1

Desirable aspects of gathering data for program evaluation from records include the

A) freedom to choose the most relevant outcome variables.
B) uniformly high validity of archival data.
C) thoroughness with which program records are routinely kept in many settings.
D) non-reactivity of the data selected from program records.

Answer 1

D

Question 2

A major disadvantage of using program records in program evaluation is:

A) records were not designed for evaluation purposes.
B) measurement processes will not affect the program participants.
C) there can be no attrition of respondents who are willing to provide data.
D) another evaluator can examine the records independently from the original evaluator.

Answer 2

A

Question 3

Among the advantages of using program participants as a source of evaluation data is:

A) the information is relatively inexpensive.
B) measurement is not reactive.
C) their views about program quality are likely to be very valid.
D) participants provide particularly reliable reactions.

Answer 3

A

Question 4

When obtaining information from program participants, one should

A) never use objective variables.
B) depend on the program staff to choose the variables.
C) question whether participants are truly stakeholders.
D) ask only about issues familiar to the participants.

Question 5

Program participants are likely to be defensive about answering an evaluator's questions whenever

A) they feel that they have something to lose by providing their comments.
B) they feel that the program might be improved by their answers.
C) the evaluator contacts them in person.
D) they are particularly pleased with the program.

Question 6

One of the major problems with using program participants as sources of evaluation data is their tendency to

A) give primarily negative evaluations of service.
B) be uninterested in program evaluation.
C) make indiscriminately favorable evaluations of services.
D) be incapable of making any judgments about service providers.

Question 7

Asking individuals familiar with the kinds of services provided to make direct observations of the program

A) is seldom disruptive of program services.
B) could produce very unbiased data if the experts are involved in neither the program nor its competitors.
C) is among the least expensive evaluation data available.
D) should never be used because it could threaten the staff.

Question 8

Failing to detect a true program outcome because a variable is affected by so many influences is a particular problem with

A) community-level indexes such as crime rate or Gross Domestic Product.
B) program participant surveys of satisfaction with a program.
C) data gathered from the program staff.
D) questions addressed to relatives of the people being served.

Question 9

The choice of the criteria of effectiveness is very sensitive in program evaluation because

A) no one is ever satisfied with the criteria selected.
B) program staff are always trying to select criteria that are too easy to achieve.
C) the choice of criteria can determine the outcome of the evaluation.
D) most evaluations show that human services are highly effective.

Question 10

Selecting one particular variable to be the criterion of program effectiveness will

A) probably corrupt it.
B) complicate the data analysis.
C) reflect the complexity of the program.
D) be desired by most staff members.

Question 11

Evaluations are more likely to be effective if the list of variables to be measured is

A) lengthy and thorough.
B) focused on important variables.
C) includes some personality characteristics.
D) costly to obtain.

Question 12

The criteria of effectiveness used in a program evaluation should

A) be limited to the objectives of the program.
B) be determined by the program staff.
C) not focus on any negative outcomes of the services.
D) be related to the information needs of program stakeholders.

Question 13

Negative side-effects of programs are important for evaluators to consider because

A) staff members frequently do not consider them.
B) participants usually experience some strong negative side effects.
C) some participants will be harmed by any program.
D) program plans usually mention possible negative side effects.

Question 14

The criteria selected for measuring program effectiveness should be sensitive to changes caused by the program. A good way to determine if a variable might be sensitive to change is to

A) ask program participants for their opinions.
B) depend on program staff for ideas.
C) see if the variable distinguishes between groups of people who differ in degree to which they achieved planned program outcomes.
D) use well-constructed, standard personality or intelligence tests.

Question 15

In evaluating the reliability of a measure to be used in program evaluation, evaluators

A) recognize that means of groups are more stable than the scores of individual assessments.
B) realize that reliability is not very important in the area of program evaluation.
C) must choose between having a valid measure or having a reliable measure; they can't have both.
D) know that surveys are probably more reliable than program effectiveness indexes based on program records.

Question 16

The most widely used type of measurement procedure in program evaluation is most likely

A) a community-level index.
B) the written survey completed by program participants.
C) a rating of the program by a significant other.
D) an observation made by an expert working with the evaluator.

Question 17

The relative costs of using various methods of data collection in program evaluations

A) cannot be considered because truth is more important than cost.
B) are always a trivial factor in the whole study.
C) seldom exceed the cost to duplicate surveys or forms.
D) should be weighed against the validity of the data sought.

Question 18

Because behavioral change is usually the goal of most human service programs, behavioral observations

A) should be the sole type of measure used in evaluations.
B) may produce very valid information.
C) are relatively cheap to gather.
D) might be valuable, but cannot be made ethically.

Question 19

In selecting indexes or variables that are to reflect program success, it is useful to choose variables that are

A) historical, instrumental, and well-known.
B) important, sensitive to change, and cost-effective.
C) summative, formative, and sensitive.
D) hidden, covert, and implicit.

Question 20

Interviewing is usually done when

A) busy people have information about the program that few others have.
B) there is not enough time to prepare a reliable survey.
C) the costs of the evaluation must be kept to a bare minimum.
D) many people have similar experiences with the program, making surveys redundant.

Question 21

Including observations of actual behavior among the measures of program success

A) makes the evaluation too expensive.
B) creates incredible anxiety among the program staff members.
C) adds markedly to the credibility of the evaluation.
D) is usually felt to be unnecessary or unethical.

Question 22

If the choice is available, evaluators will choose to

A) develop new measures rather than to use existing measures.
B) use measures more similar to psychological tests than to classroom tests.
C) measure stable traits rather than changeable states.
D) use previously developed measures rather than to develop measures of their own.

Question 23

An important characteristic of self-report surveys is that

A) they can be constructed in a short time.
B) they can be used in any program setting.
C) considerable thought needs to be devoted to designing them.
D) any criterion of program effectiveness can be measured in a survey.

Question 24

What should you say if the following item were suggested for use in an evaluation of teaching: "My teacher was well-informed and well-meaning"?

A) The item mentions two aspects of a teacher.
B) It is best to avoid the use of a personal pronoun.
C) Teaching cannot be evaluated.
D) This is a good item; let's use it.

Question 25

Using multiple measures of program achievement may not add to the validity of an evaluation if

A) the measures share the same biases.
B) all measures are based on interviews.
C) few measures have reliabilities exceeding .80.
D) objective data sources such as expert observers or artifacts were not used.

Question 26

When using official records gathered over a period of years, evaluators must be wary of

A) the inter-rater reliability of records.
B) program participants who refuse to cooperate with the evaluation.
C) threat to insecure program managers.
D) changes in the definitions of the variables.

Question 27

Contrast the strengths and weaknesses of self-report surveys with observations of behavior.

Question 28

Distinguish between the characteristics of tests used in classrooms versus tests to measure stable psychological characteristics.

Question 29

Define reliability for hypothetical stakeholders who have not studied social service research methods or assessment principles.

Question 30

Suggest some artifacts that might be used in measuring the success of a program to encourage a company's employees to quit smoking.