Deck 8: Qualitative Evaluation Methods

Question 1

Qualitative evaluation procedures are particularly appropriate when

A) the stakeholders can define clear, objective outcome goals.
B) there are ample opportunities to randomly assign program participants to groups.
C) stakeholders cannot wait until the program is over to evaluate the program.
D) when there are many groups of participants getting identical services (e.g. medicines).

Answer 1

C

Question 2

Qualitative evaluation methods are especially sensitive to

A) the unique features of the organizational setting of the program to be evaluated.
B) the test/retest reliability of the measures used.
C) the advantages of matching groups using standardized tests.
D) statistical analyses using large samples.

Answer 2

A

Question 3

When a program has vague or diffuse goals, evaluators

A) may want to refuse to conduct an evaluation.
B) may decide to do a qualitative evaluation.
C) should seek to pin the staff down to something specific even if it doesn't reflect the richness of the program.
D) suspect that the staff members really don't know what they are trying to achieve.

Answer 3

B

Question 4

If the entity being evaluated is a single unit with complex, multifaceted goals,

A) it is hard to imagine how to do an experiment.
B) evaluators should insist that the program be simplified.
C) there is no valid way of conducting an evaluation.
D) a qualitative evaluation should be ruled out.

Question 5

When a carefully done evaluation takes a long time to complete,

A) stakeholders become more and more eager to get the results.
B) some stakeholders may no longer be interested in the issue.
C) it is probably best to refrain from giving progress reports because political forces will distort them.
D) it is probably correct to conclude that it was done poorly.

Question 6

A crucial difference between qualitative and quantitative evaluations is the central importance of _______ in qualitative evaluations.

A) complex statistical analyses
B) objective assessment
C) automated data collection
D) the observations of the data gatherer

Question 7

Qualitative evaluators believe that

A) the goals of programs must be specifically laid out when the evaluation is planned.
B) the evaluator should react to initial findings redirecting the focus of the evaluation as necessary.
C) all interpretations of subjectively evaluated data are equally plausible.
D) the evaluation design should never be altered regardless of what the initial data reveal.

Question 8

The text likens the understanding derived from qualitative evaluations to the kind of qualitative understanding developed in

A) mystery stories and court decisions.
B) descriptions of sports contests.
C) the results of political races.
D) biological research.

Question 9

When qualitative data gatherers are present in a program setting but do not have a role in the interaction, they are called ______ observers.

A) participant
B) pseudo-participant
C) non-participant
D) pseudo-staff members

Question 10

The settings in which non-participant observers are most likely to fit into easily are those that are

A) relatively private.
B) laden with conflict.
C) highly complex.
D) relatively public.

Question 11

The settings in which the use of participant observers is most likely to be necessary to make direct observations are those that are

A) relatively private.
B) do not involve a defensive program staff.
C) highly complex.
D) relatively public.

Question 12

Physical artifacts of programs are called

A) silent witnesses.
B) traces.
C) accretions.
D) deposits.

Question 13

When interviewing for a qualitative evaluation, it is best to ask questions that

A) demand a "yes/no" answer similar to questions in court cross-examinations.
B) can serve to make the best argument for the success of the program.
C) encourage the respondent to elaborate his/her own views about the topic of the question.
D) are readily coded numerically and subsequently analyzed statistically.

Question 14

The first phase of a qualitative evaluation requires that an evaluator

A) draw up mock tables to be used to present the data to be gathered.
B) develop a set of questions with a definite order.
C) share openly his/her interpretations with the program staff.
D) make unrestricted observations.

Question 15

Qualitative evaluators share their interpretations of program activities with the staff and other stakeholders as a

A) good public relations practice.
B) check on subjectivity and to correct misunderstandings.
C) way to verify that observers do have access to all program activities.
D) method of gaining insights into statistical analyses.

Question 16

Qualitative evaluators defend their style of conducting evaluations by pointing out that evaluators who use only quantitative methods

A) often are influenced by subjective expectations in the choice of issues to address and variables to measure.
B) seldom have complete control over threats to internal validity.
C) have never carried out a useful evaluation.
D) both a and b

Question 17

Reliability of the interpretation of qualitative observations can be examined by

A) calculating split-half reliability coefficients.
B) repeating a set of program observations.
C) having a trained person look through the evaluator's detailed field notes and draw independent interpretations.
D) replicating the program in other settings with different program staff members.

Question 18

Some proponents of qualitative methods argue that one of the purposes of evaluation is to empower people by *

A) being sure that the most disadvantaged groups in society participate in evaluations.
B) expanding basic research related to the type of services offered by the program being evaluated.
C) being sure the evaluations match the canons of sound research design.
D) being sure that the best program is available to people.

Question 19

Stakeholder concerns and the empowerment of less powerful stakeholders is central for the _________ evaluator.

A) quantitative
B) external
C) qualitative
D) internal

Question 20

Even the most seasoned evaluators favoring quantitative methods recognize that evaluation design cannot plan for

A) the successful achievement of program goals.
B) expected negative side-effects.
C) failure to meet program goals.
D) unexpected negative side effects.

Question 21

Good probes to use in a qualitative interview encourage an interviewee to

A) elaborate on the first answer using his/her own words.
B) defend the initial answers given.
C) provide specific information that an evaluator can use to support the conclusions that the evaluator wanted to draw in the first place.
D) weigh his/her words carefully just in case confidentiality is violated.

Question 22

The Success Case Method focuses on those participants who were most successful in order to

A) provide a favorable evaluation.
B) win over cynical stakeholders.
C) shame unsuccessful participants into better behaviors.
D) identify behaviors and program experiences that are associated with success.

Question 23

The Appreciative Inquiry strategy begins by learning what staff members believe

A) needs the most improvement.
B) are the most controversial aspects of a program.
C) are the directions that they most want the program to begin.
D) are the aspects of the program that work particularly well.

Question 24

Describe the characteristics of program settings in which qualitative evaluations are probably more appropriate than quantitative evaluations.

Question 25

Compare the potential for threat to program staff members from a qualitative evaluation versus program monitoring standpoint.

Question 26

Describe the advantages of combining features of qualitative evaluations and program monitoring in terms of the credibility and interpretability of the evaluation and its potential usefulness to stakeholders.

Question 27

Which steps (that is, need assessment, implementation, outcome, cost analyses) in a program evaluation are most appropriately examined in a qualitative fashion?

Question 28

Why do some evaluators criticize qualitative evaluation methods?