Deck 11: Section 1: Evaluation Research and Problem Analysis Glossary

Question 1

Which of the following is not a variable-oriented strategy for research?

A) realist approach
B) experimental approach
C) case-study approach
D) all of these

Answer 1

B

Question 2

Problem analysis takes place in the earlier stages of the policy process. In contrast, program evaluation studies are conducted ____.

A) in the interim stages
B) In the later stages
C) In the preliminary stages
D) In the middle

Answer 2

B

Question 3

Like survey designs and field studies, evaluation research refers to a research method, not a specific research purpose.

Answer 3

False

Question 4

The policy process, like the research process in general, is fluid and does not always "start at the beginning and conclude at the end."

Question 5

Which action does not represent a step in the policy analysis process?

A) there is a demand for some new course of action or strong opposition to an existing policy
B) formulate a hypothesis that will allow for exploration
C) identify the ultimate goals and the different actions for achieving those goals
D) consider the impact of the policy outputs

Question 6

Outputs refer to the basic question of what a policy action is trying to achieve.

Question 7

Program evaluation is designed to link the intended actions and goals of criminal justice policy to ____.

A) empirical evidence that supports them having the desired effects
B) policies that are made to benefit communities
C) implementation of policies
D) randomized field experiments

Question 8

Goal-oriented public policies cannot be viewed as "if-then" statements: if some policy action is taken, then we expect some result to be produced.

Question 9

Quasi-experiments differ from true experiments in that they lack random assignment to control and experimental groups.

Question 10

In evaluation research, outcome measures serve as the independent variables.

Question 11

Not only is it necessary to measure dependent variables in impact assessment, the researcher also needs to measure the context within which the program is conducted.

Question 12

Policy analysis focuses upon whether the intended result was produced.

Question 13

One of the most important parts of conducting evaluation research is _____.

A) selecting a sample that will support the program goals
B) formulating goal statements that are empirically testable
C) gathering official documents since that is where the researcher can find complete descriptions of goals
D) making sure that the staff of the agency where you conduct research like you by doing anything to get them on your side

Question 14

Which step is of least concern in the evaluability assessment?

A) determine the degree of support for your project
B) determine what the general program goals are as well as the specific objectives
C) determine how the goals are translated into the program components
D) determine whether people in the organization have some sort of direct or indirect stake in the program

Question 15

Which of the following does not represent potential problems in doing evaluation research?

A) conflicting goals
B) vague goals
C) when the intended actions and goals are linked through empirical evidence.
D) difficulty in expressing policy goals in the "if-then" form that can be empirically tested

Question 16

Which statement is false with regard to the use of evaluation research?

A) it is gaining in popularity among researchers that really want to make a difference
B) federal requirements for program evaluations often accompany the implementation of new programs
C) funding is available for program evaluations
D) Evaluation research may only use experimental design.

Question 17

Evaluation apprehension occurs when officials fear that their own job performance is not being rated.

Question 18

Applied studies such as program evaluation is that they do not require the same degree of planning as do survey designs.

Question 19

All of the following would be examples of goal statements that must be clarified before empirical testing can follow except for ____.

A) participants will accept the philosophy of a drug-free life
B) participants will be equipped with the life skills necessary to succeed
C) participants will be able to obtain their GED upon completion of the program
D) provides participants with problem-solving skills

Question 20

Which statement is inaccurate?

A) impact assessments evaluate whether or not specified goals are attained
B) process evaluations focus on whether programs are carried out according to plans
C) problem analysis usually considers a limited range of alternative choices
D) ideally, impact assessments and process evaluations are conducted separately

Question 21

Which of the following is not accurate with respect to ethics and the political aspects of applied research?

A) There is no set of political norms that are agreed upon by all criminal justice researchers.
B) Just as there are codes for ethical conduct for researchers, there are formal codes of accepted political conduct.
C) Ethics of criminal justice research deals more with the methods used while political issues are more concerned with the substance and use of the research.
D) There is no formal code of accepted political conduct.

Question 22

Randomization may not be appropriate for assignment of people to treatment or programs for all of the following reasons except ____.

A) practical reasons
B) agency support
C) ethical reasons
D) legal reasons

Question 23

An) _______________ is an action taken for the purpose of producing some intended result.

Question 24

If the number of subjects in a program is small, statistical tests can detect only very large program effects of differences in outcome measures between the two groups. This illustrates the problem of ____.

A) generalizability
B) statistical conclusion validity
C) maturation
D) reliability

Question 25

The term _______________ refer to the basic question of what a policy action is trying to achieve.

Question 26

What can be said about the relationship between process evaluations and impact assessments?

A) process evaluations can be used in interpreting results from an impact assessment
B) process evaluations monitor program implementation but impact assessments do not answer questions about a program's effects
C) process evaluation cannot be used to interpret outcomes
D) there is no relationship between impact assessments and process evaluations

Question 27

Policy analysis is used to ____.

A) help design alternative courses of action
B) select a policy
C) seek explanations
D) select right employees

Question 28

In the home detention studies used in your text as examples of randomized studies, the authors concluded that ____.

A) randomization allows generalization
B) randomization allows for reliability
C) randomization does not control for variation in treatment integrity and program delivery
D) randomization does control for variation in treatment integrity and program delivery

Question 29

Which of the following is not correct with respect to process evaluations?

A) information about program implementation can be linked to outcome measures, even when accompanied by a process evaluation
B) process evaluation aids in interpreting results from impact assessments
C) process evaluations are useful when a researcher is interested in the performance of specific tasks within a program
D) process evaluations assume that tasks within a program are linked to program outcomes

Question 30

If a randomized experiment is used for evaluation purposes, ____.

A) case flow must be adequate to produce enough subjects in both the experimental and the control groups
B) program staff may minimize the exceptions to randomization
C) special interventions must be consistently applied
D) maintaining the desired level of control over experimental conditions

Question 31

Program evaluation differs from policy analysis with respect to ____.

A) when it occurs in the policy process
B) the cost
C) the time
D) the role of the stakeholder

Question 32

Problem solving is a fundamental tool in problem-oriented policing and involves which step?

A) Carefully defined specific problems.
B) Conduct in-depth analysis to understand their causes.
C) Undertake broad searcher for solutions to remove these causes and bring about lasting reductions in problems.
D) All of these answers are correct.

Question 33

Relating to randomization, which statement is inaccurate?

A) as the number of exceptions to random assignment increases, the statistical equivalence of the experimental and control groups decreases
B) when exceptions to random assignment are made, no bias enters the selection process
C) randomization in experiments for evaluation purposes are best suited for programs where exceptions are at a minimum
D) Random assignment of people to receive some especially desirable or punitive treatment may not be possible for legal, ethical, and practical reasons.

Question 34

With regard to evaluation designs, which of the following statements is not accurate?

A) variation in the levels of treatment delivered by a program can be a major threat to the validity of even randomized evaluation studies
B) uncontrolled variation in treatment is equivalent to unreliable measurement of the independent variable
C) controlled variation in treatment represents a threat to generalizability
D) varying degrees of participation by clients in a program will impact the outcome

Question 35

Treatment integrity is ____.

A) roughly equivalent to measurement reliability
B) roughly equivalent to measurement validity
C) the difference between the experimental intervention is delivered as intended
D) both reliable and valid

Question 36

Scientific realism is rooted in the principle that:

A) similar interventions can naturally be expected to have different outcomes in different contexts
B) the success of evaluations depends on the assignment of subjects to experimental and control groups
C) there be equivalence between experimental and control groups before an intervention
D) treatment integrity must be rigorously guarded

Question 37

Policy _______________ refer to what is actually produced.

Question 38

Program goals represent ____.

A) empirical indicators
B) desired outcomes
C) inputs
D) standards

Question 39

Policy analysis is ____.

A) infeasible courses of action
B) the generation of information and collection of evidence of the benefits
C) the consequences that would follow the implementation of various courses of action
D) fluid and does not always "start at the beginning and conclude at the end."

Question 40

Dr. Smith is interested in implementing a program that is designed to reduce the recidivism rates of property offenders. He selects two state prisons and, with the blessing of the Department of Corrections, begins the programs in the main state prison for males and a female institution that is nearby. Which design would be best for Dr. Smith to use?

A) non-equivalent groups
B) randomized study
C) interrupted times-series
D) none of these would work

Question 41

_______________ are those that have a vested interest in the organization and the outcome of evaluation research.

Question 42

In evaluation studies, the group to whom the program is appropriate is referred to as the program's _______________ population.

Question 43

You are a member of student government. Your group has been asked by the President of the University to conduct a study to determine if the newly implemented anti-drinking program all new freshmen are required to take is effective in reducing underage drinking on campus. Identify the goals and outcomes of the program. What type of design would you suggest for such a study and why?
:

Question 44

A researcher should complete an) _______________ that reflects their own understanding of program goals, elements, and operations of the organization under study.

Question 45

What does "treatment integrity" mean? Why is this an important issue in evaluation studies?

Question 46

A form of "pre-evaluation" called an) _______________ is designed to allow the researcher to determine whether conditions necessary for conducting an evaluation are present.

Question 47

Problem analysis takes place in the policymaking stage while ________ studies are conducted in later stages.

Question 48

Your criminal justice department has been requiring graduating seniors to take an exit exam as a measure of departmental outcomes. How could that tool be used to determine if the criminal justice major is effective?

Question 49

What are the strengths and weaknesses of quasi-experimental designs? Explain how the use of quasi-experimental designs may be more useful than classical experiments for conducting criminal justice research.

Question 50

Identify the differences between randomized evaluation designs and quasi-experimental designs. Give an example of the appropriate use of each.