Deck 10: Evaluation Research and Problem Analysis

Question 1

Which of the following does not represent steps in the policy analysis process?

A)there is a demand for some new course of action or strong opposition to an existing policy
B)formulate a hypothesis that will allow for exploration
C)identify the ultimate goals and the different actions for achieving those goals
D)consider the impact of the policy outputs

Answer 1

B

Question 2

Which of the following does not represent potential problems in doing evaluation research?

A)conflicting goals
B)vague goals
C)when the intended actions and goals are linked through empirical evidence.
D)difficulty in expressing policy goals in the "if-then" form that can be empirically tested

Answer 2

C

Question 3

Outcome measures in evaluation research serve as the independent variables in the research question.

Answer 3

False

Question 4

The good thing about applied studies such as program evaluation is that they do not require the same degree of planning as do survey designs.

Question 5

Not only is it necessary to measure dependent variables in impact assessment, the researcher also needs to measure the context within which the program is conducted.

Question 6

Which of the following is not something of concern in the evaluability assessment?

A)determine the degree of support for your project
B)determine what the general program goals are as well as the specific objectives
C)determine how the goals are translated into the program components
D)do people in the organization have some sort of direct or indirect stake in the program

Question 7

Which of the following is inaccurate concerning the use of evaluation research?

A)it is gaining in popularity among researchers that really want to make a difference
B)federal requirements for program evaluations often accompany the implementation of new programs
C)funding is available for program evaluations
D)Evaluation research may only use experimental design.

Question 8

The policy process, like the research process in general, is fluid and does not always "start at the beginning and conclude at the end."

Question 9

Quasi-experiments differ from true experiments in that they lack random assignment to control and experimental groups.

Question 10

One of the most important parts of conducting evaluation research is _____.

A)selecting a sample that will support the program goals
B)formulating goal statements that are empirically testable
C)gather official documents since that is where the researcher can find complete descriptions of goals
D)make sure that the staff of the agency where you conduct research like you by doing anything to get them on your side

Question 11

Which of the following statements is inaccurate?

A)impact assessments evaluate whether or not specified goals are attained
B)process evaluations focus on whether programs are carried out according to plans
C)problem analysis usually considers a limited range of alternative choices
D)ideally impact assessments and process evaluations are conducted separately

Question 12

Evaluation apprehension occurs when officials fear that their own job performance is not being rated.

Question 13

Policy analysis focuses upon whether the intended result was produced.

Question 14

Goal-oriented public policies cannot be viewed as "if-then" statements: if some policy action is taken, then we expect some result to be produced.

Question 15

Program evaluation is designed to link the intended actions and goals of criminal justice policy to __________.

A)empirical evidence that supports them having the desired effects
B)policies that are made to benefit communities
C)implementation of policies
D)randomized field experiments

Question 16

Case flow represents the process through which subjects are accumulated in which two kinds of groups?

A)experimental and control
B)deductive and inductive
C)equivalent and nonequivalent
D)voluntary and involuntary

Question 17

All of the following would be examples of goal statements that must be clarified before empirical testing can follow except for _____.

A)participants will accept the philosophy of a drug-free life
B)participants will be equipped with the life skills necessary to succeed
C)participants will be able to obtain their GED upon completion of the program
D)provides participants with problem-solving skills

Question 18

Which of the following is not a variable-oriented strategy for research?

A)realist approach
B)experimental approach
C)case-study approach
D)all of these

Question 19

Like survey designs and field studies, evaluation research refers to a research method not a specific research purpose.

Question 20

Outputs refer to the basic question of what a policy action is trying to achieve while impacts refer to the means of achieving desired policy goals.

Question 21

Which of the following is not correct with respect to process evaluations?

A)information about program implementation can be linked to outcome measures, even when accompanied by a process evaluation
B)process evaluation aids in interpreting results from impact assessments
C)process evaluations are useful when a researcher is interested in the performance of specific tasks within a program
D)process evaluations assume that tasks within a program are linked to program outcomes

Question 22

If the number of subjects in a program is small, statistical tests can detect only very large program effects of differences in outcome measures between the two groups.This illustrates the problem of _____.

A)generalizability
B)statistical conclusion validity
C)maturation
D)reliability

Question 23

Program evaluation differs from policy analysis with respect to _____.

A)the time dimension within the policy process
B)the cost
C)the time
D)the role of the stakeholder

Question 24

Program goals represent _____.

A)empirical indicators
B)desired outcomes
C)inputs
D)standards

Question 25

In the home detention studies used in your text as examples of randomized studies, the authors concluded that _____.

A)randomization allows generalization
B)randomization allows for reliability
C)randomization does not control for variation in treatment integrity and program delivery
D)randomization does control for variation in treatment integrity and program delivery

Question 26

Policy analysis searches for _____.

A)infeasible courses of action
B)the generation of information and collection of evidence of the benefits
C)the consequences that would follow the implementation of various courses of action
D)fluidity and does not always "start at the beginning and conclude at the end."

Question 27

The relationship between policy analysis and program evaluation is best characterized as _____.

A)both relying upon clearly specified goals and objectives
B)program evaluation used to test the achievement of goals and objectives worked out through policy analysis
C)both a and b are correct
D)there is no relationship between policy analysis and program evaluation

Question 28

Any public policy discussion involves politics.Which of the following is not accurate with respect to ethics and the political aspects of applied research?

A)there is no set of political norms that are agreed upon by all criminal justice researchers
B)just as there are codes for ethical conduct for researchers, there are formal codes of accepted political conduct
C)ethics of criminal justice research deals more with the methods used while political issues are more concerned with the substance and use of the research
D)there is no formal code of accepted political conduct

Question 29

Relating to randomization, which of the following statements is inaccurate?

A)as the number of exceptions to random assignment increases, the statistical equivalence of the experimental and control groups decreases
B)when exceptions to random assignment are made, no bias enters the selection process
C)randomization in experiments for evaluation purposes are best suited for programs where exceptions are at a minimum
D)Random assignment of people to receive some especially desirable or punitive treatment may not be possible for legal, ethical, and practical reasons.

Question 30

Randomized experiments for evaluation purposes require which of the following?

A)case flow must be adequate to produce enough subjects in both the experimental and the control groups
B)program staff may minimize the exceptions to randomization
C)special interventions must be consistently applied
D)maintaining the desired level of control over experimental conditions

Question 31

With regard to evaluation designs, which of the following statements is not accurate?

A)variation in the levels of treatment delivered by a program can be a major threat to the validity of even randomized evaluation studies
B)uncontrolled variation in treatment is equivalent to unreliable measurement of the independent variable
C)controlled variation in treatment represents a threat to generalizability
D)varying degrees of participation by clients in a program will impact the outcome

Question 32

Dr.Smith is interested in implementing a program that is designed to reduce the recidivism rates of property offenders.He selects two state prisons and, with the blessing of the Department of Corrections, begins the programs in the main state prison for males and a female institution that is nearby.Which design would be best for Dr.Smith to use?

A)non-equivalent groups
B)randomized study
C)interrupted times-series
D)none of these would work

Question 33

Problem analysis, coupled with scientific realism, helps public officials use:

A)research to select different methods
B)research to measure the effects of policies
C)research to determine which program to start
D)research to select and assess alternative courses of action

Question 34

Randomization may not be appropriate for assignment of people to treatment or programs for all of the following reasons except _____.

A)practical reasons
B)agency support
C)ethical reasons
D)legal reasons

Question 35

Problem solving is a fundamental tool in problem-oriented policing and involves which one of the following analytic steps?

A)Carefully defined specific problems.
B)Conduct in-depth analysis to understand their causes.
C)Undertake broad searcher for solutions to remove these causes and bring about lasting reductions in problems.
D)All of the above

Question 36

Policy analysis is used to _____.

A)help design alternative courses of action
B)select a policy
C)seek explanations
D)select right employees

Question 37

Treatment integrity is _____.

A)roughly equivalent to reliability
B)roughly equivalent to validity
C)the difference between the experimental intervention is delivered as intended
D)both reliable and valid

Question 38

Scientific realism is rooted in the principle that:

A)similar interventions can naturally be expected to have different outcomes in different contexts
B)the success of evaluations depends on the assignment of subjects to experimental and control groups
C)there be equivalence between experimental and control groups before an intervention
D)treatment integrity must be rigorously guarded

Question 39

What can be said about the relationship between process evaluations and impact assessments?

A)process evaluations can be used in interpreting results from an impact assessment
B)process evaluations monitor program implementation but impact assessments do not answer questions about a program's effects
C)process evaluation cannot be used to interpret outcomes
D)there is no relationship between impact assessments and process evaluations

Question 40

When dealing with stakeholders a researcher should be aware of all of the following except _____.

A)be able to identify all the stakeholders
B)get particularly friendly with the stakeholders so they will not interfere with your research
C)find out the stakeholders' perspectives on the program
D)find out the stakeholders' perspectives on the evaluation

Question 41

___________________ are those that have a vested interest in the organization and the outcome of evaluation research.

Question 42

In evaluation studies, the group to whom the program is appropriate is referred to as the program's _______________________ population.

Question 43

You are a member of student government.Your group has been asked by the President of the University to conduct a study to determine if the newly implemented anti-drinking program all new freshmen are required to take is effective in reducing underage drinking on campus.Identify the goals and outcomes of the program.What type of design would you suggest for such a study and why?

Question 44

What does "treatment integrity" mean? Why is this an important issue in evaluation studies?

Question 45

Problem analysis takes place in the policymaking stage while ________ studies are conducted in later stages.

Question 46

Learning policy goals is a key first step in doing ________________.

Question 47

The __________________ refer to the basic question of what a policy action is trying to achieve.

Question 48

Your criminal justice department has been requiring graduating seniors to take an exit exam as a measure of departmental outcomes.How could that tool be used to determine if the criminal justice major is effective?

Question 49

Identify the differences between randomized evaluation designs and quasi-experimental designs.Give an example of the appropriate use of each.

Question 50

What are the strengths and weaknesses of quasi-experimental designs? Explain how the use of quasi-experimental designs may be more useful than classical experiments for conducting criminal justice research.

Question 51

A(n)______________________ is an action taken for the purpose of producing some intended result.

Question 52

A researcher should complete a(n)____________________ that reflects their own understanding of program goals, elements, and operations of the organization under study.

Question 53

A form of "pre-evaluation" called a(n)_______________________ is designed to allow the researcher to determine whether conditions necessary for conducting an evaluation are present.

Question 54

____________________ is appropriate whenever some policy intervention occurs or is planned.

Question 55

Policy ___________ refer to what is actually produced.