Deck 13: Evaluation Research and Problem Analysis

Question 1

Evaluation apprehension occurs when people fear that their own job performance is being rated.

Answer 1

True

Question 2

At what stage does problem analysis take place?

A) policy implementing
B) later study
C) program evaluation
D) policy making

Answer 2

policy making

Question 3

Which of the following represents potential problems in doing evaluation research?

A) conflicting goals
B) clearly stated goals
C) ability to express policy goals in the "if-then" form that can be empirically tested
D) all of the above

Answer 3

conflicting goals

Question 4

Stakeholders are persons and organizations with an indirect interest in a program.

Question 5

Quasi-experiments differ from true experiments in that they lack random assignment to control and experimental groups.

Question 6

Like survey designs and field studies, evaluation research refers to the research purpose not a specific research method.

Question 7

Which of the following is not something of concern in the evaluability assessment?

A) determine the degree of support for your project
B) determine what the general program goals are as well as the specific objectives
C) determine how the goals are translated into the program components
D) whether the people in the organization have some sort of direct or indirect stake in the program

Question 8

Which of the following is accurate concerning the use of evaluation research?

A) it is gaining in popularity among researchers but it will likely not make a difference
B) federal requirements for program evaluations often accompany the implementation of new programs
C) funding is usually not available for program evaluations
D) it is only of value to the agency you study

Question 9

This type of analysis can help public officials plan and select an alternative action:

A) evaluation analysis
B) evidence-based analysis
C) policy analysis
D) problem analysis

Question 10

Goal-oriented public policies can be viewed as "if-then" statements: if some policy action is taken, then we expect some result to be produced.

Question 11

The policy process, like the research process in general, is fluid and does not always "start at the beginning and conclude at the end."

Question 12

Which of the following would not be an example of problem analysis?

A) find the most cost effective manner to maintain prison capacity at established levels
B) determine the most efficient manner in assigning felony cases to circuit judges
C) determine the average sentence given for those convicted of armed robbery within the state of Georgia
D) what is the best manner to reduce the response time by police officers in calls for service

Question 13

Which of the following is a type of program evaluation?

A) exploratory studies
B) content assessment
C) impact assessment
D) content analysis

Question 14

Which of the following does not represent steps in the policy analysis process?

A) there is a demand for some new course of action or strong opposition to an existing policy
B) formulate a hypothesis that will allow for exploration
C) identify the ultimate goals and the different actions for achieving those goals
D) consider the impact of the policy outputs

Question 15

Evaluation research is designed to:

A) be cost effective
B) determine if a program is effective in its implementation
C) determine what the public policy should be
D) be published

Question 16

Which of the following statements is accurate concerning program evaluations?

A) program evaluation is designed to link the intended actions and goals of policy to empirical evidence that supports them having the desired effects
B) program evaluation is concerned only with whether policies are popular
C) program evaluation is concerned with whether policies are ethical
D) all of the above

Question 17

Not only is it necessary to measure dependent variables in impact assessment, the researcher also needs to measure the context within which the program is conducted.

Question 18

Problem analysis focuses upon whether the intended result was produced.

Question 19

The good thing about applied studies such as program evaluation is that they do not require the same degree of planning as do survey designs.

Question 20

"Outputs" refers to the basic question of what a policy action is trying to achieve while "impacts" refers to the means of achieving desired policy goals.

Question 21

In order to measure a program's success in meeting goals, the researcher should be able to do which of the following in advance of the inception of the research?

A) involve measurable performance dimensions
B) have noncontroversial definitions for all important terms
C) specify relevant standards or criteria
D) all of the above

Question 22

With regard to evaluation designs, which of the following statements is not accurate?

A) variation in the levels of treatment delivered by a program can be a major threat to the validity of even randomized evaluation studies
B) uncontrolled variation in treatment is equivalent to unreliable measurement of the independent variable
C) controlled variation in treatment represents a threat to generalizability
D) varying degrees of participation by clients in a program will impact the outcome

Question 23

Randomized experiments for evaluation purposes require which of the following?

A) case flow must be adequate to produce enough subjects in both the experimental and the control groups
B) program staff must accept random assignment and minimize the exceptions to randomization
C) experimental interventions must be consistently applied
D) all of the above

Question 24

Which of the following is not correct with respect to process evaluations?

A) information about program implementation can be linked to outcome measures, even when accompanied by a process evaluation
B) process evaluation aids in interpreting results from impact assessments
C) process evaluations are useful when a researcher is interested in the performance of specific tasks within a program
D) process evaluations assume that tasks within a program are linked to program outcomes

Question 25

Which of the following is in contrast to incident-oriented policing?

A) problem-oriented policing
B) reactive-oriented policing
C) call for service policing
D) report-driven policing

Question 26

Relating to randomization, which of the following statements is not accurate?

A) as the number of exceptions to random assignment increases, the statistical equivalence of the experimental and control groups decreases
B) when exceptions to random assignment are made, bias enters the selection process
C) randomization in experiments for evaluation purposes are best suited for programs where exceptions are at a minimum
D) randomization in experiments for evaluation purposes are best suited for programs where there are many exceptions

Question 27

The relationship between policy analysis and program evaluation is best characterized as:

A) both rely upon clearly identified goals and objectives
B) program evaluation is used to test the achievement of goals and objectives worked out through policy analysis
C) program evaluation must precede policy analysis
D) there is no relationship between policy analysis and program evaluation

Question 28

Policy analysis searches for:

A) actions that will be accepted by funding agencies
B) operational definitions of policy terms
C) the consequences that would follow the implementation of various courses of action
D) whether the policy should be implemented

Question 29

What can be said about the relationship between process evaluations and impact assessments?

A) process evaluations can be used in interpreting results from an impact assessment
B) process evaluations monitor program implementation and questions about a program's effects
C) process evaluations are a method of assessing the accuracy of the data collected
D) there is no relationship between impact assessments and process evaluations

Question 30

Which of the following would a researcher completing an impact assessment be concerned with?

A) understanding how participants are selected
B) measuring the dependent variable
C) measuring the success of the program
D) measuring the independent variable

Question 31

In the home detention studies used in your text as examples of randomized studies, the authors concluded that:

A) randomization allows generalization
B) randomization allows for reliability
C) randomization does not control for variation in treatment integrity and program delivery
D) randomization does control for variation in treatment integrity and program delivery

Question 32

If the number of subjects in a program is small, statistical tests can detect only very large program effects of differences in outcome measures between the two groups. This illustrates the problem of:

A) generalizability
B) statistical conclusion validity
C) maturation
D) reliability

Question 33

All of the following would be examples of goal statements that must be clarified before empirical testing can follow except for:

A) encouraging participants to accept the philosophy of a drug-free life
B) encouraging participants to be equipped with the life skills necessary to succeed
C) encouraging participants to obtain their GED upon completion of the program
D) providing participants with problem-solving skills

Question 34

Treatment integrity is roughly equivalent to:

A) reliability
B) validity
C) generalizability
D) the accuracy of the data

Question 35

Randomization may not be appropriate for assignment of people to treatment or programs for all of the following reasons except:

A) practical reasons
B) agency support
C) ethical reasons
D) legal reasons

Question 36

Program goals represent:

A) empirical indicators
B) desired outcomes
C) inputs
D) standards

Question 37

When a researcher prepares her understanding of a program's goals, elements, and operations, she is preparing a program:

A) evaluation
B) description
C) analysis
D) hypothesis

Question 38

Policy analysis is used to:

A) help gather data
B) select an alternative course of action
C) produce an explanation
D) all of the above

Question 39

When dealing with stakeholders a researcher should be aware of all of the following except:

A) be able to identify all the stakeholders
B) get particularly friendly with the stakeholders so they will not interfere with your research
C) find out the stakeholders' perspectives on the program
D) find out the stakeholders' perspectives on the evaluation

Question 40

Any public policy discussion involves politics. Which of the following is not accurate with respect to ethics and the political aspects of applied research?

A) there is no set of political norms that are agreed upon by all criminal justice researchers
B) just as there are codes for ethical conduct for researchers, there are formal codes of accepted political conduct
C) ethics of criminal justice research deals more with the methods used while political issues are more concerned with the substance and use of the research
D) there is no formal code of accepted political conduct

Question 41

Dr. Smith is interested in implementing a program that is designed to reduce the recidivism rates of property offenders. He selects two state prisons and with the blessing of the Department of Corrections begins the programs in the main state prison for males and a female institution that is nearby. Which design would be best for Dr. Smith to use?

A) nonequivalent groups
B) randomized study
C) interrupted times-series
D) none of these would work

Question 42

Your state prison system is determined to reduce the recidivism rates of adult offenders leaving the system. To accomplish this goal, a prerelease program is implemented. The director of the department of corrections suggested an ideal design involving:

A) randomization
B) interrupted time-series
C) nonequivalent groups
D) quasi-experimental design

Question 43

When evaluating full-coverage programs, it is typically best to use:

A) interrupted time-series
B) quasi-experimental design
C) ex post evaluation
D) randomized design

Question 44

In which of the following would a researcher be least likely to use randomized designs?

A) program evaluation studies
B) ex post evaluations
C) victim-centered studies
D) time-series designs

Question 45

Nesting a quasi-experimental design within a randomized experiment refers to:

A) a way to study specific parts of a town
B) a way of backing up a true experiment
C) a substitute for a classical experiment
D) a way to examine the impact of the experiment on the subjects

Question 46

Specifying program goals and objectives in operational terms produces:

A) empirical indicators of program outcomes
B) conceptual definitions
C) program outcomes
D) program integrity

Question 47

According to your text, "theory driven" evaluations that consider how each element of a program should affect its targets should:

A) develop a micro model of the program production process
B) develop a vision statement that will be later used by the program staff
C) develop rigid measure of validity
D) define all inputs and outcomes as early as possible

Question 48

Which of the following can color research in many ways?

A) evaluation
B) politics
C) results
D) analyses

Question 49

A scouting report is a product of:

A) a process evaluation
B) an outcome evaluation
C) an evaluability assessment
D) defining policy objectives

Question 50

Which of the following would not be an example of an evidence-based policy?

A) the use of CCTV in public entertainment districts to reduce property crime
B) acquaintance rape of college students
C) underage drinking
D) Compstat

Question 51

A community policing program began in an entertainment area in a large metropolitan area. City leaders are interested to see if the change in policing strategy has reduced the number of crimes and has asked faculty from a local university to complete a study. What type of evaluation would be best?

A) interrupted time-series
B) ex post evaluation
C) nonequivalent groups
D) class experiment

Question 52

Many law enforcement agencies are using a program to detect emerging patterns in crime, this program is:

A) problem-oriented policing
B) conceptualization
C) reactive analysis
D) computerized crime maps

Question 53

Process evaluations:

A) assess the extent to which policies achieve their intended goals
B) are essentially the same as impact assessments
C) focus on policy demands and agenda
D) assess the extent to which policies are being implemented as intended

Question 54

In its most elementary form, evaluation research is a process of:

A) evaluating whether a piece of research is valid
B) determining the reliability of research findings
C) determining whether the intended result of a given policy intervention was produced
D) evaluating whether a given piece of legislation is effective

Question 55

There are two jails in your city: one houses state detainees awaiting trial while the other houses federal prisoners awaiting trial. You have been asked to assess why there are differences in the institutional behavior between these two institutions. Which design would be appropriate?

A) nonequivalent groups
B) true experiment
C) classical experiment
D) interrupted time-series

Question 56

Which of the following embody the means to achieve desired policy goals?

A) outputs
B) inputs
C) impacts
D) achievements

Question 57

The relationship between process evaluations and impact assessments can be stated as:

A) impact assessments must precede process evaluations
B) impact assessments answer questions about program effects while process evaluations monitor a program's implementation
C) only process evaluations can be used to determine if a program is effective
D) all of the above

Question 58

Which of the following is not a concern when a researcher uses an interrupted time-series design?

A) reliability of experimental treatment
B) history
C) instrumentation
D) construct validity

Question 59

True or "classical" experiments differ from quasi-experiments with respect to the fact that

A) classical experiments use random assignment to control and experimental groups while quasi-experimental designs do not
B) classical experiments use experimental groups while quasi-experimental designs use experimental and control groups
C) quasi-experimental designs afford more control than do classical designs
D) only one independent variable can be used in a classical experiment, while there are no limits to the number of independent variables in a quasi-experimental design

Question 60

Scientific realism is rooted in the principle that:

A) similar interventions can naturally be expected to have different outcomes in different contexts
B) the success of evaluations depends on the assignment of subjects to experimental and control groups
C) there be equivalence between experimental and control groups before an intervention
D) treatment integrity must be rigorously guarded

Question 61

____________________ takes place in the policymaking stage while program evaluation studies are conducted in later stages.

Question 62

Which of the following would not be an example of a program's goal statement?

A) to encourage participants to take responsibility for their actions
B) to study why some offenders were allowed in the program and others were not
C) to provide a safer community through the use of neighborhood watch programs
D) to provide life skills to inmates to participate in the prerelease program

Question 63

Your text discusses a concept called problem guides. Which of the following would be an example of this concept?

A) the analysis of specific problems such as crimes against the elderly
B) the analysis of broad issues such as race and crime
C) the analysis of specific crime databases such as UCR
D) the analysis of criminal justice agencies

Question 64

In process evaluation, the focus is on:

A) the inputs to the program
B) the people who work in the process
C) the program outputs
D) the program process

Question 65

Which of the following best describes an example of the use of evidence-based policy?

A) agencies use evaluation to decide what evidence backs up current funding
B) justice agencies are linked to evidence used for planning and evaluation
C) agencies are linked to other agencies
D) all of the above

Question 66

Evaluation in the criminal justice system seeks to link what?

A) the intended actions and goals of the policy to evidence that the policies are carried out
B) the policies to the people who work within the agency
C) the people within the agency to their specific responsibilities
D) the written policies and the written goals

Question 67

____________________ are those that have a vested interest in the organization and the outcome of evaluation research.

Question 68

What is the relationship between desired outcomes and empirical indicators?

A) desired outcomes represent the program goals while empirical indicators determine whether the desired outcomes are achieved
B) desired outcomes are the empirical indicators
C) desired outcomes represent the results of a program while empirical indicators are used to identify the goals
D) desired outcomes are what an agency hopes to get while empirical indicators identify what must be done to get them

Question 69

Which is considered a sort of "preevaluation"?

A) process evaluation
B) outcome evaluation
C) evaluability assessment
D) impact assessment

Question 70

The ____________________ refer to the basic question of what a policy action is trying to achieve.

Question 71

A form of "pre-evaluation" called a(n) ____________________ assessment is designed to allow the researcher to determine whether conditions necessary for conducting an evaluation are present.

Question 72

____________________ research is appropriate whenever some policy intervention occurs or is planned.

Question 73

When doing ____________________ research, the first key step is to learn policy goals.

Question 74

What is the fundamental first step in conducting evaluation studies?

A) measuring program goals
B) obtaining evaluable statements
C) specifying outcomes
D) clearly specifying program goals

Question 75

It is often said that in evaluation research, the research questions may already be formulated. Which of the following is an example of that?

A) in problem analysis, usually there is a wide range of alternative choices
B) in process evaluation, the focus is on whether specified goals are attained
C) in impact assessments, the evaluation is focused on whether specified goals are attained
D) with the questions already formulated, it decreases the amount of involvement for the researcher

Question 76

A researcher should complete a program ____________________ that reflects their own understanding of program goals, elements, and operations of the organization under study.

Question 77

Policy ____________________ refer to what is actually produced.

Question 78

Which of the following is accurate with respect to the relationship between impacts and outputs?

A) outputs are policy actions while impacts are goals
B) impacts are related to what the policy goals attempt to achieve while outputs are related to the means of achieving the goals
C) impacts are the goals and outputs are what is achieved
D) outputs are policy goals while impacts are how the policies are applied

Question 79

A(n) ____________________ refers to researchers and program personnel together considering how each program element affects the targets.

Question 80

A policy ____________________ is an action taken for the purpose of producing some intended result.