Question 1

Describe the organizational conditions that favor conducting an experimental evaluation.

Accepted Answer

Several organizational conditions can favor conducting an experimental evaluation. First, having a supportive and open-minded leadership team is crucial. Leaders who value evidence-based decision making and are willing to invest resources in experimental evaluations create an environment where such evaluations can thrive.

Second, a culture of data-driven decision making is important. Organizations that prioritize collecting and analyzing data to inform their strategies are more likely to see the value in experimental evaluations and be willing to allocate resources to support them.

Third, having a dedicated research or evaluation team can greatly facilitate experimental evaluations. This team can be responsible for designing and implementing experiments, analyzing the results, and communicating findings to relevant stakeholders.

Additionally, having access to necessary resources such as funding, time, and expertise is essential for conducting experimental evaluations. Without these resources, it can be challenging to design and execute high-quality experiments.

Finally, clear communication and buy-in from all stakeholders is critical. When everyone in the organization understands the purpose and value of experimental evaluations, they are more likely to support and participate in the process. This can lead to more successful and impactful evaluations.

Question 2

A good approach to interpretation when a program fails to achieve its objectives is

Accepted Answer

A)  to hold tight to previously stated objectives and call a spade, a spade. 
B)  to recognize that the objectives may have been too ambitious and to focus on some phase of the program that worked. 
C)  to treat well-conceived programs as experiments and search for what can be learned to improve the next version of the program. 
D)  to point out that repeated failures probably indicates low staff motivation. 
A)  to hold tight to previously stated objectives and call a spade, a spade. 
B)  to recognize that the objectives may have been too ambitious and to focus on some phase of the program that worked. 
C)  to treat well-conceived programs as experiments and search for what can be learned to improve the next version of the program. 
D)  to point out that repeated failures probably indicates low staff motivation. C

Question 3

How might an evaluator seek to reduce attrition among the members of control groups?

Accepted Answer

An evaluator looking to reduce attrition among the members of control groups in a study or program evaluation can implement several strategies to maintain participant engagement and minimize dropout rates. Attrition can threaten the validity of a study by leading to non-random differences between those who remain and those who leave the study. Here are some strategies to reduce attrition:

1. **Effective Communication**: Maintain clear, consistent, and open lines of communication with participants. Explain the importance of their continued participation and how their involvement contributes to the study's goals.

2. **Incentives**: Offer incentives for participation, which could be monetary or non-monetary rewards such as gift cards, vouchers, or access to services. Incentives should be ethical and not coercive.

3. **Convenience**: Make participation as convenient as possible. This could involve flexible scheduling, multiple locations for participation, or providing online or mobile platforms for data collection.

4. **Engagement**: Foster a sense of community or engagement among participants. This could be through social events, newsletters, or online forums where participants can share experiences and feel part of a group.

5. **Feedback**: Provide regular feedback to participants about the progress of the study and how their data is contributing. This can help participants feel valued and see the importance of their continued involvement.

6. **Support**: Offer support for participants who might face barriers to continued participation. This could include providing childcare, transportation, or addressing language barriers.

7. **Minimal Burden**: Design the study to minimize the burden on participants. This includes reducing the frequency and length of assessments, streamlining procedures, and ensuring that participation does not become overly burdensome or intrusive.

8. **Follow-up**: Implement diligent follow-up procedures for participants who miss appointments or are at risk of dropping out. Personalized contact, such as phone calls or home visits, can be more effective than impersonal reminders.

9. **Ethical Treatment**: Ensure that participants are treated ethically and with respect. This includes protecting their privacy, ensuring confidentiality, and being sensitive to their needs and concerns.

10. **Understanding Reasons for Attrition**: Conduct exit interviews or surveys with participants who choose to leave the study to understand their reasons for attrition. This information can be used to make adjustments to retain remaining participants and inform future studies.

11. **Interim Analysis**: Regularly analyze data for patterns of attrition that may indicate particular issues with certain subgroups or aspects of the study, allowing for timely interventions.

By implementing these strategies, an evaluator can help ensure that control group members remain engaged and active throughout the study, thereby preserving the integrity of the research and the validity of the results.

Question 4

One can expect the most objections to a proposal for an experimental evaluation of a program when

Accepted Answer

A)  a new program is introduced. 
B)  resources are scarce. 
C)  there is controversy about the effectiveness of the program. 
D)  the program has been in place and accepted for years. 
A)  a new program is introduced. 
B)  resources are scarce. 
C)  there is controversy about the effectiveness of the program. 
D)  the program has been in place and accepted for years.

Question 5

An evaluation that _____________ is a waste of resources.

Accepted Answer

A)  does not focus on causes 
B)  cannot be interpreted 
C)  uses correlation coefficients 
D)  does not support the program 
A)  does not focus on causes 
B)  cannot be interpreted 
C)  uses correlation coefficients 
D)  does not support the program

Question 6

Interpretations of outcomes of experimental designs are much more useful when

Accepted Answer

A)  stakeholders focus on statistical significance. 
B)  t tests are used. 
C)  the amount of change a program could cause has been discussed before examining the findings. 
D)  correlations are calculated for program and control groups separately. 
A)  stakeholders focus on statistical significance. 
B)  t tests are used. 
C)  the amount of change a program could cause has been discussed before examining the findings. 
D)  correlations are calculated for program and control groups separately.

Question 7

When determining the costs of a valid, experimental evaluation of a new medical treatment, innovators often overlook the costs of

Accepted Answer

A)  making a Type II error. 
B)  data collection. 
C)  monitoring the program and control groups participants. 
D)  not setting the probability of a Type I error accurately. 
A)  making a Type II error. 
B)  data collection. 
C)  monitoring the program and control groups participants. 
D)  not setting the probability of a Type I error accurately.

Question 8

One objection made about true experiments that is not valid is that

Accepted Answer

A)  the control group may miss out on a valuable program. 
B)  random assignment creates some artificiality in the program setting. 
C)  they are more expensive than pretest/posttest designs. 
D)  evaluators are only interested in data, not people's lives. 
A)  the control group may miss out on a valuable program. 
B)  random assignment creates some artificiality in the program setting. 
C)  they are more expensive than pretest/posttest designs. 
D)  evaluators are only interested in data, not people's lives.

Question 9

When an ambitious program evaluation of a public policy (such as welfare reform) fails to find that the new policy had a positive impact,

Accepted Answer

A)  the effort in implementing and evaluating the program was a waste of time and money. 
B)  one should assume that the evaluator did a poor job. 
C)  one can expect that the quality of the debate over the policy will be more enlightened. 
D)  the policy will be changed within months of the evaluation report. 
A)  the effort in implementing and evaluating the program was a waste of time and money. 
B)  one should assume that the evaluator did a poor job. 
C)  one can expect that the quality of the debate over the policy will be more enlightened. 
D)  the policy will be changed within months of the evaluation report.

Question 10

Service providers sometimes object to random assignment to groups because they try to match particular social services to particular individuals,

Accepted Answer

A)  however, it is impossible to match services to program participants. 
B)  however, there is considerable variation in the services provided to people with similar needs even in the absence of an evaluation. 
C)  therefore, evaluators often permit program staff to form the comparison groups. 
D)  moreover, services are usually planned to meet each participant's unique needs. 
A)  however, it is impossible to match services to program participants. 
B)  however, there is considerable variation in the services provided to people with similar needs even in the absence of an evaluation. 
C)  therefore, evaluators often permit program staff to form the comparison groups. 
D)  moreover, services are usually planned to meet each participant's unique needs.

Question 11

The use of pretests with an experimental evaluation

Accepted Answer

A)  permits one to control for random individual pretest differences, thereby lowering Type II error. 
B)  gives the appearance of credibility to the evaluation but is of no practical use. 
C)  is a violation of experimental design. 
D)  will not be understood by stakeholders. 
A)  permits one to control for random individual pretest differences, thereby lowering Type II error. 
B)  gives the appearance of credibility to the evaluation but is of no practical use. 
C)  is a violation of experimental design. 
D)  will not be understood by stakeholders.

Question 12

When control groups feel rivalry with the experimental group, it is possible that

Accepted Answer

A)  attrition will increase among the members of the experimental group. 
B)  an effective program will look less effective than it is. 
C)  an ineffective program will look more effective than it is. 
D)  the performance of the experimental group will improve. 
A)  attrition will increase among the members of the experimental group. 
B)  an effective program will look less effective than it is. 
C)  an ineffective program will look more effective than it is. 
D)  the performance of the experimental group will improve.

Question 13

One of the many differences between quasi-experiments and true experiments is that

Accepted Answer

A)  threats to external validity do not have to be considered in true experiments. 
B)  outcomes observed reflect the degree of experimental control. 
C)  a true experiment does not require a pretest. 
D)  results from quasi-experiments are usually more interpretable than results from true experiments. 
A)  threats to external validity do not have to be considered in true experiments. 
B)  outcomes observed reflect the degree of experimental control. 
C)  a true experiment does not require a pretest. 
D)  results from quasi-experiments are usually more interpretable than results from true experiments.

Question 14

If a control group experiences resentful demoralization because its members realize that they are not receiving desired program benefits, it is possible that

Accepted Answer

A)  attrition will increase among the members of the experimental group. 
B)  an effective program will look less effective than it is. 
C)  the program will look more effective than it is. 
D)  the performance of the control groups will improve. 
A)  attrition will increase among the members of the experimental group. 
B)  an effective program will look less effective than it is. 
C)  the program will look more effective than it is. 
D)  the performance of the control groups will improve.

Question 15

Attempts to compensate the control group tends to

Accepted Answer

A)  make an effective program look less effective than it is. 
B)  make an ineffective program look more effective than it is. 
C)  make a good program appear to have a negative impact. 
D)  nullify the effects of regression to the mean. 
A)  make an effective program look less effective than it is. 
B)  make an ineffective program look more effective than it is. 
C)  make a good program appear to have a negative impact. 
D)  nullify the effects of regression to the mean.

Question 16

A randomized experiment

Accepted Answer

A)  cannot be a good evaluation design. 
B)  relieves the evaluator of worrying about most of the threats to internal validity. 
C)  has high external validity. 
D)  lowers Type I error compared to a quasi-experimental design. 
A)  cannot be a good evaluation design. 
B)  relieves the evaluator of worrying about most of the threats to internal validity. 
C)  has high external validity. 
D)  lowers Type I error compared to a quasi-experimental design.

Question 17

Once stakeholders agree that random assignment to groups will be permitted, it is crucial that

Accepted Answer

A)  the evaluator periodically monitor the outcome variables. 
B)  stakeholders be permitted to make the actual group assignments. 
C)  different rates of attrition among groups not be permitted to introduce bias. 
D)  regression to the mean be carefully examined. 
A)  the evaluator periodically monitor the outcome variables. 
B)  stakeholders be permitted to make the actual group assignments. 
C)  different rates of attrition among groups not be permitted to introduce bias. 
D)  regression to the mean be carefully examined.

Question 18

Whenever preexisting, intact groups are used for an evaluation,

Accepted Answer

A)  one can treat the groups as if they were equivalent. 
B)  one can be sure that the participants were randomly assigned to the groups. 
C)  the evaluation is readily interpretable. 
D)  the groups usually differ on many variables. 
A)  one can treat the groups as if they were equivalent. 
B)  one can be sure that the participants were randomly assigned to the groups. 
C)  the evaluation is readily interpretable. 
D)  the groups usually differ on many variables.

Question 19

When services or medical treatments are offered before being carefully evaluated,

Accepted Answer

A)  little harm is done because if a treatment is ineffective, people will quickly notice and discard it. 
B)  much harm can be done because some ineffective treatments have come to be considered standard treatments. 
C)  theoretically some harm could be done, but people seldom implement a harmful program. 
D)  much harm will occur because nearly all non-evaluated programs are harmful. 
A)  little harm is done because if a treatment is ineffective, people will quickly notice and discard it. 
B)  much harm can be done because some ineffective treatments have come to be considered standard treatments. 
C)  theoretically some harm could be done, but people seldom implement a harmful program. 
D)  much harm will occur because nearly all non-evaluated programs are harmful.

Question 20

Usually a new program cannot be implemented in all parts of an organization simultaneously. This pattern

Accepted Answer

A)  is unfortunate for evaluation. 
B)  can be utilized by an evaluator to create control and experimental groups. 
C)  creates threats to internal validity that are impossible to control. 
D)  violates ethical standards since this is unfair to those who receive a program later. 
A)  is unfortunate for evaluation. 
B)  can be utilized by an evaluator to create control and experimental groups. 
C)  creates threats to internal validity that are impossible to control. 
D)  violates ethical standards since this is unfair to those who receive a program later.

Describe the organizational conditions that favor conducting an experimental evaluation.

A good approach to interpretation when a program fails to achieve its objectives is

How might an evaluator seek to reduce attrition among the members of control groups?

One can expect the most objections to a proposal for an experimental evaluation of a program when

An evaluation that _____________ is a waste of resources.

Interpretations of outcomes of experimental designs are much more useful when

When determining the costs of a valid, experimental evaluation of a new medical treatment, innovators often overlook the costs of

One objection made about true experiments that is not valid is that

When an ambitious program evaluation of a public policy (such as welfare reform) fails to find that the new policy had a positive impact,

Service providers sometimes object to random assignment to groups because they try to match particular social services to particular individuals,

The use of pretests with an experimental evaluation

When control groups feel rivalry with the experimental group, it is possible that

One of the many differences between quasi-experiments and true experiments is that

If a control group experiences resentful demoralization because its members realize that they are not receiving desired program benefits, it is possible that

Attempts to compensate the control group tends to

A randomized experiment

Once stakeholders agree that random assignment to groups will be permitted, it is crucial that

Whenever preexisting, intact groups are used for an evaluation,

When services or medical treatments are offered before being carefully evaluated,

Usually a new program cannot be implemented in all parts of an organization simultaneously. This pattern

Program Evaluation : an Overview

Planning an Evaluation

Developing and Using a Theory of the Program

Developing Measures of Implementation and Outcomes

Ethics in Program Evaluation

The Assessment of Need

Monitoring the Implementation and the Operation of Programs

Qualitative Evaluation Methods

Outcome Evaluations With One Group

Quasi-Experimental Approaches to Outcome Evaluation

Analyses of Costs and Outcomes

Evaluation Reports: Interpreting and Communicating Findings

How to Encourage Utilization

Filters

Exam 11: Using Experiments to Evaluate Programs

Describe the organizational conditions that favor conducting an experimental evaluation.

A good approach to interpretation when a program fails to achieve its objectives is

How might an evaluator seek to reduce attrition among the members of control groups?

One can expect the most objections to a proposal for an experimental evaluation of a program when

An evaluation that _____________ is a waste of resources.

Interpretations of outcomes of experimental designs are much more useful when

When determining the costs of a valid, experimental evaluation of a new medical treatment, innovators often overlook the costs of

One objection made about true experiments that is not valid is that

When an ambitious program evaluation of a public policy (such as welfare reform) fails to find that the new policy had a positive impact,

Service providers sometimes object to random assignment to groups because they try to match particular social services to particular individuals,

The use of pretests with an experimental evaluation

When control groups feel rivalry with the experimental group, it is possible that

One of the many differences between quasi-experiments and true experiments is that

If a control group experiences resentful demoralization because its members realize that they are not receiving desired program benefits, it is possible that

Attempts to compensate the control group tends to

A randomized experiment

Once stakeholders agree that random assignment to groups will be permitted, it is crucial that

Whenever preexisting, intact groups are used for an evaluation,

When services or medical treatments are offered before being carefully evaluated,

Usually a new program cannot be implemented in all parts of an organization simultaneously. This pattern

Program Evaluation : an Overview

Planning an Evaluation

Developing and Using a Theory of the Program

Developing Measures of Implementation and Outcomes

Ethics in Program Evaluation

The Assessment of Need

Monitoring the Implementation and the Operation of Programs

Qualitative Evaluation Methods

Outcome Evaluations With One Group

Quasi-Experimental Approaches to Outcome Evaluation

Analyses of Costs and Outcomes

Evaluation Reports: Interpreting and Communicating Findings

How to Encourage Utilization

Filters