Deck 8: Experiments for Evaluation Research

Question 1

Program evaluation is research that ______.

A) determines the needs of a population for new programs
B) evaluates the effectiveness of existing programs or policies
C) evaluates whether a program needs to conduct research
D) evaluates the best choice for choosing a program for a particular population

Answer 1

evaluates the effectiveness of existing programs or policies

Question 2

Experiments are used to test ______.

A) Association
B) significance
C) validity
D) causality

Answer 2

causality

Question 3

Classical studies are typically done in laboratories. When we do evaluation research in real-life settings, what can happen?

A) Your results are invalid if all forms of bias were not able to be ruled out.
B) Your study could become nonreplicable, and thus meaningless.
C) You may not be able to conclude that your results were caused by the treatment and not by something else.
D) Your research may be deemed unethical if it affects anyone other than your subjects.

Answer 3

You may not be able to conclude that your results were caused by the treatment and not by something else.

Question 4

Classical experiments are done in laboratories because researchers can better control ______.

A) the treatment
B) the effects
C) outside influences
D) validity

Question 5

Because it is based on positivist methodology, experimental research must be ______.

A) valid
B) descriptive
C) generalizable
D) replicable

Question 6

Research questions for experimental evaluation research ______.

A) are usually qualitative
B) include at least three variables (independent, dependent, and treatment)
C) cannot imply causation
D) ask about causation

Question 7

Experimental researchers can usually control what thing that survey researchers can't?

A) temporal order
B) association
C) objectivity
D) unit of analysis

Question 8

In experimental evaluation research, the hypotheses usually all have ______.

A) the same independent variable
B) the same dependent variable
C) different independent and dependent variables
D) the same variable sometimes used as an independent variable and sometimes as a dependent variable

Question 9

Which is the most appropriate research question for experimental evaluation research?

A) What effects, if any, do illiteracy rates have on child poverty?
B) What effects does smoking have on teenagers' popularity?
C) What effects, if any, does age have on voting habits in local elections?
D) What effects, if any, does instituting flexible work schedules have on interpersonal dynamics within work groups?

Question 10

Which of the following is the best research question for experimental evaluation research?

A) What effects, if any, do gray winter skies have on depression?
B) What effects, if any, does the New Orleans Police Officer Diversity Training program have on police officers' assessment of the degree of danger they feel when apprehending male suspects of different races and ethnicities?
C) What effects, if any, would stricter penalties for driving under the influence of marijuana have on accident fatalities?
D) What effects, if any, does playing video games for 4 or more hours per day have on children's brain development?

Question 11

Which of the following is a POOR example of a hypothesis for experimental evaluation research?

A) Women who take the Assertiveness Training Workshop are more likely to state when they feel that their personal boundaries have been crossed than women who do not take the training.
B) New fathers who work for the State of New York and take the paid parental leave available to them when their child is born are more likely to report feeling productive in their jobs upon their return than those who do not take the paid leave available to them.
C) Those democrats who voted for Hillary Clinton in the 2016 primaries are more likely to vote for a woman in the 2020 primaries than are those who voted for Bernie Sanders.
D) Independent business owners who participate in the Work for Dignity program, in which they hire a homeless person to do 5 hours of work at their establishment per week paid for by a grant from the County, are more likely to volunteer their services or donate their products to the local homeless shelters than those who do not participate in the program.

Question 12

Which of the following is considered LEAST damaging to the validity of experimental design?

A) not using a control group
B) not using a pretest
C) not ruling out bias when assigning people to treatment and control groups
D) not using a posttest

Question 13

Which of the following would be an acceptable reason for not using probability sampling in assigning groups in experimental evaluation research?

A) The groups don't seem to be biased anyway, so it's not needed.
B) The study is easier to manage with fewer groups.
C) The groups are naturally occurring.
D) The number of subjects is too small.

Question 14

What is the problem with preexperimental designs?

A) They are very difficult to replicate.
B) You can't rule out plausible alternatives.
C) They are unreliable.
D) They lack objectivity.

Question 15

You are conducting research on the effects of a grief support group on adults' coping with the death of a loved one. You decide that you will measure ability to cope by observing behavior in the group, by asking the research subjects to keep a log rating their emotions at the end of each day, and by asking them survey items about coping strategies, activities engaged in, and other measures of coping. What component of research is this?

A) operationalizing
B) conceptualizing
C) preexperimental design
D) sampling

Question 16

You are evaluating the efficacy of a disaster preparation program in campus dorms focused on emergency evacuation. You decide to provide surveys before and after residents complete the program to gauge self-reported levels of disaster preparedness. What component of research is this?

A) conceptualizing
B) operationalizing
C) preexperimental design
D) sampling

Question 17

In drawing a sample for experiments for evaluation, which sampling method is most often used?

A) cluster
B) simple random
C) systematic
D) convenience

Question 18

You are conducting experimental research in order to evaluate the effectiveness of a new sexual harassment training program at a small firm. Which would be the most appropriate way to choose your subjects?

A) cluster sample
B) census
C) systematic sampling
D) convenience sample

Question 19

You have two groups in your experimental evaluation research: treatment and control. To each group you assign five males and five females of each of the following ethnicities: White, Black, Asian and Latino. This is called ______

A) random assignment
B) matching
C) stratifying
D) systematic sampling

Question 20

You have two groups in your experimental evaluation research: treatment and control. Each subject is given a number. You then use a random numbers generator to assign each person to a group. This is called ______

A) random assignment
B) matching
C) stratifying
D) systematic sampling

Question 21

In addition to matching, what is another method for minimizing bias when assigning subjects to experimental groups?

A) convenience sampling
B) theoretical sampling
C) probability sampling
D) cluster sampling

Question 22

If you use a convenience sample for participation in the experiment, and then probability sampling for group assignment, can you generalize your results to the larger population?

A) Yes, you would be able to generalize your results.
B) No, you would not be able to generalize your results.
C) only if you used simple random sampling for the group assignment
D) only if you conducted stratified sampling for the group assignment

Question 23

Under what circumstances do you need to debrief your research subjects?

A) when you use deception in the experiment
B) when they have been part of the treatment group
C) you always need to debrief subjects after an experiment
D) whenever you must provide full informed consent

Question 24

The program you are evaluating is time-consuming, and so people keep dropping out of your treatment group because they are overwhelmed by the time commitment. This is an example of ______.

A) Debriefing
B) contamination
C) placebo effect
D) attrition

Question 25

Even if you randomly assign subjects to groups, which of the following may result in you ending up with groups that are not comparable?

A) attrition
B) contamination
C) placebo effect
D) maturation

Question 26

You are evaluating the efficacy of a new safety training course at a shipping company. To ensure accurate results, it is vital that the control group does not have access to the information in the course; however, subjects from your control group and your treatment group frequently interact with one another and you are concerned that they may share this information. Which of the following are you most concerned with?

A) debriefing
B) contamination
C) placebo effect
D) attrition

Question 27

Internal validity means ______.

A) that maturation has occurred
B) that the results can be generalized to your population
C) that the effect you observe is actually caused by the treatment
D) that your hypotheses are supported

Question 28

If you use a control group, to which of the following threats to internal validity are you still most vulnerable?

A) your own expectations affecting the results
B) subjects changing over time
C) outside conditions changing over time
D) attrition rates

Question 29

Why would you choose to do experimental evaluation research? What is it good at understanding?

Question 30

Why can causality be tested in experimental research better than in survey research?

Question 31

What is quality control? List three ways that you could ensure consistent quality in your data collection.

Question 32

Write a research question for experimental evaluation research.

Question 33

Write four hypotheses for your research question.

Question 34

What is an executive summary, and why would you use it?

Question 35

You are conducting research on the effects of a grief support group on the coping abilities of adult children who have had a parent die. Write a research question for this research project.

Question 36

Write four hypotheses for this research question.

Question 37

Identify the independent and dependent variables in your hypotheses.

Question 38

Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?

Question 39

How will you draw a sample of participants for your research? Why?

Question 40

How will you assign subjects to research groups? Why? What are the pros and cons of this approach?

Question 41

Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?

Question 42

What steps will you need to take to protect your participants?

Question 43

Will informed consent be needed for your study? Why or why not?

Question 44

What other ethical issues might be likely to arise in conducting this research?

Question 45

What steps would you take to prepare for data collection?

Question 46

How will you ensure quality control in your data collection procedures?

Question 47

How would you try to diminish the attrition rate? How will it affect your results if the rate is high?

Question 48

What is contamination, and how would you try to minimize it?

Question 49

How would you clean your data?

Question 50

What are the threats to internal validity, and how would you try to minimize them?

Question 51

Evaluate the external validity of your study.

Question 52

You are conducting evaluation research for the county hospital by studying the effects of the elimination of double shifts and overtime on patient care. Write a research question for this research project.

Question 53

Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?


Question 54

How will you draw a sample of participants for your research? Why?

Question 55

How will you assign subjects to research groups? Why? What are the pros and cons of this approach?

Question 56

Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?

Question 57

What steps will you need to take to protect your participants?

Question 58

Will informed consent be needed for your study? Why or why not?

Question 59

What other ethical issues might be likely to arise in conducting this research?

Question 60

What steps would you take to prepare for data collection?

Question 61

How will you ensure quality control in your data collection procedures?

Question 62

How would you try to diminish the attrition rate? How will it affect your results if the rate is high?

Question 63

What is contamination, and how would you try to minimize it?

Question 64

How would you clean your data?

Question 65

What are the threats to internal validity, and how would you try to minimize them?

Question 66

Evaluate the external validity of your study.

Question 67

You are doing research evaluating the effectiveness of an anti-harassment campaign in your company. You will ask subjects to fill out a survey about their attitudes about and experiences with workplace harassment both before and after the campaign. Write a research question for this research project.

Question 68

Write four hypotheses for this research question.

Question 69

Identify the independent and dependent variables in your hypotheses.

Question 70

Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?

Question 71

How will you draw a sample of participants for your research? Why?

Question 72

How will you assign subjects to research groups? Why? What are the pros and cons of this approach?

Question 73

Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?

Question 74

What steps will you need to take to protect your participants?

Question 75

Will informed consent be needed for your study? Why or why not?

Question 76

What other ethical issues might be likely to arise in conducting this research?

Question 77

What steps would you take to prepare for data collection?

Question 78

How will you ensure quality control in your data collection procedures?

Question 79

How would you try to diminish the attrition rate? How will it affect your results if the rate is high?

Question 80

What is contamination, and how would you try to minimize it?