Deck 10: Quasi-Experimental Designs and Program Evaluation

Question 1

A researcher was interested in determining whether more frequent breaks (i.e., "coffee breaks") in a business setting would help employees to be more productive. With the cooperation of the management, employees on one floor of the corporate offices were allowed to take a 10-minute break each hour (at any time) between 8:00 and 11:00 A.M. (for a total of 30 minutes). The comparison group comprised employees on different floors who followed the usual corporate policy of taking a 30-minute break sometime during the morning (at any time). Measures of productivity were gathered for each employee according to his or her job (e.g., number of reports written, number of sales made, etc.). A time series analysis was applied to compare the productivity of both groups of employees for six months before and after the intervention (started in July). Quite surprisingly, the productivity of both groups increased following the onset of the intervention, suggesting to the researcher that the timing of breaks makes no difference.
a graph that displays results demonstrating the hypothesized effectiveness of the intervention.

Answer 1

(1)The graph should display increases in performance for both groups after the intervention,and little difference between groups. (2)The graph should show an increase in performance beginning in July for the group that received three 10-minute breaks,but no comparable increase for the comparison group (30-minute break).

Question 2

How does a time series design with nonequivalent control group differ from a nonequivalent control group design? What threat to internal validity is controlled by adding a nonequivalent control group to a simple interrupted time series design?

Answer 2

The time series design with nonequivalent control group differs from a nonequivalent control group design because in the time series design there are multiple observations before and after the time the treatment is administered.Many potential threats to internal validity due to history are controlled by adding a nonequivalent control group to a simple interrupted time series design.

Question 3

Distinguish between evaluation of outcome and evaluation of efficiency as these are used in program evaluation.

Answer 3

Evaluation of outcome uses experimental and quasi-experimental methods to determine whether a given program is effective in meeting its stated goals.Evaluation of the efficiency of a program assesses the cost of the program.The program's success (outcome evaluation)and its cost (efficiency evaluation)are used to make decisions among possible services that a government or other institution is capable of delivering.

Question 4

A researcher was interested in determining whether more frequent breaks (i.e., "coffee breaks") in a business setting would help employees to be more productive. With the cooperation of the management, employees on one floor of the corporate offices were allowed to take a 10-minute break each hour (at any time) between 8:00 and 11:00 A.M. (for a total of 30 minutes). The comparison group comprised employees on different floors who followed the usual corporate policy of taking a 30-minute break sometime during the morning (at any time). Measures of productivity were gathered for each employee according to his or her job (e.g., number of reports written, number of sales made, etc.). A time series analysis was applied to compare the productivity of both groups of employees for six months before and after the intervention (started in July). Quite surprisingly, the productivity of both groups increased following the onset of the intervention, suggesting to the researcher that the timing of breaks makes no difference.
What type of research design was used in this study?

Question 5

At a large university, a group of education specialists tested the effectiveness of a new academic improvement course. Students seeking help at the university counseling center because of academic difficulties were asked to participate in this 4-week program. Only students who were judged to be deficient in reading comprehension and other study-related skills were chosen for the program. Students who sought help at the counseling center for emotional difficulties were included in the study as a comparison group. They received the usual treatment offered at the counseling center. A group of 30 students completed the academic-improvement program at the counseling center. Average test grades from the students' courses for the two groups were compared before (midterm exams) and after (final exams) the program. Analyses revealed that a statistically significant majority of the students were doing better in school after completing the academic-improvement program than before.
What type of research design was used for this study?

Question 6

The most critical defining characteristic of a true experiment is often seen to be a high degree of control over the

A)arrangement of experimental conditions.
B)choice of dependent variables.
C)random assignment of participants to experimental conditions.
D)systematic manipulation of independent variables.

Question 7

Research done in natural settings,as compared to research done in laboratory settings,is more likely to emphasize

A)practical goals.
B)methodological goals.
C)ethical goals.
D)abstract goals.

Question 8

Why is the nonequivalent control group design superior to the one-group pretest-posttest design in terms of controlling for threats to internal validity?

Question 9

Experiments in natural settings are likely to differ from laboratory experiments on four critical dimensions: control,external validity,goals,and consequences.Briefly describe the nature of the difference for each of these four dimensions.

Question 10

The external validity of research done in natural settings is likely to be emphasized more when the research represents

A)social experimentation as the basis for large-scale changes.
B)an extension of a specific laboratory finding.
C)an experiment done to address a specific question raised by a specific company.
D)a theoretically motivated social psychology experiment.

Question 11

A researcher was interested in determining whether more frequent breaks (i.e., "coffee breaks") in a business setting would help employees to be more productive. With the cooperation of the management, employees on one floor of the corporate offices were allowed to take a 10-minute break each hour (at any time) between 8:00 and 11:00 A.M. (for a total of 30 minutes). The comparison group comprised employees on different floors who followed the usual corporate policy of taking a 30-minute break sometime during the morning (at any time). Measures of productivity were gathered for each employee according to his or her job (e.g., number of reports written, number of sales made, etc.). A time series analysis was applied to compare the productivity of both groups of employees for six months before and after the intervention (started in July). Quite surprisingly, the productivity of both groups increased following the onset of the intervention, suggesting to the researcher that the timing of breaks makes no difference.
Describe two ways in which contamination may have influenced the results of this study.

Question 12

In the context of conducting experiments in natural settings,random assignment of participants to conditions is

A)usually perceived by possible participants as the fairest procedure.
B)the fairest procedure if the effectiveness of the treatment is not known.
C)the fairest procedure if the effectiveness of the treatment is known.
D)usually perceived by those in positions of authority as the fairest procedure.

Question 13

At a large university, a group of education specialists tested the effectiveness of a new academic improvement course. Students seeking help at the university counseling center because of academic difficulties were asked to participate in this 4-week program. Only students who were judged to be deficient in reading comprehension and other study-related skills were chosen for the program. Students who sought help at the counseling center for emotional difficulties were included in the study as a comparison group. They received the usual treatment offered at the counseling center. A group of 30 students completed the academic-improvement program at the counseling center. Average test grades from the students' courses for the two groups were compared before (midterm exams) and after (final exams) the program. Analyses revealed that a statistically significant majority of the students were doing better in school after completing the academic-improvement program than before.
What major threats to internal validity are controlled by this research design?

Question 14

One of the main ways that true experiments differ from quasi-experiments is that true experiments use

A)correlational methods.
B)random assignment to conditions.
C)random selection from the population.
D)all of these

Question 15

A researcher was interested in determining whether more frequent breaks (i.e., "coffee breaks") in a business setting would help employees to be more productive. With the cooperation of the management, employees on one floor of the corporate offices were allowed to take a 10-minute break each hour (at any time) between 8:00 and 11:00 A.M. (for a total of 30 minutes). The comparison group comprised employees on different floors who followed the usual corporate policy of taking a 30-minute break sometime during the morning (at any time). Measures of productivity were gathered for each employee according to his or her job (e.g., number of reports written, number of sales made, etc.). A time series analysis was applied to compare the productivity of both groups of employees for six months before and after the intervention (started in July). Quite surprisingly, the productivity of both groups increased following the onset of the intervention, suggesting to the researcher that the timing of breaks makes no difference.
Describe one threat to internal validity that might be present in this study because the independent variable manipulation was implemented on different floors of the building.What information would you need to know in order to rule out this threat to internal validity?

Question 16

When it comes to control over the conditions in the experiment or over the assignment of participants to groups in the experiment,a researcher in a natural setting is likely to have

A)more control than a researcher in a laboratory setting.
B)the same amount of control as a researcher in a laboratory setting.
C)more control over the conditions in the experiment but less control over assignment of participants.
D)less control than a researcher in a laboratory setting.

Question 17

A researcher was interested in determining whether more frequent breaks (i.e., "coffee breaks") in a business setting would help employees to be more productive. With the cooperation of the management, employees on one floor of the corporate offices were allowed to take a 10-minute break each hour (at any time) between 8:00 and 11:00 A.M. (for a total of 30 minutes). The comparison group comprised employees on different floors who followed the usual corporate policy of taking a 30-minute break sometime during the morning (at any time). Measures of productivity were gathered for each employee according to his or her job (e.g., number of reports written, number of sales made, etc.). A time series analysis was applied to compare the productivity of both groups of employees for six months before and after the intervention (started in July). Quite surprisingly, the productivity of both groups increased following the onset of the intervention, suggesting to the researcher that the timing of breaks makes no difference.
Explain why the "Hawthorne effect" may have influenced the results of this study.

Question 18

One factor that may make it difficult to interpret the result of even a true experiment in a natural setting is contamination (when there is communication of information about the experiment between groups of participants).Briefly describe the three types of problems that can result from contamination.

Question 19

The difference between a true experiment conducted in a natural setting and a quasi-experiment conducted in a natural setting is that the

A)quasi-experiment will have greater external validity.
B)quasi-experiment will have greater internal validity.
C)quasi-experiment will lack a comparison condition.
D)none of these

Question 20

At a large university, a group of education specialists tested the effectiveness of a new academic improvement course. Students seeking help at the university counseling center because of academic difficulties were asked to participate in this 4-week program. Only students who were judged to be deficient in reading comprehension and other study-related skills were chosen for the program. Students who sought help at the counseling center for emotional difficulties were included in the study as a comparison group. They received the usual treatment offered at the counseling center. A group of 30 students completed the academic-improvement program at the counseling center. Average test grades from the students' courses for the two groups were compared before (midterm exams) and after (final exams) the program. Analyses revealed that a statistically significant majority of the students were doing better in school after completing the academic-improvement program than before.
Identify and describe two threats to internal validity in this study.That is,show why there are at least two plausible alternative hypotheses for the obtained change in grades other than due to the academic-improvement program.

Question 21

One threat to the internal validity of research that affects both true experiments and quasi-experiments is

A)selection threats.
B)experimenter expectancy effects.
C)additive effects with selection.
D)all of these

Question 22

When individuals' performance on a posttest differs from their initial testing not because of a treatment but because of familiarity with the measure,a __________ threat to internal validity is likely.

A)regression
B)instrumentation
C)testing
D)contamination

Question 23

A threat to internal validity that occurs when information about the experiment is communicated between the different groups of participants is known as

A)the cross-communication threat.
B)novelty effects.
C)contamination.
D)the Hawthorne effect.

Question 24

In a study of the effectiveness of a treatment for depression,a psychologist assesses patients' symptoms of depression using a reliable questionnaire both before and for several months after treatment.During the same time period,the psychologist also assesses the symptoms of individuals who are randomly assigned to a waiting-list control group.The results indicate that individuals in both groups experienced a decrease in their symptoms over the 8-month time period of the study.One threat to internal validity the researcher should consider is

A)regression.
B)maturation.
C)selection.
D)all of these

Question 25

An argument for the internal validity of quasi-experiments

A)can always be made solely on the basis of the results of the quasi-experiments.
B)can be made on the basis of supplementary data and logical analysis in addition to the results of the quasi-experiment itself.
C)can only be made if a corresponding true experiment has been done.
D)cannot be made because of inherent threats to internal validity in these designs.

Question 26

Contamination that results from communication of information between groups of participants in an experiment can lead to all of the following except

A)greater compliance with the instructions within each group.
B)resentment on the part of participants receiving less desirable treatments.
C)rivalry among participants receiving different treatments.
D)general diffusion of treatments across the groups.

Question 27

Students on two college campuses serve as treatment and control groups in a study investigating the effectiveness of an alcohol-abuse prevention campaign.A well-known student on one of the campuses dies of alcohol intoxication in the course of the study;students on the other campus did not learn of the student's death.The reaction of other students to the student's death on their campus could represent a potential threat to the internal validity of the study called

A)history.
B)selection.
C)additive effects of selection and history.
D)additive effects of selection and maturation.

Question 28

The main problem associated with subject attrition during the course of a study is that

A)groups initially created to be equivalent may no longer be equivalent.
B)the group with the most participants remaining at the end of a study will likely have higher posttest scores.
C)the researcher can no longer use statistical analysis to understand the results.
D)regression to the mean on the dependent variable measure is likely.

Question 29

An intervention in an office setting leads employees to be pleased that the management is interested in their welfare.If the employees' performance improves in this situation,the researcher should be concerned about potential

A)contamination effects.
B)lack of discontinuity in the time series.
C)selection threats to internal validity.
D)Hawthorne effects.

Question 30

When people's performance changes because they are enthusiastic or energized by an intervention,the results of a study are likely affected by

A)contamination.
B)treatment enthusiasm.
C)novelty effects.
D)diffusion of treatment.

Question 31

The hallmark of a quasi-experiment is

A)high levels of control over treatment conditions.
B)random assignment of participants to conditions.
C)absence of any specific intervention or treatment.
D)lack of random assignment of participants to conditions.

Question 32

A high school teacher conducted a test of a new approach to teaching math.Students were given a pretest when their math class began and a posttest at the end of the semester.The students' math performance increased.The teacher learned near the end of the semester,however,that in their science classes the students were using new computer software that included much of the math the teacher covered in his course.Which of the following threats to internal validity does the new computer software represent?

A)selection
B)regression
C)testing
D)history

Question 33

In clinical trials involving tests of new medical treatments it may be extremely difficult to get patients to agree to be randomly assigned to either the treatment or the control group.In these situations researchers can use

A)quasi-experimental designs.
B)natural groups designs.
C)yoked control designs.
D)matched groups designs.

Question 34

A researcher trains observers to complete checklists while observing children's behavior on the schoolyard during recess.Over the course of the study,observers become more reliable in their observations.Any effect of a treatment in this study might be confounded with an ____________ threat to internal validity.

A)observation
B)expectancy effect
C)additive
D)instrumentation

Question 35

Random assignment to conditions is used in true experiments to control which of the following threats to internal validity?

A)testing
B)selection
C)history
D)subject attrition

Question 36

When events that occur during the course of a study have a different effect on one group of participants than on another,the possible threat to internal validity is

A)history.
B)selection.
C)differential regression.
D)additive effects of selection and history.

Question 37

A physician wants to test the effectiveness of a new allergy medication.The physician consults a researcher who regularly conducts clinical trials.The physician tells the researcher that she has a backlog of 100 patients she is treating effectively with the prevailing medication but who could potentially benefit even more from the new medication.The researcher recommends that the physician do a true experiment because of the availability of a(n)

A)excess-demand control group.
B)quasi-treatment control group.
C)waiting-list control group.
D)convenience control group.

Question 38

A psychologist examines the effect of a new therapy by first assessing patients' symptoms using a pretest,then administering eight weeks of therapy,and then administering a posttest.Based on this research design,the psychologists will be able to

A)make a causal claim about whether the treatment is effective.
B)make a claim about the applicability of the treatment to other potential patients.
C)neither (A)nor (B)
D)both (A)and (B)

Question 39

When,from the outset of a study,differences exist between the kinds of individuals in one group of an experiment and those in another,there is a potential threat to internal validity called

A)selection.
B)maturation.
C)additive effect of selection and history.
D)contamination.

Question 40

Participants for a treatment group are chosen because they score very low on a pretest measure of performance.When their performance improves on the posttest,the researcher

A)can be confident that the treatment was effective.
B)can rule out testing and instrumentation threats to internal validity.
C)should consider the possibility that statistical regression influenced posttest scores.
D)must conclude that additive effects with selection are responsible for the outcome.

Question 41

Which of the following is not one of the four questions addressed in program evaluation?

A)assessment of needs
B)evaluation of process
C)final decision making
D)evaluation of outcome

Question 42

The one-group pretest-posttest design is

A)a bad experiment.
B)an important quasi-experimental design.
C)the best design for clinical research.
D)least affected by threats to internal validity.

Question 43

The relationship between basic research and applied research can best be described as

A)an independent relationship.
B)a reciprocal (or circular)relationship.
C)a hierarchical relationship with basic research taking precedence over applied research.
D)a top-down relationship with applied research taking precedence over basic research.

Question 44

If the residents,staff,and facilities of a nursing home where a quasi-experiment was done are likely to be different from those in other nursing homes,then the findings of the study are likely to lack

A)reliability.
B)statistical significance.
C)internal validity.
D)external validity.

Question 45

After implementing a program to improve admitting procedures in a hospital emergency room,a researcher seeks to determine whether the program is being implemented as proposed.Which research method will best serve this purpose?

A)survey method
B)observational method
C)experimental design
D)quasi-experimental design

Question 46

Program evaluation is a hybrid discipline whose professional activity

A)relies almost exclusively on the experimental method.
B)has as its basic goal providing feedback to providers of human service activities.
C)is confined almost entirely to the discipline of psychology.
D)focuses primarily on conducting basic research.

Question 47

Among the following designs,which allows researchers to rule out the most threats to internal validity?

A)times series design with nonequivalent control group
B)one group pretest-posttest design
C)simple interrupted time series design
D)nonequivalent control group design

Question 48

When individuals in one group of a nonequivalent control group design experience natural changes at a faster rate than individuals in the other group,there is danger of a threat to internal validity called

A)additive effects of selection and history.
B)additive effects of selection and instrumentation.
C)additive effects of selection and maturation.
D)differential statistical regression.

Question 49

The effect of a treatment in a simple interrupted time-series design is indicated by

A)a clear discontinuity (abrupt increase or decrease)in the dependent variable at the point the treatment is administered.
B)a decreasing trend in the dependent variable that is present both before and after the treatment.
C)an increasing trend in the dependent variable that is present both before and after the treatment.
D)a gradual change in the dependent variable that begins just before the treatment is implementeD.

Question 50

Social policy research (like the research done by Ellen Langer on the effects of living in a nursing home among the elderly)serves the purpose(s)of

A)informing policy decisions only.
B)advancing scientific theory and knowledge only.
C)advancing scientific theory and knowledge or informing policy decisions but not both.
D)advancing scientific theory and knowledge and informing policy decisions.

Question 51

A human services agency was interested in finding out whether establishing a food distribution center in their community would be providing a service the community would use.Which of the following types of assessments in program evaluation would this agency find useful for this question?

A)evaluation of process
B)evaluation of outcome
C)assessment of needs
D)evaluation of efficiency

Question 52

Which of the following threats to internal validity is controlled in the nonequivalent control group design?

A)differential regression
B)additive effect of selection and testing.
C)additive effect of selection and history.
D)testing

Question 53

A researcher was asked to evaluate the outcome of a program implemented to enhance recycling in a community.Which method will the researcher likely select to answer the question of whether the program was effective?

A)time series with nonequivalent control group design
B)participant observation
C)random groups design
D)cross-sectional survey design

Question 54

A student used a nonequivalent control group design to examine the effectiveness of a computer module the library uses to introduce first year students to the library's resources.Which threat to internal validity could she potentially rule out by examining the pretest scores for both groups?

A)a maturation threat
B)a regression threat
C)a selection threat
D)an instrumentation threat

Question 55

Even when pretest scores are the same,on average,for treatment and comparison groups in a nonequivalent control group design,the two groups may not be equivalent because

A)the pretest measure is unlikely to be relevant to the dependent variable.
B)the fact that the two groups are equal on the pretest measure does not ensure that the groups are equivalent on other characteristics relevant to the study.
C)the posttest measure is unlikely to be the same as the pretest measure,and the two groups must be equivalent at the posttest.
D)the natural growth rates of two groups from different populations are likely to be the same,but the pretest estimate of equality may still be in error.

Question 56

A clinical psychologist examined the effect of a treatment designed to reduce the amount of cigarette smoking in the day room of an inpatient psychiatric facility.His treatment consisted of a several components to maximize its potential effectiveness (e.g. ,increasing the number of available activities;giving patients responsibility for care of the unit's plants,fish,etc. ;the opportunity to exchange cigarettes for positive reinforcers).At the end of three months,his data indicated that cigarette smoking had declined significantly.Based on this,he is able to conclude that

A)any of the components,implemented by itself,would reduce smoking in the day room.
B)the treatment will work only when all of the components are present.
C)both (A)and (B)
D)neither (A)nor (B)

Question 57

When two groups in a nonequivalent control group design do not differ on the pretest,the researcher can conclude that

A)the two groups are equivalent only on the dependent variable measured on the pretest.
B)the two groups are not equivalent on any dependent variables.
C)the two groups will show the same natural growth rate over the course of the study.
D)the two groups are equivalent.

Question 58

A simple interrupted time series design provides a reliable measure of the effect of the treatment because there are many observations both before and after the treatment.The major threat to internal validity in this design is

A)maturation.
B)history.
C)selection.
D)additive effect of selection and history.

Question 59

Strong evidence of a treatment effect in a time series with nonequivalent control group design is indicated by

A)discontinuity in neither the treatment group nor the comparison group.
B)discontinuity in both the treatment group and the comparison group.
C)discontinuity in the comparison group but not in the treatment group.
D)discontinuity in the treatment group but not in the comparison group.

Question 60

Program evaluation represents perhaps the extreme case of

A)basic research.
B)applied research.
C)theory development.
D)experimental control.

Question 61

,the most significant change needed if our society is to approach social reforms as experiments is for public officials to emphasize

A)the efficiency of social programs.
B)the views expressed by particular interest groups.
C)the importance of the problem rather than the importance of the solution.
D)the evidence in favor of the public official's preferred solution.

Question 62

Which of the following is likely to be the most important benefit from the use of program evaluation based on sound experimental methodology?

A)providing information to policy makers that can be helpful in making more informed choices among possible treatments for social problems
B)advancing the influence and prestige of the social sciences in decision making by social policy makers
C)advancing the development of basic research in the social sciences by focusing on social policy issues
D)providing definitive information to policy makers such that program evaluation will eventually become the primary basis of social policy decisions

Question 63

Perhaps the greatest difference between basic research and program evaluation lies in the differences in

A)the political and social realities surrounding each of them.
B)statistical procedures.
C)the importance of the research.
D)methodology.