Deck 4: Criteria for Inferring Effectiveness: How Do We Know What Works

Question 1

Which of the following questions is/are important when critically appraising a study that purports to test the effectiveness of an intervention, program, or policy?

A) From a logical standpoint, was the intervention, program, or policy the most plausible cause of the observed outcome, or is some alternative explanation also quite plausible?
B) Was outcome measured in a valid and unbiased manner?
C) What is the probability that the apparent effectiveness, or lack thereof, can be attributed to statistical chance?
D) Do the study participants, intervention procedures and results seem applicable to your practice context?
E) All of the above.

Answer 1

E

Question 2

Internal validity refers to:

A) the extent that its design arrangements enable us to ascertain logically whether the observed outcome really reflects the effectiveness (or lack thereof) of our intervention versus some alternative explanation.
B) whether a measurement instrument really measures what it intends to measure.
C) whether a study has blind raters.
D) statistical significance.

Answer 2

A

Question 3

To conclude logically that an intervention really is the cause of a particular outcome:

A) the provision of the intervention must precede or coincide with the change in the outcome being measured.
B) changes in outcome must be associated with changes in the intervention condition.
C) plausible alternative explanations for any correlation between intervention and outcome must be ruled out.
D) All of the above.

Answer 3

D

Question 4

As a threat to internal validity, history refers to:

A) a client's earlier life experiences.
B) the possibility that other events may have coincided with the provision of the intervention and may be the real cause of the observed outcome.
C) the passage of time as the explanation for an outcome.
D) None of the above.

Question 5

Immediately after experiencing a trauma, a client begins receiving trauma-focused therapy and continues receiving the intervention for 3 months. At the conclusion of treatment the client's trauma symptoms are much less severe than at the start of treatment. From this information we can conclude that:

A) the treatment was effective.
B) the improvement may have been caused by the mere passage of time.
C) history might be the explanation for the improvement.
D) Both b and c are correct.

Question 6

A special unit in a child welfare agency provides an innovative treatment to those cases with the worst assessment scores. To evaluate the unit's effectiveness, the degree of improvement among its clients in assessment scores is compared to the degree of improvement among cases in other units. The evaluation finds that the cases in the special unit had a significantly higher average degree of improvement than did the cases in the other units. From this information we can conclude that:

A) the special unit is effective.
B) the difference in outcome might be due to statistical regression to the mean.
C) the study had a high degree of internal validity.
D) Both a and c are correct.

Question 7

A free, voluntary continuing education workshop in EBP is offered to practitioners in a large agency. Several months after the workshop is provided, a program evaluator finds that workshop attendees are much more likely to be engaging in EBP than those who did not attend. From this information we can conclude that:

A) the workshop was effective.
B) the difference in outcome might be due to a selectivity bias.
C) the study had a high degree of internal validity.
D) Both a and c are correct.

Question 8

The best way to avoid a selectivity bias is to:

A) randomly assign clients to different treatment conditions.
B) find two groups of clients that seem to be comparable in light of your practice expertise and knowledge of client characteristics.
C) provide services on a first-come, first-served basis, putting the later ones on a wait list.
D) None of the above.

Question 9

Which of the following statements is/are true about statistical significance?

A) If a finding is statistically significant, then we need not worry about design or measurement flaws.
B) Mathematics dictates that the significance level must always be .05.
C) It can be influenced by sample size.
D) All of the above.
E) None of the above.

Question 10

If a study has excellent internal validity and finds that an intervention is very effective, we can conclude that:

A) its conclusions will be applicable to our clients.
B) its sample is representative.
C) the intervention will be effective in our practice context.
D) All of the above.
E) None of the above.

Question 11

If a study is internally valid we can generalize it to our practice context.

Question 12

A serious measurement bias can destroy the credibility of an otherwise rigorously designed randomized experiment.

Question 13

History refers to the possibility that other events may have coincided with the provision of the intervention and may be the real cause of the observed outcome.

Question 14

Testing one group of clients before and after they receive an intervention is sufficient for controlling for the passage of time.

Question 15

Random assignment of clients to treatment conditions means the same as haphazard assignment to treatment conditions.

Question 16

Statistical chance refers to flukes in random assignment that can be the reason why one group's outcome is better than another's.

Question 17

When reading a report of a study that failed to get statistically significant results, you should take into account its sample size.

Question 18

The external validity of a study pertains to whether the study participants, intervention procedures, and results seem applicable to the evidence-based practitioner's practice context.

Question 19

Define internal validity, list the main threats to it, and explain each threat.

Question 20

Explain the function and importance of random assignment.

Question 21

Explain the need to rule out statistical chance as a plausible explanation for study outcomes.

Question 22

Define external validity and describe its role in making EBP decisions.