Deck 13: Experiments and Quasi-Experiments

Question 1

For quasi-experiments,

A)there is a particularly important potential threat to internal validity, namely whether the "as if" randomization in fact can be treated reliably as true randomization.
B)there are the same threats to internal validity as for true randomized controlled experiments, without modifications.
C)there is little threat to external validity, since the populations are typically already different.
D)OLS estimation should not be used.

Answer 1

A

Question 2

The following is not a threat to external validity:

A)the experimental sample is not representative of the population of interest.
B)the treatment being studied is not representative of the treatment that would be implemented more broadly.
C)experimental participants are volunteers.
D)partial compliance with the treatment protocol.

Answer 2

D

Question 3

Assume that data are available on other characteristics of the subjects that are relevant to determining the experimental outcome. Then including these determinants explicitly results in

A)the limited dependent variable model.
B)the differences in means test.
C)the multiple regression model.
D)large scale equilibrium effects.

Answer 3

C

Question 4

Program evaluation

A)is conducted for most departments in your university/college about every seven years.
B)is the field of study that concerns estimating the effect of a program, policy, or some other intervention or "treatment."
C)tries to establish whether EViews, SAS or Stata work best for your econometrics course.
D)establishes rating systems for television programs in a controlled experiment framework.

Question 5

The following estimation methods should not be used to test for randomization when X_i, is binary:

A)linear probability model (OLS)with homoskedasticity-only standard errors.
B)probit.
C)logit.
D)linear probability model (OLS)with heteroskedasticity-robust standard errors.

Question 6

Your textbooks gives several examples of quasi experiments that were conducted. The following is not an example of a quasi experiment:

A)labor market effects of immigration.
B)effects on civilian earnings of military service.
C)the effect of cardiac catheterization.
D)the effect of unemployment on the inflation rate.

Question 7

Causal effects that depend on the value of an observable variable, say W_i,

A)cannot be estimated.
B)can be estimate by interacting the treatment variable with W_i.
C)result in the OLS estimator being inefficient.
D)requires use of homoskedasticity-only standard errors.

Question 8

To test for randomization when X_i is binary,

A)you regress X_i, on all W's and compute the F-statistic for testing that all the coefficients on the W's are zero. (The W's measure characteristics of individuals, and these are not affected by the treatment.)
B)is not possible, since binary variables can only be regressors.
C)requires reordering the observations randomly and re-estimating the model. If the coefficients remain the same, then this is evidence of randomization.
D)requires seeking external validity for your study.

Question 9

Experimental data are often

A)observational data.
B)binary data, in that the subject either does or does not respond to the treatment.
C)panel data.
D)time series data.

Question 10

The Hawthorne effect refers to

A)subjects dropping out of the study after being randomly assigned to the treatment or control group.
B)the failure of individuals to follow completely the randomized treatment protocol.
C)the phenomenon that subjects in an experiment can change their behavior merely by being included in the experiment.
D)assigning individuals, in part, as a result of their characteristics or preferences.

Question 11

A repeated cross-sectional data set

A)is also referred to as panel data.
B)is a collection of cross-sectional data sets, where each cross-sectional data set corresponds to a different time period.
C)samples identical entities at least twice.
D)is typically used for estimating the following regression model Y_it = β₀ + β₁X_it + β₂W_1,it + ... + β_{1+ r}W_r,it + u_it

Question 12

Experimental effects, such as the Hawthorne effect,

A)generally are not germane in quasi-experiments.
B)typically require instrumental variable estimation in quasi-experiments.
C)can be dealt with using binary variables in quasi-experiments.
D)are the most important threat to internal validity in quasi-experiments.

Question 13

In the context of a controlled experiment, consider the simple linear regression formulation Y_i = ?₀ + ?₁X_i + u_i. Let the Y_i be the outcome, X_i the treatment level when the treatment is binary, and u_i contain all the additional determinants of the outcome. Then calling $\hat \beta _ { 1 }$ a differences estimator

A)makes sense since it is the difference between the sample average outcome of the treatment group and the sample average outcome of the control group.
B)and $\hat \beta _ { 0 }$ the level estimator is standard terminology in randomized controlled experiments.
C)does not make sense, since neither Y nor X are in differences.
D)is not quite accurate since it is actually the derivative of Y on X.

Question 14

In a quasi-experiment

A)quasi differences are used, i.e., instead of ?Y you need to use ( $\bar { Y } \text { after }$ - ? × $\bar { Y } \text { before }$ ), where 0 < ? < 1.
B)randomness is introduced by variations in individual circumstances that make it appear as if the treatment is randomly assigned.
C)the causal effect has to be estimated through quasi maximum likelihood estimation.
D)the t-statistic is no longer normally distributed in large samples.

Question 15

In the context of a controlled experiment, consider the simple linear regression formulation Y_i = ?₀ + ?₁X_i + u_i. Let the Y_i be the outcome, X_i the treatment level, and u_i contain all the additional determinants of the outcome. Then

A)the OLS estimator of the slope will be inconsistent in the case of a randomly assigned X_i since there are omitted variables present.
B)X_i and u_i will be independently distributed if the X_i be are randomly assigned.
C)?₀ represents the causal effect of X on Y when X is zero.
D)E(Y | X = 0)is the expected value for the treatment group.

Question 16

With panel data, the causal effect

A)cannot be estimated since correlation does not imply causation.
B)is typically estimated using the probit regression model.
C)can be estimated using the "differences-in-differences" estimator.
D)can be estimated by looking at the difference between the treatment and the control group after the treatment has taken place.

Question 17

The following does not represent a threat to internal validity of randomized controlled experiments:

A)attrition.
B)failure to follow the treatment protocol.
C)experimental effects.
D)a large sample size.

Question 18

All of the following are reasons for using the differences estimator with additional regressors, with the exception of

A)efficiency.
B)providing a check for randomization.
C)providing an adjustment for "conditional" randomization.
D)making the difference estimator easier to calculate than in the case of the differences estimator without the additional regressors.

Question 19

The following are reasons for studying randomized controlled experiment in an econometrics course, with the exception of

A)at a conceptual level, the notion of an ideal randomized controlled experiment provides a benchmark against which to judge estimates of causal effects in practice.
B)when experiments are actually conducted, their results can be very influential, so it is important to understand the limitations and threats to validity of actual experiments as well as their strength.
C)randomized controlled experiments in economics are common.
D)external circumstances sometimes produce what appears to be randomization.

Question 20

Heterogeneous population

A)implies that heteroskedasticity-robust standard errors must be used.
B)suggest that multiple characteristics must be used to describe the population.
C)effects can be captured through interaction terms.
D)refers to circumstances in which there is unobserved variation in the causal effect with the population.

Question 21

In a sharp regression discontinuity design,

A)crossing the threshold influences receipt of the treatment but is not the sole determinant.
B)the population regression line must be linear above and below the threshold.
C)X_i will in general be correlated with u_i.
D)receipt of treatment is entirely determined by whether W exceeds the threshold.

Question 22

Small sample sizes in an experiment

A)biases the estimators of the causal effect.
B)may pose a problem because the assumption that errors are normally distributed is dubious for experimental data.
C)do not raise threats to the validity of confidence intervals as long as heteroskedasticity-robust standard errors are used.
D)may affect confidence intervals but not hypothesis tests.

Question 23

Quasi-experiments

A)provide a bridge between the econometric analysis of observational data sets and the statistical ideal of a true randomized controlled experiment.
B)are not the same as experiments, and lessons learned from the use of the latter can therefore not be applied to them.
C)most often use difference-in-difference estimators, which are quite different from OLS and instrumental variables methods studied in earlier chapters of the book.
D)use the same methods as studied in earlier chapters of the book, and hence the interpretation of these methods is the same.

Question 24

In the case of heterogeneous causal effects, the following is not true:

A)in the circumstances in which OLS would normally be consistent (when E(u_i | X_i)= 0), the OLS estimator continues to be consistent.
B)OLS estimation using heteroskedasticity-robust standard errors is identical to TSLS.
C)the OLS estimator is properly interpreted as a consistent estimator of the average causal effect in the population being studied.
D)the TSLS estimator in general is not a consistent estimator of the average causal effect if an individual's decision to receive treatment depends on the effectiveness of the treatment for that individual.

Question 25

Testing for the random receipt of treatment

A)is not possible, in general.
B)entails testing the hypothesis that the coefficients on W_1i, …, W_ri are non-zero in a regression of X_i on W_1i, …, W_r.
C)is not meaningful since the LHS variable is binary.
D)entails testing the hypothesis that the coefficients on W_1i, …, W_ri are zero in a regression of X_i on W_1i, …, W_r.

Question 26

Failure to follow the treatment protocol means that

A)the OLS estimator cannot be computed.
B)instrumental variables estimation of the treatment effect should be used where the initial random assignment is the instrument for the treatment actually received.
C)you should use the TSLS estimator and regress the outcome variable Y on the initial random assignment in the first stage to get predicted values of the outcome variable.
D)the Hawthorne effect plays a crucial role.

Question 27

Threats to internal validity of quasi-experiments include

A)failure of randomization.
B)failure to follow the treatment protocol.
C)attrition.
D)all of the above with some modifications from true randomized controlled experiments.

Question 28

Earnings functions provide a measure, among other things, of the returns to education. It has been argued these regressions contain a serious omitted variable bias due to differences in abilities. Furthermore, ability is hard to measure and bound to be highly correlated with years of schooling. Hence the standard estimate of about a 10 percent return to every year of schooling is upward biased. Suggest some ways to address this problem. One famous study looked at earnings of identical twins. Explain how this can be viewed as a quasi-experiment, and mention some of the threats to internal and external validity that such a study might encounter.

Question 29

If the causal effect is different for different people, then the population regression equation for a binary treatment variable X_i, can be written as

A)Y_i = β₀ + β₁X_i + u_i.
B)Y_i = β₀ + β_1iX_i + u_i.
C)Y_i = β_0i β_1iX_i + u_i.
D)Y_i = β₀ + β₁G_i + β₂D_t + u_i.

Question 30

One of the major lessons learned in the chapter on experiments and quasi-experiments

A)is that there are almost no true experiments in economics and that quasi-experiments are a poor substitute.
B)you should always use TSLS when estimating causal effects in quasi-experiments.
C)populations are always homogeneous.
D)is that the insights of experimental methods can be applied to quasi-experiments, in which special circumstances make it seem "as if" randomization has occurred.

Question 31

Assume for the moment that the student-teacher ratio effect on test scores was large enough that you would advocate reducing class sizes in elementary schools. In 1996, the State of California reduced class sizes from K-3 to no more than 20 students across all public elementary schools (Class Size Reduction Act)at a cost of approximately $2 billion. In a short essay, discuss why the general equilibrium effects might differ from the results obtained using experiments.

Question 32

Canada and the United States had approximately the same aggregate unemployment rates from the 1920s to 1981. In 1982, a two percentage point gap appears, which has roughly persisted until today, with the Canadian unemployment rate in the third quarter of 2002 being 7.6 percent while the American rate stood at 5.9 percent in the same period. Several authors have investigated this phenomenon. One study, published in 1990, contained the following statement: "It is a clichė that, as compared to analysis in the physical sciences, economic analysis is hampered by the lack of controlled experiments. In this regard, study of the Canadian economy can be much facilitated by comparison with the behavior of the US …" Discuss what the authors may have had in mind. List some potential threats to internal and external validity when comparing aggregate unemployment rate behavior between countries.

Question 33

A causal effect for a single individual

A)can be deduced from the average treatment effect.
B)cannot be measured.
C)depends on observable variables only.
D)is observable since it is used as part of calculating the mean of individual causal effects.

Question 34

A potential outcome

A)is the outcome for an individual under a potential treatment.
B)cannot be observed because most individuals do not achieve their potential.
C)is the same as a causal effect.
D)is none of the above.

Question 35

Roughly ten percent of elementary schools in California have a system whereby 4^th to 6^th graders share a common classroom and a single teacher (multi-age, multi-grade classroom). Suggest an experimental design that would allow you to assess the effect of learning in this environment. Mention some of the threats to internal and external validity and how you would attempt to circumvent these.

Question 36

The major distinction between the experiments and quasi-experiments chapter and earlier chapters is the

A)frequent use of binary variables.
B)type of data analyzed and the special opportunities and challenges posed when analyzing experiments and quasi-experiments.
C)superiority of TSLS over OLS.
D)use of heteroskedasticity-robust standard errors.

Question 37

Randomization based on covariates is

A)not of practical importance since individuals are hardly ever assigned in this fashion.
B)dependent on the covariances of the error term (serial correlation).
C)a randomization in which the probability of assignment to the treatment group depends on one of more observable variables W.
D)eliminates the omitted variable bias when using the difference estimator based on Y_i = β₀ + β₁X_i + u_i, where Y is the outcome variable and X is the treatment indicator.

Question 38

A repeated cross-sectional data set is

A)a collection of cross-sectional data sets, where each cross-sectional data set corresponds to a different time period.
B)the same as a balanced panel data set.
C)what Card and Krueger used in their study of the effect of minimum wages on teenage employment.
D)time series.

Question 39

You want to study whether or not the use of computers in the classroom for elementary students has an effect on performance. Explain in some detail how you would ideally set up such an experiment and what threats to internal and external validity there might be.

Question 40

Describe the major differences between a randomized controlled experiment and a quasi-experiment.

Question 41

Your textbook gives a graphical example of , where outcome is plotted on the vertical axis, and time period appears on the horizontal axis. There are two time periods entered: "t = 1"
and "t = 2." The former corresponds to the "before" time period, while the latter represents the "after" period. The assumption is that the policy occurred sometime between the time periods (call this "t = p"). Keeping in mind the graphical example of , carefully read what a reviewer of the Card and Krueger (CK)study of the minimum wage effect on employment in the New Jersey-Pennsylvania study had to say:
"Two assumptions are implicit throughout the evaluation of the 'natural experiment:' (1)[ ] would be zero if the treatment had not occurred, so a nonzero [ ] indicates the effect of the treatment (that is, nothing else could have caused the difference in the outcomes to change), and (2)… the intervention occurs after we measure the initial outcomes in the two groups. … Three conditions are particularly relevant in interpreting CK's work: (1)[t = 1] must be sufficiently before [t = p] that [the treatment group] did not adjust to the treatment before [t=1] - otherwise [ - ] will reflect the effect of the treatment; (2)[t = 2] must be sufficiently after [t = p] to allow the treatment's effect to be fully felt; and (3)we must be sure that the same difference [ - ] would have been observed at [t = 2] if the treatment had not been imposed, that is, [the control group must be good enough] that there is no need to adjust the differences for factors other than the treatment that might have caused them to change."
Use a figure similar to the textbook to explain what this reviewer meant.

Question 42

Consider the simple population regression model where the treatment is the same for the members of the treatment group, and hence X is a binary variable. Explain why the coefficient on X represents the difference between two means. How is the test for the statistical significance of the coefficient on X related to the test for differences in means between two populations, when their variances are different? Write down the null and alternative hypothesis in each case.

Question 43

The New Jersey-Pennsylvania study on the effect of minimum wages on employment mentioned in your textbook used a comparison in means "before" and "after" analysis. The difference-in-difference estimate turned out to be 2.76 with a standard error of 1.36.
The authors also used a difference-in-differences estimator with additional regressors of the type
ΔY_i = β₀ + β₁X_i + β₂W_1,t + ... + β_{1+ r}W_r,i + u_i
where i = 1, …, 410. X is a binary variable taking on the value one for the 331 observations in New Jersey. Since the authors looked at Burger King, KFC, Wendy's, and Roy Rogers fast food restaurants and the restaurant could be company owned, four W-variables were added.
(a)Given that there are four chains and the possibility of a company ownership, why did the authors not include five W-variables?
(b)OLS estimation resulted in ₁ of 2.30 with a standard error of 1.20. Test for statistical significance and specify the alternative hypothesis.
(c)Why is this estimate different from the number calculated from Δ - Δ = 2.76? What is the advantage of employing this estimator of the simple difference-in-difference estimator?

Question 44

Define the in terms of observable differences in the treatment and control group, before and after the treatment. Explain why this presentation is the equivalent of calculating the coefficient in a regression framework.

Question 45

Present alternative estimators for causal effects using experimental data when data is available for a single period or for two periods. Discuss their advantages and disadvantages.

Question 46

(Requires Appendix material)Discuss how the differences-in-differences estimator can be extended to multiple time periods. In particular, assume that there are n individuals and T time periods. What do the individual and time effects control for?

Question 47

Your textbook mentions use of a quasi-experiment to study the effects of minimum wages on employment using data from fast food restaurants. In 1992, there was an increase in the (state)minimum wage in one U.S. state (New Jersey)but not in neighboring location (Eastern Pennsylvania). To calculate the you need the change in the treatment group and the change in the control group. To do this, the study provides you with the following information Where FTE is "full time equivalent" and the numbers are average employment per restaurant.
(a)Calculate the change in the treatment group, the change in the control group, and finally Since minimum wages represent a price floor, did you expect to be positive or negative?
(b)If you look at , is this number primarily due to a change in the treatment group or the control group? Is this what you expected?
(c)The standard error for is 1.36. Test whether or not the coefficient is statistically significant, given that there are 410 observations. If you believed that the benefit from small minimum wage increases outweighed the cost in terms of employment loss, would finding that this coefficient was not statistically significant discourage you?

Question 48

Specify the multiple regression model that contains the difference-in-difference estimator (with additional regressors). Explain the circumstances under which this model is preferable to the simple difference-in-difference estimator. Explain how the W's can be used to test for randomization. How does the interpretation of the W variables change compared to the differences estimator with additional regressors?

Question 49

To analyze the effect of a minimum wage increase, a famous study used a quasi-experiment for two adjacent states: New Jersey and (Eastern)Pennsylvania. A was calculated by comparing average employment changes per restaurant between to treatment group (New Jersey)and the control group (Pennsylvania). In addition, the authors provide data on the employment changes between "low wage" restaurants and "high wage" restaurants in New Jersey only. A restaurant was classified as "low wage," if the starting wage in the first wave of surveys was at the then prevailing minimum wage of $4.25. A "high wage" restaurant was a place with a starting wage close to or above the $5.25 minimum wage after the increase.
(a)Explain why employment changes of the "high wage" and "low wage" restaurants might constitute a quasi-experiment. Which is the treatment group and which the control group?
(b)The following information is provided Where FTE is "full time equivalent" and the numbers are average employment per restaurant.
Calculate the change in the treatment group, the change in the control group, and finally Since minimum wages represent a price floor, did you expect to be positive or negative?
(c)The standard error for is 1.48. Test whether or not this is statistically significant, given that there are 174 observations.

Question 50

Let the vertical axis of a figure indicate the average employment fast food restaurants. There are two time periods, t = 1 and t = 2, where time period is measured on the horizontal axis. The following table presents average employment levels per restaurant for New Jersey (the treatment group)and Eastern Pennsylvania (the control group). Enter the four points in the figure and label them , , , and Connect the points. Finally calculate and indicate the value for