Deck 4: The Scientific Method: The Gold Standard for Establishing Causality

Question 1

Let Y_1i be the output for factory i when it receives a new machine, and let Y_0i be the output for factory i when it does not receive a new machine. Also let Y_i be the realized output for factory i and let D_i be a binary variable that equals one if factory i actually received a new machine. If the company's headquarters randomly assigns new machines to a subset of the company's factories, this ensures:

A) E(Y_i|D_i = 1) = E(Y_1i - Y_0i)
B) E(Y_i|D_i = 1) - E(Y_i|D_i = 0) = E(Y_1i - Y_0i)
C) E(Y_i|D_i = 0) - E(Y_i|D_i = 1) = E(Y_1i - Y_0i)
D) E(Y_i|D_i = 1) = E(Y_i|D_i = 0)

Answer 1

B

Question 2

In the scientific method, why is it crucial to start with a question before conducting the empirical analysis?

A) It makes writing the results easier.
B) It facilitates survey responses from subjects.
C) It motivates the specific variation in the treatment required to test the hypothesis.
D) Questions are easier to communicate than propositions.

Answer 2

C

Question 3

In running an experiment, it is crucial that there exists both subjects who receive the treatment, but also subjects who do not because:

A) without variation in the treatment, testing the hypothesis would not be feasible.
B) the treatment will be well calibrated.
C) this allows you to run regressions in the analysis step.
D) None of the answers is correct.

Answer 3

A

Question 4

As long as our sample is large enough it will be the case that the average outcome for the untreated group should have what distribution?

A) Normal
B) Skewed distribution
C) T-distribution
D) The same as the underlying population distribution

Question 5

Which of the following is not one of the six steps of the scientific method?

A) Ask a question
B) Conduct an experiment
C) Communicate the findings
D) Gather market research data

Question 6

When implementing the scientific method, one moves from a research question to a proposed idea based on limited evidence that justifies further investigation, also known as a:

A) test statistic.
B) hypothesis.
C) p-value.
D) determining function.

Question 7

The process designed to generate knowledge through the collection and analysis of experimental data is known as the:

A) scientific method.
B) survey-based research.
C) business analytics paradigm.
D) "big data" revolution.

Question 8

While there are a few instances in the business world where one will observe experimental data, understanding the scientific method is critical because:

A) it's the benchmark for analysis in the "hard" sciences.
B) it's the gold standard for establishing causality.
C) it leads to larger treatment effects.
D) it is often cheaper to implement than observational studies.

Question 9

Randomizing the treatment in an experiment facilitates all of the following conclusions except for what?

A) ETT = ATE
B) Selection Bias = 0
C) ATE = 0
D) None of the answers is correct.

Question 10

The government runs an experiment where a random sample of 200 adults over 40 get a 5% tax cut and a random sample of 200 adults under 40 get no tax cut. Their results show that spending, on average, increased 8% with a 5% tax cut. They conclude that a 5% tax cut for all adult Americans will increase spending by 8%. Why is this logic flawed?

A) The effect of the treatment on the treated may not be the average treatment effect.
B) A 5% tax cut for all Americans suffers from selection bias.
C) The measured effect of the treatment on the treated may be biased.
D) The logic is not flawed.

Question 11

Among the six steps of the scientific method, the first step involves:

A) writing a grant proposal.
B) asking a question.
C) gathering market data.
D) writing up the results.

Question 12

A treatment effect is:

A) an unbiased estimate.
B) the change in the outcome resulting from variation in the treatment.
C) the variation in the treatment.
D) the different treatments among treated and control groups.

Question 13

Often times a hypothesis test will involve determining the average treatment effect (ATE) or the:

A) treatment effect of the average subject in the sample.
B) treatment effect averaged over many replications of the same experiment.
C) average difference in the treated and untreated outcome across subjects in the population.
D) None of the answers is correct.

Question 14

The critical hurdle with measuring treatment effects is that:

A) our subjects cannot be both untreated and treated at the same time.
B) we have selection bias.
C) it's always equal to the effect of the treatment on the treated.
D) it's hard to have a truly random treatment assignment.

Question 15

When running an experiment, suppose we assume the participants in the experiment are a random sample of the population. Let Y_i be the outcome for individual i and let D_i equal one if individual i receives the treatment and zero otherwise. What does the assumption of a random treatment imply?

A) - approximates E(Y_i)
B) approximates E(Y_i|D_i = 1) - E(Y_i|D_i = 0)
C) - = 0
D) Selection Bias = 0

Question 16

Regardless of how treatment was assigned, what can be assumed about what the difference in the average outcomes of the treated and untreated groups is equal to?

A) Effect of the treatment on the treated + Selection Bias
B) Effect of the treatment on the treated
C) Average treatment effect (ATE)
D) (ETT)

Question 17

Which of the following limits the use of experimental data in business settings/applications?

A) The software to handle the analysis of experimental data is prohibitively expensive.
B) Experimental data is often too large for businesses to handle.
C) Conducting experiments relevant for business questions are often not feasible.
D) Most business applications do not involve questions centered on causality.

Question 18

Something that is administered to members of at least one participating group is known as the:

A) treatment.
B) treatment effect.
C) potential outcome.
D) average treatment effect.

Question 19

The change in the outcome resulting from variation in the treatment is known as the:

A) selection bias.
B) random treatment.
C) treatment effect.
D) effect of the treatment on the treated.

Question 20

As long as our sample is large enough, it will be the case that the average outcome for the treated group should have what distribution?

A) Normal
B) Skewed distribution
C) T-distribution
D) The same as the underlying population distribution

Question 21

In evaluating the hypothesis test on experimental data, all the following will change the resulting p-value except for what?

A) Number of subjects receiving the treatment
B) Number of subjects receiving the control
C) Hypothesized value in the null hypothesis
D) Confidence level

Question 22

Suppose you wanted to determine if you should reject the null hypothesis that running a playing fast tempo (as opposed to slow) music in your store has no effect on sales. On 65 randomly selected days with fast tempo music on the average store sales is $2,345 with a sample standard deviation of 45, while on 75 randomly selected days with slow tempo music on the average store sales is $2,555, with a sample standard deviation of 65. What would be the proper t-stat for this hypothesis test?

A)
B)
C)
D)

Question 23

If you have a sufficiently large random sample from the population, but do not know how treatment was assigned, what object will a confidence interval centered on the difference in average outcomes for the treated and control groups be describing?

A) ATE
B) ETT
C) Selection Bias
D) ETT + Selection Bias

Question 24

In evaluating the hypothesis test on experimental data, all the following will change the resulting test statistic except for what?

A) Number of subjects receiving the treatment
B) Number of subjects receiving the control
C) Hypothesized value in the null hypothesis
D) Confidence level

Question 25

In building a confidence interval for ATE in experimental data, which of the following is not required?

A) The average outcome of the treated group
B) The average outcome of the control group
C) The number of subjects in the control group
D) Null hypothesis proposed value of the ATE

Question 26

In building a confidence interval for the ATE of an experiment, all the following objects will influence the spread of the confidence interval except what?

A) Confidence level
B) Number of subjects in the treated group
C) Number of subjects in the control group
D) Sample average outcome for the control group

Question 27

Given the average outcomes for the treated and control groups, and their respective sample standard deviations how does the number of observations impact the p-value of a hypothesis test?

A) Only increasing the number of treated will lower the p-value.
B) Only increasing the number of control groups will lower the p-value.
C) Only increasing the number of treated will raise the p-value.
D) None of the answers is correct.

Question 28

If you have a sufficiently large sample that was randomly drawn from your target population with a randomly assigned treatment then all of the following conditions hold except for what?

A) Selection bias = 0
B) Difference in average outcomes for treated and control groups is an unbiased estimate of the ATE.
C) The distribution for the difference in average outcomes for treated and control groups will be distributed normally.
D) None of the answers is correct.

Question 29

Which of the following departments is likely to experience experimental data more frequently than nonexperimental data?

A) Pricing
B) Marketing
C) HR
D) R&D

Question 30

Suppose you wanted to determine if you should reject the null hypothesis that running a playing fast tempo (as opposed to slow) music in your store decreases sales by $100 on average. On 65 randomly selected days with fast tempo music on the average store sales is $2,345 with a sample standard deviation of 45, while on 75 randomly selected days with slow tempo music on the average store sales is $2,555, with a sample standard deviation of 65. What would be the proper t-stat for this hypothesis test?

A)
B)
C)
D)

Question 31

Given the average outcomes for the treated and control groups, and their respective sample standard deviations, how does the number of observations impact the spread of a confidence interval?

A) Only increasing the number of treated will decrease the spread of the confidence interval
B) Only increasing the number of control groups will decrease the spread of the confidence interval
C) Only increasing the number of treated will increase the spread of the confidence interval
D) None of the answers is correct.

Question 32

Suppose you want to build a 95% confidence interval for the ATE when the average outcome for the treated (55 subjects) is 12, with a sample standard deviation of 4, while the average outcome for the control (65 subjects) is 10, with a sample standard deviation of 6. Which of the following would be the proper confidence interval?

A) 0 ± 1.96
B) 0 ± 2.58
C) (12 - 10) ± 1.96
D) 0 ± 1.96

Question 33

Which of the following fields is likely to experience nonexperimental data more frequently than experimental data?

A) Business
B) Chemistry
C) Pharma
D) Psychology

Question 34

In building a confidence interval for the ATE of an experiment, what object will determine the center of the confidence interval?

A) The average outcome for the treated group
B) The average outcome for the control group
C) The average outcome for the treated group minus the average outcome for the control group
D) None of the answers is correct.

Question 35

Suppose you want to build a 90% confidence interval for the ATE when the average outcome for the treated (55 subjects) is 12, with a sample standard deviation of 4, while the average outcome for the control (65 subjects) is 10, with a sample standard deviation of 6. Which of the following would be the proper confidence interval?

A) 0 ± 1.96
B) 0 ± 1.65
C) 2 ± 1.65
D) 0 ± 1.96

Question 36

Which of the following is not an assumption required to build a confidence level for an ATE in an experimental design?

A) You have a random sample from the population.
B) You randomly assigned treatment.
C) The outcomes for the treated group and control group are the same.
D) Your sample is large enough.

Question 37

In testing if promotions (binary) P_i have an effect on sales, S_i with a null hypothesis that the average treatment effect is zero, the null hypothesis could be written as:

A) E [S_i (P_i = 1) - S_i (P_i = 0)] = 0
B) E [P_i - S_i ] = 0
C) E [P_i - S_i ] = 1
D) E [S_i (P_i = 1) - S_i (P_i = 0)] = ETT

Question 38

Which of the following will yield nonexperimental data?

A) A clinical trial for determining the efficacy of a new drug
B) Crop yields from plots subject to different types of fertilizers
C) Investment performance over the last ten years of several portfolio managers
D) None of the answers is correct.

Question 39

Suppose Apple is considering increasing its advertising expenditures to promote the most recent iPhone. To try to assess the effect of such a move, it looks at sales in several small markets. In some of these markets Apple increased advertising expenditures by 30% and in others there was no change. When conducting this "experiment," what is the treatment?

A) iPhone sales in the small markets that were studied
B) The number of small markets that received an increase in advertising expenditures
C) The increase in advertising expenditures
D) The increase in iPhone sales in markets experiencing the increase in advertising expenditures

Question 40

In building a confidence interval for the ATE of an experiment, what object will determine the spread of the confidence interval?

A) Null hypothesis value
B) Average outcome for the treated group
C) Average outcome for the control group
D) Sample standard deviation for the treated group

Question 41

How can applying the insights from the experiment ideal and the scientific method approach facilitate better analysis with nonexperimental data?

A) Sharpen attention towards the variation in treatment that is most appropriate for measuring treatment effects
B) Illustrates the role of measurement error
C) Illustrates the role of sample size in appealing to the use of the normal distribution for probability assessments
D) None of the answers is correct.

Question 42

Why should we expect to estimate the treatment effect of price on quantity sold to be so difficult in nonexperimental data?

A) Firms are not likely to keep detailed records of their prices.
B) Obtaining actual quantity information from sales records is difficult.
C) Firms vary their prices strategically in response to expectations of the resulting sales.
D) Sales taxes create wedges between prices paid and revenue received by the firm.

Question 43

In trying to measure the "treatment effect" of getting an MBA on career earnings, the observation that individuals with an MBA have higher salaries than individuals without them is likely to suffer from which of the following?

A) ETT < ATE
B) Selection Bias > 0
C) ETT > ATE
D) ETT = ATE

Question 44

In trying to measure the "treatment effect" of taking Tylenol on reducing your "next day's temperature from today", and given the fact that you only take Tylenol when you're feeling sick, which of the following conditions are likely?

A) ETT > ATE
B) Selection Bias > 0
C) Selection Bias < 0
D) ETT = ATE

Question 45

Given that nonexperimental datasets are likely to have treatments that have not been randomly assigned, what is likely to contribute to difference in average outcomes between groups with different treatment levels?

A) Some of those differences are driven by a selection bias
B) Non-normality of the underlying population distribution
C) Skewed perceptions of quality
D) Heteroskedasticity

Question 46

Nonexperimental data are likely to have issues with all of the follow assumptions except for what?

A) Random sample from the population
B) Random assignment of treatment
C) Sufficiently large sample size to appeal to normality
D) None of the answers is correct.

Question 47

In nonexperimental data the nonrandom assignment of treatment leads to what?

A) Selection Bias ≠ 0
B) Non-normal underlying population distribution
C) Mismeasured standard deviations
D) Loss of degrees of freedom

Question 48

Which of the following is an example of nonrandom treatment assignment in nonexperimental data?

A) Effect of being drafted (in a lottery) on career earnings
B) Prices impact on number of products sold
C) Lottery winners' propensity to file for bankruptcy
D) None of the answers is correct.

Question 49

The most robust (but perhaps impractical) way to estimate the price elasticity of demand for your product would be:

A) to ask past customers their willingness to pay.
B) convert your products prices in real terms and estimate a demand curve.
C) randomize the price over a period of time and estimate the difference in sales resulting from those changes.
D) announce an upcoming discount, and track sales increases pre-post.

Question 50

What is the most critical hurdle using nonexperimental data must overcome?

A) Lack of the random assignment of the treatment
B) Measuring outcomes effectively
C) Overcoming high attrition rates
D) Removing measurement error