Deck 24: Significance, Importance, and Undetected Differences

Question 1

What is the statistical reason that small samples result in a harder time detecting real relationships or differences in the population than large samples? (Assume data quality is not an issue.)

Answer 1

small samples have more variability from sample to sample, making it harder for the test to rule out chance as the explanation for the relationship or difference.

Question 2

Which of the following statements must be true regarding a p-value of .0001?

A)Since this p-value is so small, the results will have practical as well as statistical significance.
B)The magnitude of the effect found in this study must have been huge.
C)There must have been very little variability in the results of this study.
D)None of the above.

Answer 2

None of the above.

Question 3

Suppose a group of people is interested in promoting the upcoming school tax levy.They took a survey asking people's opinion on the upcoming school tax levy (yes, no, no opinion) and they created a confidence interval for the proportion of people who were opposed to it.Their confidence interval fell completely under 50%, so less than a majority of the people oppose the levy, according to these results.Is it fair for them to conclude that "a majority of the people are in favor of upcoming school tax levy?" If yes, explain why.If no, rewrite the conclusion to be correct.

Answer 3

No, Rewrite: "A majority of the people are either in favor or have no opinion on the upcoming tax levy." Or, "A majority of the people do not oppose the upcoming tax levy."

Question 4

Explain how the natural variability in the population can affect a test's ability to detect a difference or relationship in the population that actually does exist.

Question 5

Suppose a study found a strongly statistically significant relationship between two variables.The test statistic and the p-value for this study __________ (choose: would, would not) provide information about the magnitude of the effect.

Question 6

Explain why it is important to not only be aware of the proportion of people who were in favor or opposed to something, but the proportion of undecideds as well.

Question 7

Suppose a company says they deliver their packages in 2 days or less, on average, and you are concerned that the packages are taking longer to deliver than promised.You conduct a hypothesis test to help you answer this question.Which alternative hypothesis would give you the best chance of detecting an actual problem, a one-sided alternative (average time of delivering packages is greater than 2 days) or a two-sided alternative (average time delivering packages is not equal to 2 days)? Explain your answer.If both have the same chance, explain why.

Question 8

Explain how the public can be misled by the results of a "very large study." (Assume data quality is not an issue.)

Question 9

Which of the following statements is false?

A)If the sample size is large enough, almost any null hypothesis can be rejected.
B)There is almost always a slight relationship between two variables, or a difference between two groups, and if you collect enough data, you will find it.
C)Larger samples sizes always produce more meaningful results than small sample sizes.
D)None of the above statements are false.

Question 10

Is there such a thing as a sample size that was too large, in terms of producing misleading conclusions about the population? __________ (choose: yes, no)

Question 11

Suppose a chi-square statistic testing for a relationship between two variables was found to be 1.746, with p-value 0.19.What happens to the chi-square statistic and the resulting p-value if you triple the sample size?

A)The chi-square statistic triples and so does the p-value.
B)The chi-square statistic triples resulting in a smaller p-value.
C)The chi-square statistic goes up by a factor of 3 squared, and the p-value changes accordingly.
D)None of the above.

Question 12

Suppose a study found a strongly statistically significant relationship between two variables.From the p-value for this study you __________ (choose: could, could not) find the probability that there is a relationship between these two variables in the population.

Question 13

Suppose a researcher examines a possible relationship in the population and the results are found to not be statistically significant.Can the researcher conclude that there was no significant relationship in the population? If yes, explain why; if no, describe how the researcher should state their conclusion regarding a possible relationship.

Question 14

Suppose a researcher examined a relationship between taking aspirin every day (yes/no) and the incidence of a heart attack (yes/no).The sample data showed that those taking aspirin had a lower chance of incidence of heart attack.A chi-square test on the data collected resulted in a p-value less than 0.0001.The data were collected in a well-designed randomized experiment.Which of the following conclusions is appropriate?

A)The probability that there is a relationship between the two variables is more than 99.99%.
B)For this population, taking aspirin reduces the risk of heart attack.
C)Heart attacks happened much more often for the non-aspirin takers than the aspirin takers.
D)All of the above conclusions are appropriate.

Question 15

Suppose you read the following headline: "Taking an aspirin a day can cut your risk of heart attack by almost half!" Suppose the news report goes on to tell you that the research is based on a very large well-designed randomized experiment with a p-value less than 0.0001.Assume this information is technically correct, given the results of the study.Which of the following statistics could actually represent the results of this study?

A)The rates of heart attack were 9.4 per thousand for the aspirin group and 17.1 per thousand for the placebo group (non aspirin).
B)94 people in the aspirin group had a heart attack, compared to 171 people in the non-aspirin group.
C)The risk of a heart attack for the aspirin group was 45% of the risk of a heart attack for the non-aspirin group.
D)Any of the above could have been possible.

Question 16

Is it possible for the same relative risk to produce two different conclusions in two different statistical studies? Explain why or why not.(Assume data quality is not an issue.)

Question 17

What two pieces of information do you get from a confidence interval that can help you evaluate whether or not statistically results are really important to you?

Question 18

Explain the difference between the statistical meaning of the word 'significant results' and the regular conversational meaning of the word 'significant results.'

Question 19

What information can a confidence interval provide that a p-value alone cannot?

A)The direction of a difference or relationship, if one is concluded to exist.
B)The amount of variability in the results.
C)The actual size of an effect such as relative risk of heart attacks.
D)All of the above.

Question 20

Explain how the results of a very small study can be misleading.(Assume data quality is not an issue.)

Question 21

Why is it important, when reading the results of a hypothesis test, to determine whether the test was one or two sided?

A)If a test is one sided, and details aren't reported, you could be misled into thinking there is no difference, when in fact there was one in the opposite direction than was hypothesized.
B)If the researchers planned to conduct a two-sided test, then later changed to a one-sided test, you should be very suspicious.
C)It gives you some insight as to what the researchers originally intended to investigate, and/or the direction in which they originally hypothesized the relationship to be.
D)All of the above.

Question 22

When it comes to sample size, "results may be larger (more significant) than they appear" is a good way to describe one of the possible problems with a hypothesis test.Explain how this can happen.

Question 23

The word 'significant' is often used to try to convince you that there is an important effect or relationship.Explain how this word can take on different meanings and how you need to be aware of that when consuming statistical information.

Question 24

Which of the following statements is false?

A)A small enough sample size could miss even the strongest of relationships that exists in a population.
B)Small sample sizes make it harder to rule out the null hypothesis.
C)A test based on a small sample size has less statistical power than a test using a large sample size, even if the percentages, means, and other statistics found in the sample are the same in both situations.
D)None of the above.

Question 25

What is the danger of researchers only reporting on the tests that came out significant (versus reporting on all the tests they conducted)?

A)If nothing interesting is happening in the population, then 1 in 20 tests are expected to achieve statistical significance just by chance.
B)The researchers could have been 'data snooping.'
C)Results that are not statistically significant give valuable information to the public as well; not just those that are statistically significant.
D)All of the above.

Question 26

Suppose a pilot study tells you that there is a great deal of natural variability in the population.How will this impact the design of your actual, full blown study?

A)It will cause you to take a larger sample than you may have originally planned on.
B)It will cause you to abandon the bigger study; you won't be able to detect any differences or relationships even if they did exist.
C)Since there is so much variability, you might as well use a smaller sample size, because smaller sample sizes have more variability as well.
D)The pilot study shouldn't impact your design at all; that would be 'data snooping.'

Question 27

Suppose a researcher conducted a 2-sided hypothesis test for a population mean (significance level .05) and his test statistic was +1.84.(The significance level was .05.) He knows this test statistic is not large enough to reject the null hypothesis, because the test statistic was not greater than 1.96.Then he realizes that if he had done a one-sided test to begin with (alternative hypothesis ">" ) his results would have been statistically significant, because his test statistic was greater than 1.65.At this point, he decides to change to a one-sided hypothesis test, and reports his results as being statistically significant.Is this acceptable? Why or why not.

Question 28

If 100 independent hypothesis tests are conducted using a significance level of 0.10, and if all of the null hypotheses are true, what is the expected number of false positives (type I errors)?

A)0
B)5
C)10
D)There is not enough information to say.

Question 29

Suppose you got a p-value with a small study that was less than .05.Had you gotten the same sample results with a larger study done in a similar way, the p-value from the larger study would have been _______________ (choose: less than, greater than, equal to) the p-value from the smaller study.

Question 30

The simplest and most conservative method for handling type 1 error probabilities in multiple comparisons is the _____________________ method.

Question 31

The natural variability in the population can affect a test's ability to detect a difference or relationship in the population that actually does exist.Explain how this is taken into account in the formula for the test statistic for testing a population mean.

Question 32

Choosing beforehand to conduct a two-sided alternative will make it __________to detect a real difference in one single direction (choose: harder, easier, just as hard or easy).

Question 33

When it comes to sample size, "results may be smaller (less significant) than they appear" is a good way to describe one of the possible problems with a hypothesis test.Explain how this can happen.

Question 34

Imagine you wish to test 4 independent hypotheses so that the total type 1 error probability is 0.10.According to the Bonferroni method, each test should be conducted at what significance level?

A)0.01
B)0.025
C)0.05
D)0.10

Question 35

Which of the following conclusions should make you suspicious as an educated consumer of statistical information?

A)"Based on our sample results we know there is no relationship between these two variables in the population."
B)"We looked at all possible correlations between these 10 variables, and this was the only one that was significant, signifying its tremendous importance."
C)"There is a relationship between these two variables in the population.Our statistics proved it."
D)All of the above.

Question 36

Large samples make it easier to detect real relationships or differences in the population than small samples (assuming everything else is equal).Explain how this is taken into account in the formula for the test statistic for testing a population mean.

Question 37

Suppose you examined a possible relationship between gender and opinion on an issue in a population using two different studies.The percentages found were exactly the same for both studies, but one study had a much larger sample size than the other.Which of the following is not a possible outcome in this scenario?

A)Both studies find the relationship to be statistically significant.
B)Neither study finds the relationship to be statistically significant.
C)The larger study finds a statistically significant relationship but the smaller study does not.
D)The smaller study finds a statistically significant relationship but the larger study does not.

Question 38

Suppose researchers tell you that they did not find a statistically significant relationship between two variables, and the p-value was .84.Which of the following must be true?

A)The sample size must have been small.
B)There must not be a relationship in the population.
C)There must not have been a relationship observed in their sample.
D)None of the above.