Deck 6: Developing Your Research Protocol

Question 1

Outline three different types of constraints on your research design and provide an example for each.

Answer 1

Money constraints: If you have limited amounts of funding, that limits how much you can pay participants, and subsequently how many participants you can recruit in total which could influence your power.
Equipment constraints: If you don't have access to certain equipment, such as brain imaging technology, you are limited to not asking questions that involve them.
Participant constraints: If you do not have access to certain types of population, such as children, you cannot address questions that directly involve these populations.

Question 2

Distinguish between the four types of measurement scales and provide an example of each.

Answer 2

Nominal scales are categorical measures, such as political party.
Ordinal scales have rank ordered properties, such as placement in a race.
Interval scales have rank ordered properties and equal intervals between ranks, such as temperature in Fahrenheit or Celsius.
Ratio scales have rank ordered properties, equal intervals, and an absolute zero, such as height or weight.

Question 3

Explain what statistical power is, identify two types of power analyses and why you might choose one over the other.

Answer 3

Statistical power refers to the ability to detect an effect, given that an effect actually exists. The two types of power analyses are prospective and retrospective. A prospective power analyses is preferred and great to help calculate the number of participants needed to obtain a certain level of power, given a certain effect size. You would use a prospective power analysis over a retrospective analysis in order to calculate the number of participants you would need to reach a certain level of power, given an estimated effect size.
A retrospective power analysis is useful to calculate observed power in a study, which is the actual level of power a study had. This is generally preferred over prospective power analyses when it is difficult to estimate an effect size before conducting the study. When conducting a retrospective power analysis, it is important to realize that your observed effect size can differ from the expected effect size, which would then influences how much power you have. One would use a retrospective power analysis to identify if power was higher or lower than expected.

Question 4

Identify three different sampling methods. Explain when you would use each one.

Question 5

Name four steps in designing your research protocol, and discuss the goals of each step.

Question 6

Many psychology departments in the United States have subject pools. Undergraduate students enrolled in introduction to psychology are often awarded extra credit or sometimes even required to complete research studies in order to earn points towards their class. The issue with relying on WEIRD samples through a subject pool in psychological research is most likely a result of what kind of sampling?

A) Representative sampling
B) Random sampling
C) Snowball sampling
D) Convenience sampling

Question 7

Which of the following type of sampling is most likely not captured by using Mturk participants in online sampling?

A) Convenience sampling
B) Self-selection
C) Non-probability sampling
D) Snowball sampling

Question 8

Researchers often select one type of measurement scale (e.g., ratio vs. ordinal) over another. What is not a reason for this?

A) Statistical power
B) The topic or subject of interest
C) The target population
D) The ability to have stronger validity

Question 9

With respect to power, it is the probably of detecting an effect when…

A) You are applying for a grant.
B) You are studying vulnerable populations.
C) An effect is small.
D) An effect exists.

Question 10

The benefit of a large effect is that…

A) it requires a less representative sample to be detected.
B) it requires less power be detected.
C) it requires more power but fewer participants to be detected.
D) it requires lower power but more participants to be detected.

Question 11

Which of the following pieces of information can only be detected using a retrospective power analysis, and not in a prospective power analysis?

A) If your sample size is large enough.
B) If the effect size is smaller than theorized.
C) If the effect is actually real.
D) If the findings are statistically significant.

Question 12

Jason is conducting research on a population with extreme income and education discrepancies. He hopes to achieve a random, but generalizable sample so that he is able to make inferences about the entire SES spectrum. Which of the following is the best option?

A) Utilize a random sampling strategy because this will ensure that he captures the full range.
B) Utilize a stratified random sampling strategy because this will ensure that he captures the full range.
C) Utilize a non-probability sampling strategy to try to randomly obtain the participants of interest.
D) Utilize a convenience sampling strategy because inferences made along the SES spectrum do not require the full range of SES values.

Question 13

Jenny is an undergraduate student with a full course load studying the relationship between exercise and self-esteem. She is interested in learning whether self-esteem differs between people who exercise regularly and those who do not. She is also interested in the types of exercise, intensity of exercise, and how these differences affect various aspects of self-esteem such as self-efficacy, confidence, and self-worth. She needs to complete data collection within a semester by herself. However, she needs 250 participants for sufficient power. Which of the following would be the best strategy for Jenny to take to complete her study?

A) Recruit 250 participants from the Introduction to Psychology subject pool and run each of them individually in hour-long interviews asking about self-esteem and exercise habits.
B) Print and handout 500 surveys on campus and analyze the data from the first 250 students who return the survey.
C) Upload a survey on self-esteem and exercise habits and recruit 250 students from the Introduction to Psychology subject pool to do this survey online in their own time.
D) Reduce the number of participants to 60 and interview 60 students approached randomly on campus.

Question 14

Which of the following describes stratified random sampling?

A) Dividing the population into homogenous groups along a particular dimension and randomly sampling from each group.
B) Recruiting more participants from underrepresented groups to ensure that they are represented sufficiently in the data.
C) Asking ten participants to send a survey to ten randomly selected friends.
D) Every member of the population having an equal chance of being sampled.

Question 15

What does it mean for a research study to use a random sample?

A) The survey participants are systematically selected on random traits.
B) The survey participants are randomly chosen from the population.
C) The survey participants are not part of the population of interest.
D) The survey participants are self-selected.

Question 16

Researchers often select one type of measurement scale (e.g., ratio vs. ordinal) over another. Which of the following is generally not a reason that influences this decision?

A) What your analysis plans are.
B) The topic of subject of interest.
C) The target population of interest.
D) Concerns related to validity.

Question 17

With regards to the use of WEIRD samples, what has been proposed as a valid argument for their use?

A) WEIRD samples are convenient, and regardless if they are representative of the true population of adults, we should conduct the research with this type of sample and it is highly likely that the research will result in findings generalizable to the rest of the population
B) If the construct and/or measures of interest have been validated across samples of different populations, then it should be okay to use these measures with a WEIRD sample.
C) WEIRD samples should be used as a comparison or control group for all cross cultural research because they represent a distinct culture as a baseline.
D) WEIRD samples provide a good basis for validating survey measures because they are convenient. This allows researchers to then use the results from this sample and make generalizations and inferences to the general population.

Question 18

Which of the following is not true about p-hacking?

A) It increases your chances of obtaining statistically significant findings.
B) It involves manipulating factors in your research, such as dropping specific cases or adding analyses that were not initially planned.
C) It sometimes comes about when researchers engage in data exploration.
D) It is generally acceptable as long as researchers report honestly the steps they took to p-hack.

Question 19

Which of the following statements regarding scales of measurement is false?

A) From a statistical sense, ordinal and nominal scales produce data that are usually less powerful than interval and ratio scales do.
B) Ordinal and nominal scales produce data that are always less valid than interval and ratio scales do.
C) Ordinal and nominal scales are analyzed using non-parametric statistical analyses whereas interval and ratio data are analyzed using parametric statistical analyses.
D) Interval and ratio scales always contain properties of ordinal scales.

Question 20

Ordinal scales are often best analyzed with _______________ tests, even though they are sometimes treated as _________________ in the case of Likert scales. Nominal scales are best analyzed with ________________ tests.

A) non-parametric; interval scales; parametric
B) parametric; nominal scales; non-parametric
C) non-parametric; interval scales; non-parametric
D) non-parametric; nominal scales; parametric

Question 21

What is the main reason why experiments in psychology don't usually involve the entire population of interest?

A) It is impractical to study the entire population.
B) Researchers often have difficulty defining their population.
C) Statistical methods only deal with samples and are not equipped to analyze data from entire populations.
D) Since statistical methods allow us to draw inferences about a population from our sample, it is unethical to study the entire population.

Question 22

When is a prospective power analysis conducted?

A) Before recruiting participants.
B) Before data analysis.
C) While reporting the initial findings.
D) While collecting data from participants.

Question 23

Which of the following is not a valid criticism of the use of WEIRD participants?

A) The sample might not be representative of the general population.
B) It is a method of random sampling and thus results in biased sampling.
C) It is often due to convenient sampling and is subject to self-selection.
D) Findings using these participants might not generalize to other non-WEIRD participants.

Question 24

Which of the following is the best alternative to random sampling to achieve a representative sample?

A) Oversampling
B) Snowball sampling
C) Convenient sampling
D) Online sampling

Question 25

Research on statistical power in existing research studies has demonstrated that…

A) power should not be a concern since most studies have sufficient power.
B) although earlier studies have suggested studies were low in power, it is no longer an issue as researchers have become more aware of its necessity and conduct power analyses regularly.
C) power continues to be an area of concern as many as half of studies in prominent journals are underpowered.
D) it is extremely confusing why researchers do not care about power because it is very easily achievable.

Question 26

Which of the following describes the best practices with regards to data exploration?

A) It is never a good thing.
B) It is always a good thing.
C) While some cases can be bad practice, fully disclosing the steps you took in your analyses is always good.
D) As long as you describe your steps, you can explore and manipulate our data in anyway you like.

Question 27

Which of the following is not part of formulating a protocol and plan for your research?

A) Considering how to obtain a sample.
B) Choosing your measures.
C) Conducting a power analysis.
D) Analyzing your data.

Question 28

Which of the following is not true with regards to random sampling?

A) It can be impractical to use.
B) It is a great method to obtain a representative sample.
C) One example of true random sampling is stratified random sampling.
D) All of the above statements are true.

Question 29

Which of the following differentiates ratio from interval scales?

A) Interval scales have equal intervals between units and ratio scales do not.
B) Interval scales have equal intervals between units and ratio scales are rank ordered but do not have equal intervals between units.
C) Interval scales have true zeros and ratio scales do not.
D) Interval scales do not have true zeros and ratio scales do.

Question 30

Which of the following is a reason why researchers might differ on whether or not they think it is important to formulate and stick with an analysis plan?

A) Some think that it is more important to quickly get to the research as soon as possible and doing so is a waste of time.
B) This is a disagreement stemming from attitudes toward data exploration and researchers disagree on the extent to which it is acceptable.
C) The analyses can't be predicted in most research. Therefore researchers disagree on whether or not predicting what analysis you need to do is a good idea.
D) Some think that formulating research plans is too costly and want to reserve financial resources for other parts of research.

Question 31

Your population generally does not influence the type of scale you use.

Question 32

It is generally a good idea to pay participants as much as you can afford because then they feel incentivized to participate and stay in your study.

Question 33

Conducting a prospective power analysis is considered as data exploration and is often considered questionable practice.

Question 34

Data measured on an ordinal scale have equal intervals between their ranks.

Question 35

Choosing your measures is often the most difficult task in designing a research protocol because you always need to make up your measures from scratch.

Question 36

Nominal scales are not preferred because there are no statistical tests that can be used on them.

Question 37

Even with technological advances, there are still no perfect measurements free of measurement error.

Question 38

Snowball sampling is more likely to lead to a representative sample than stratified random sampling.

Question 39

If you do not have an expected effect size for your research, it can be difficult to conduct a prospective power analysis.

Question 40

A reliable scale should yield different results if tested multiple times on the same person.

Question 41

Statistical power allows researchers to detect true effects if they exist.

Question 42

To use random sampling correctly, the researcher needs access to the entire population of interest.

Question 43

Convenience sampling is an alternative to random sampling that still produces generalizable results to the societal population at large.

Question 44

In the history of psychological science, power has always been over emphasized, partly because underpowered studies never get published.