Deck 3: Reliability

Question 1

The most significant difficulty with estimating reliability with the alternate or parallel forms procedure is ​

A)calculating the correlation coefficient using two different forms.
B)the effects of remembering specific items in the second testing.
C)developing two sound instruments that are equivalent or parallel. ​
D)finding two similar sets of test takers.

Answer 1

C

Question 2

Standard error of measurement is designed to: ​

A)tell the clinician if an instrument is reliable.
B)provide an estimate of the probable range of scores for an individual.
C)indicate the percentage of error in the reliability coefficient.
D)determine the statistical significance of the reliability coefficient.

Answer 2

B

Question 3

Correlation coefficients range from: ​

A)-1.0 to 1.0.
B)0 to 1.0. ​
C)0 to -1.0. ​
D)-5.0 to 5.0. ​

Answer 3

A

Question 4

According to classical test theory, if the observed variance of a test is 50 and the true variance is 40, what is the estimated reliability of the test? ​

A)0.40
B)0.50 ​
C)0.80 ​
D)0.90

Question 5

The decision to use either the Kuder-Richardson 20 (KR 20)or the Kuder-Richardson 21 (KR 21)is based on whether: ​

A)the method used to determine reliability is the test-retest or the alternate form method.
B)correlation or regression is going to be used. ​
C)the items measure a homogeneous or heterogeneous behavior domain. ​
D)the item format is multiple-choice or true-false. ​

Question 6

In evaluating an instrument's reliability, a counselor should: ​

A)always select the instrument with the highest reliability coefficients.
B)select instruments where coefficient alphas have been calculated.
C)consider how the instrument is going to be used.
D)never use an instrument where the reliability coefficient is less than 0.92.

Question 7

Systematic error (as compared to unsystematic error): ​

A)significantly lowers the reliability of an instrument.
B)insignificantly lowers the reliability of an instrument. ​
C)increases the reliability of an instrument. ​
D)has no effect on the reliability of an instrument. ​

Question 8

The Spearman-Brown formula is used: ​

A)to correct a test-retest reliability coefficient.
B)to correct a split-half reliability coefficient. ​
C)when the items are all of the same difficulty level. ​
D)when the items are of differing levels of difficulty.

Question 9

What is the most appropriate way to determine reliability with Likert scales, where different answers receive different weightings? ​

A)Test-retest
B)Spearman-Brown ​
C)Kuder-Richardson 20 ​
D)Coefficient alpha (or Cronbach's Alpha)​

Question 10

A correlation coefficient is an indicator of: ​

A)the validity of an assessment.
B)the variability of the obtained scores.
C)the relationship between two sets of data.
D)the fluctuation of an individual's score over time.

Question 11

If the reliability coefficient of an instrument is .91, and the standard deviation is 10,  then a client's  score of 59 could be interpreted that 95 percent of the time his/her true score will fall between  _____ and _____ using standard error of measurement ​

A)56; 62
B)53; 65 ​
C)49; 69
D)-3.00; 3.00

Question 12

Which of the following is NOT one of the assumptions that should be met when establishing the reliability of an instrument using the test-retest method? ​

A)Test-retest reliability is only valid when measuring situational traits
B)The characteristic or trait measured should be stable over time ​
C)There should be no differential in practice effect
D)There should be no differential in learning between the test and retest

Question 13

The statistic that represents the percentage of shared variance between two variables is the: ​

A)correlation coefficient.
B)reliability correlation. ​
C)coefficient of determination.
D)coefficient of shared variance.

Question 14

If an instrument requires some professional judgments in scoring, then the manual should also include information on: ​

A)interrater reliability.
B)corrections of the reliability coefficients using the Spearman-Brown formula.
C)both KR 20s and KR 21s. ​
D)test-retest reliability coefficients. ​

Question 15

With a reliability coefficient of .81, using classical test theory we would interpret that the: ​

A)amount of error variance to observed variance is 81 percent.
B)amount of true variance to observed variance is 81 percent.
C)instrument has good enough reliability.
D)instrument's validity coefficient would be 0.81 ² .

Question 16

If we interpreted a reliability coefficient to indicate that 80% of the variance was true variance to observed variance, what would be the ratio of error variance to observed variance? ​

A)0.20
B)0.40 ​
C)0.60 ​
D)0.80

Question 17

For which of the following tests would a test-retest reliability estimate be least appropriate? ​

A)Intelligence tests
B)Tests of  moment-to-moment mood changes ​
C)Academic achievement tests
D)Instruments to measure art aptitude

Question 18

As the reliability of an instrument increases, the standard error of measurement _______. ​

A)decreases
B)increases
C)could either increase or decrease
D)is unaffected

Question 19

When calculating correlations, the most common method used is the: ​

A)Pearson-Product Moment Correlation Coefficient.
B)Correlation Coefficient of Most Common Factors. ​
C)Reliability Coefficient of Correlation. ​
D)Reliability Correlation Coefficient. ​

Question 20

What statistical technique is often used to calculate an estimate of reliability? ​

A)Regression
B)Analysis of variance ​
C)Factor analysis
D)Correlation

Question 21

Can you think of some situations where the typical methods of estimating reliability are not appropriate?  Why should counselors give special consideration when determining reliability in these instances?  How is this done? ​

Question 22

Have you ever taken a test or assessment where you believe reliability issues influenced your performance?  Explain. ​

Question 23

Give three examples of variables that might have a positive correlation, three examples of variables that might have a negative correlation, and three examples of variables that might have no correlation. ​

Question 24

Jennifer took an aptitude test and is interested in whether the score on the verbal aptitude subscale is significantly higher than her mathematical aptitude subscale score.  What statistic would you be interested in to answer her question? ​

A)Standard error of difference
B)Standard error of estimate
C)Standard error of measurement
D)Standard deviation

Question 25

When is name of the meta-analytic method, developed by Vacha-Haase (1998), that explores variability in reliability estimates across studies? ​

A)Universal reliability
B)Cross-validation reliability
C)Generalizability estimation
D)Reliability generalization

Question 26

Why is a competent counselor responsible for more than just looking at reliability coefficients? ​

Question 27

Why do you think the experts in measurement and assessment are suggesting that counselors explain scores using standard error of measurement? ​

Question 28

In generalizability or domain sampling theory, the focus is on: ​

A)using measures of internal consistency.
B)determining the standard error of measurement.
C)identifying specific sources of variation under defined conditions.
D)identifying where an individual's true score would fall.

Question 29

According to the Standards for Educational and Psychological Testing , which statistic should be used for the interpretation of group test scores? ​

A)Standard error of measurement
B)Correlation coefficient
C)Cronbach's alpha
D)Standard error of the group means

Question 30

What makes Item Response Theory different from classical theory?

A)It is a family of theories of methods used to develop instruments.
B)Traditional forms of calculating reliability are appropriate.
C)The constructs being measured are observable. ​
D)The focus of IRT is on error rates.