Deck 4: Validity and Item Analysis

Question 1

Which of the following would not be considered an example of best practice for gathering evidence about the validity of a test?

A)Looking over the assessment questions to determine if the test is valid
B)Evaluating the validation evidence in the test manual to determine if there is sufficient evidence for its use
C)Questioning examinees about their performance strategies and/or responses to items
D)Analyzing the internal structure of an instrument using factor analysis

Answer 1

Looking over the assessment questions to determine if the test is valid

Question 2

If an instrument measures consistently but does not measure what it was designed to measure, the instrument is:

A)valid but not reliable.
B)reliable but not valid.
C)standardized but not reliable.
D)reliable but not normative.

Answer 2

reliable but not valid.

Question 3

In using an assessment with subscales, a counselor should examine:

A)the validation evidence regarding the internal structure of the instrument with population that correspond to the counselor's clients
B)the standard error of estimate for each subscale and only use subscales where the standard error of estimate are close to 1.00
C)the expectancy tables for each subscale
D)whether there is an equal number of items for each subscale as this is considered best practice in assessment

Answer 3

the validation evidence regarding the internal structure of the instrument with population that correspond to the counselor's clients

Question 4

The difference between concurrent and predictive validity is whether the:

A)prediction is made in the current context or in the future.
B)focus is on the normative sample or the content of the items.
C)instrument is designed for adults or children.
D)evidence is analyzed using regression or analysis of variance.

Question 5

A multitrait-multimethod matrix involves:

A)correlating the instrument with other measures that are theoretically-related.
B)correlating the instrument with other measures that are not theoretically-related.
C)correlating the instrument with other measures that use the same and different assessment methods.
D)all of the above.

Question 6

A high correlation between a self-report inventory that measures depression and a depression measure completed by the therapist is an example of:

A)convergent evidence
B)discriminant evidence
C)reliability
D)content-related evidence

Question 7

Validation evidence based on instrument content is related to the concept of:

A)criterion-related validity
B)reliability
C)content validity
D)none of the above

Question 8

A low, non-significant correlation between an observation-based assessment of symptoms of autism and a teacher-report of ADHD symptoms provides ______________________ of the validity of the autism measure.

A)convergent evidence
B)multimethod evidence
C)discriminant evidence
D)differential evidence

Question 9

It is necessary for all good tests and instruments to:

A)be reliable and valid.
B)have appropriate national norms.
C)have a normalized distribution of scores.
D)be reviewed in the Mental Measurements Yearbook.

Question 10

A statistical technique often used to examine internal structure of an instrument is:

A)factor analysis
B)meta-analysis
C)internal analysis
D)structural analysis

Question 11

In examining validation evidence, Messick suggests that clinicians should consider the meaning of the score and the :

A)representativeness of the norming group
B)social value or consequence of using the instrument
C)magnitude of the validity coefficients
D)construct's ability to predict accurately

Question 12

An instructor announces that an examination will cover the topics of counseling theories and techniques.You spend hours studying both concepts, but all of the test questions are only on techniques.You might correctly conclude that the examination:

A)lacked content validity
B)lacked criterion-related validity
C)was not reliable
D)all of the above

Question 13

Test specifications:

A)concern the organizational framework used to development an instrument.
B)include the process of identifying assessment goals.
C)address how objectives were developed.
D)all of the above

Question 14

A validity coefficient is best illustrated by the correlation between:

A)test and retest scores.
B)two different tests.
C)test scores and performance on a criterion.
D)an item and the total test score.

Question 15

If the validity evidence focuses on the degree to which the evidence indicates the items, questions, or tasks adequately represent the intended behavior domain, which traditional category of validity would it be most related to?

A)content-related validity
B)criterion-related validity
C)concurrent validity
D)predictive validity

Question 16

The most comprehensive type of validity that incorporates a gradual accumulation of evidence is __________ validity.

A)concurrent
B)content
C)predictive
D)construct

Question 17

An example of gathering evidence based on response processes is:

A)comparing scores on one assessment to a similar assessment.
B)asking individual's to "think aloud" while completing an inventory.
C)looking for patterns of answers within an identified assessment.
D)having individuals rate their perceived accuracy of a completed assessment.

Question 18

In evaluating an instrument's validity, practitioners should:

A)always select an instrument that has the highest validity coefficients.
B)be convinced by the preponderance of evidence on the appropriate use of that instrument.
C)not use an instrument unless a factor analysis has been conducted.
D)select those instruments that have a multitrait-multimethod matrix.

Question 19

In order to determine if an instrument can be used appropriately in a certain situation, the counselor must:

A)talk to a clinician who has used the instrument.
B)examine the reliability coefficients.
C)examine the validation evidence.
D)calculate a validity coefficient.

Question 20

Of the three historical types of validation evidence, which one is believed by many experts to be the most pertinent with assessments used by counselors?

A)Content-related
B)Construct-related
C)Criterion
D)Predictive

Question 21

The higher the item difficulty index , the ________ the item.

A)easier
B)harder
C)more reliable
D)less reliable

Question 22

If an assessment identifies a child as having ADHD when the child does not have ADHD, this would be an example of :

A)a false negative
B)a false positive
C)a correct diagnosis
D)differential diagnosis

Question 23

An important parameter in item response theory is the slope of the curve, which is an indicator of how well the item discriminates.If there is a ________ slope then the item is probably a good discriminator ,whereas a ________ slope probably indicates the item does not discriminate very well.

A)flat; steep
B)symmetrical; asymmetrical
C)steep; flat
D)asymmetrical; symmetrical Discussion Questions

Question 24

An example of a false negative is when an instrument:

A)identifies a client as not being suicidal when the client is
B)correctly identifies a client as being suicidal
C)identifies a client as being suicidal when the client is not
D)correctly identifies a client as not being suicidal

Question 25

The difference between the percentage of examinees in the upper group who answer an item correctly and the percentage of examinees in the lower group who answer the item correctly is called the ____________ index.

A)internal consistency
B)item difficulty
C)item reliability
D)item discrimination

Question 26

When considering evidence of validity and associated social consequences of instruments counselors should consider all of the following except for:

A)use.
B)interpretation.
C)cost effectiveness.
D)social implications.

Question 27

Using decision theory, both expectancy tables and group separation help to determine:

A)differences between the standardization group and the population of interest
B)whether an instrument differentiates between groups
C)whether or not counselors should decide to use an instrument based on the groups represented in the norming sample
D)whether or not males and females should be separated during group test administration

Question 28

Standard error of measurement is to standard error of estimate as:

A)concurrent validity is to predictive validity
B)content validity is to face validity
C)a reliability coefficient is to a validity coefficient
D)a normal distribution is to a skewed distribution

Question 29

In using regression to predict an individual's performance on the criterion based on his or her score on the instrument, the prediction is least accurate when the correlation is:

A)-1.00
B)0.00
C).50
D)1.00

Question 30

A method used in order to examine whether an instrument differentiates between the groups using the approach of decision theory is:

A)Expectancy Tables
B)Tables of Specification
C)Coefficient Alpha
D)regression analysis

Question 31

If an instructor in an appraisal in counseling course was interested if an exam question was too easy or too difficult, she would want to examine the ________ of that question.

A)internal consistency
B)item difficulty
C)item discrimination
D)item variance

Question 32

An Expectancy Table is a useful device for determining the:

A)correlation between a selection test and a criterion.
B)proportion of error that is expected in any prediction.
C)cutoff score of a selection test.
D)possible range of an individual's score on the criterion.

Question 33

When considering the criterion-related evidence of validity, one should be concerned with whether the criterion is:

A)reliable
B)unbiased
C)free from criterion contamination
D)all of the above

Question 34

A statistical procedure used to determine the generalizability of an instrument's validity is:

A)generalizable analysis.
B)factor analysis.
C)regression analysis
D)meta-analysis.

Question 35

The standard error of estimate is an index of the:

A)accuracy of the instrument scores.
B)content validity of a test.
C)reliability of an instrument.
D)probable range of scores on the criterion.

Question 36

In the equation Y' = a + bX, the b represents the:

A)intercept or intercept constant
B)slope or regression coefficients
C)standard error of estimate
D)predicted score on the criterion

Question 37

An item difficulty index provides:

A)the proportion of individuals who got an item incorrect
B)the number of individuals who got an item incorrect
C)the proportion of individuals who got an item correct
D)the number of individuals who got an item correct

Question 38

When we use Expectancy Tables, hit is to ________ as miss is to _________ .

A)reliability; validity
B)right; wrong
C)probability; possibility
D)high score; low score

Question 39

In appraisal, regression equations are most frequently used to:

A)predict test-takers' performance on the criterion.
B)compute test-retest reliability.
C)measure developmental changes.
D)analyze the makeup of the norming group.

Question 40

Validity evidence is to ________ as item analysis is to ________.

A)the entire instrument; qualities of each item
B)validity; reliability
C)accuracy; specificity
D)new instrument; old/outdated instrument