Deck 4: Validity and Item Analysis

Question 1

If the validity evidence focuses on the degree to which the evidence indicates the items, questions, or tasks adequately representing the intended behavior domain, which traditional category of validity would it be most related to?

A)Content-related validity
B)Criterion-related validity
C)Concurrent validity
D)Predictive validity

Answer 1

A

Question 2

A method used in order to examine whether an instrument differentiates between the groups using the approach of decision theory is:

A)Expectancy Tables.
B)Tables of Specification.
C)Coefficient Alpha.
D)regression analysis.

Answer 2

A

Question 3

In order to determine if an instrument can be used appropriately in a certain situation, the counselor must:

A)talk to a clinician who has used the instrument.
B)examine the reliability coefficients.
C)examine the validation evidence.
D)calculate a validity coefficient.

Answer 3

C

Question 4

Standard error of measurement is to standard error of estimate as:

A)concurrent validity is to predictive validity.
B)content validity is to face validity.
C)a reliability coefficient is to a validity coefficient.
D)a normal distribution is to a skewed distribution.

Question 5

The standard error of estimate is an index of the:

A)accuracy of the instrument scores.
B)content validity of a test.
C)reliability of an instrument.
D)probable range of scores on the criterion.

Question 6

Using decision theory, both expectancy tables and group separation help to determine:

A)differences between the standardization group and the population of interest.
B)whether an instrument differentiates between groups.
C)whether or not counselors should decide to use an instrument based on the groups represented in the norming sample.
D)whether or not males and females should be separated during group test administration.

Question 7

In appraisal, regression equations are most frequently used to:

A)predict test-takers' performance on the criterion.
B)compute test-retest reliability.
C)measure developmental changes.
D)analyze the makeup of the norming group.

Question 8

A low, non-significant correlation between an observation-based assessment of symptoms of autism and a teacher-report of ADHD symptoms provides ______________________ evidence of the validity of the autism measure.

A)convergent
B)multimethod
C)discriminant
D)differential

Question 9

The most comprehensive type of validity that incorporates a gradual accumulation of evidence is __________ validity.

A)concurrent
B)content
C)predictive
D)construct

Question 10

When we use Expectancy Tables, hit is to ________ as miss is to _________ .

A)reliability; validity
B)right; wrong
C)probability; possibility
D)high score; low score

Question 11

In the equation Y' = a + b X, the b represents the:

A)intercept or intercept constant.
B)slope or regression coefficients.
C)standard error of estimate.
D)predicted score on the criterion.

Question 12

A high correlation between a self-report inventory that measures depression and a depression measure completed by the therapist is an example of:

A)convergent evidence.
B)discriminant evidence.
C)reliability.
D)content-related evidence.

Question 13

If an instrument measures consistently but does not measure what it was designed to measure, the instrument is:

A)valid but not reliable.
B)reliable but not valid.
C)standardized but not reliable.
D)reliable but not normative.

Question 14

A statistical technique often used to examine internal structure of an instrument is:

A)factor analysis.
B)meta-analysis.
C)internal analysis.
D)structural analysis.

Question 15

The difference between concurrent and predictive validity is whether the:

A)prediction is made in the current context or in the future.
B)focus is on the normative sample or the content of the items.
C)instrument is designed for adults or children.
D)evidence is analyzed using regression or analysis of variance.

Question 16

An example of gathering evidence based on response processes is:

A)comparing scores on one assessment to a similar assessment.
B)asking individual's to "think aloud" while completing an inventory.
C)looking for patterns of answers within an identified assessment.
D)having individuals rate their perceived accuracy of a completed assessment.

Question 17

If an assessment identifies a child as having ADHD when the child does not have ADHD, this would be an example of:

A)a false negative.
B)a false positive.
C)a correct diagnosis.
D)differential diagnosis.

Question 18

Of the three historical types of validation evidence, which one is believed by many experts to be the most pertinent with assessments used by counselors?

A)Content-related
B)Construct-related
C)Criterion
D)Predictive

Question 19

Validity evidence is to ________ as item analysis is to ________.

A)the entire instrument; qualities of each item
B)validity; reliability
C)accuracy; specificity
D)new instrument; old/outdated instrument

Question 20

In evaluating an instrument's validity, practitioners should:

A)always select an instrument that has the highest validity coefficients.
B)be convinced by the preponderance of evidence on the appropriate use of that instrument.
C)not use an instrument unless a factor analysis has been conducted.
D)select those instruments that have a multitrait-multimethod matrix.

Question 21

One fundamental assumption of item response theory is that each item is constructed so that it measures only one trait or ability. This assumption is known as:

A)the one-trait assumption.
B)single ability measurement.
C)unitary attribute theory.
D)unidimensionality.

Question 22

Validity is explained as the gradual accumulation of evidence that indicates how an instrument can be used and what can be inferred from the results.  What does this mean for the use of newly-developed instruments that have not existed long enough for evidence of validity to accumulate?

Question 23

An important parameter in item response theory is the slope of the curve, which is an indicator of how well the item discriminates.  If there is a ________  slope then the item is probably a good discriminator ,whereas a ________ slope probably indicates the item does not discriminate very well.

A)flat; steep
B)symmetrical; asymmetrical
C)steep; flat
D)asymmetrical; symmetrical

Question 24

The higher the item difficulty index (i.e., 0.00 to 1.00), the ________ the item.

A)easier
B)harder
C)more reliable
D)less reliable

Question 25

Briefly explain the basis of item response theory.  How does it differ from classical test theory?  What are some of the benefits and challenges to each of these theories?  Are there times when one would be more appropriate to employ than the other?  What are item characteristic curves, and what information can they provide?

Question 26

Discuss the theoretical movement from considering validity as being three distinct types to considering it as a comprehensive whole.  What are the advantages and disadvantages of each viewpoint?

Question 27

There are many methods of gathering construct validity information.  Briefly discuss each process and its advantages and disadvantages: convergent and discriminant evidence; multitrait-multimethod matrix; factor analysis; meta-analysis; correlational method; regression.

Question 28

The Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999)state that when counselors are considering using an instrument they are responsible for examining the evidence related to the consequences of testing.  How do you understand this statement?

Question 29

The difference between the percentage of examinees in the upper group who answer an item correctly and the percentage of examinees in the lower group who answer the item correctly is called the ____________ index.

A)internal consistency
B)item difficulty
C)item reliability
D)item discrimination