Deck 13: Classical Test Theory Item Analysis

Question 1

Based on Module 13 in Shultz, Whitney, and Zickar (2021), if an item you spent a lot of time constructing performs poorly based on the item analysis statistics, typically your best bet is to _________.

A) simply toss the item out and start over
B) replace the item with a better performing item
C) revise the item based on the item analysis statistics
D) compute additional item analysis statistics to evaluate the item more fully

Answer 1

C

Question 2

From a purely psychometric standpoint of wanting to maximize variability and reliability, the ideal average p-value across items on a norm-referenced test would be _______.

A) 25%
B) 50%
C) 75%
D) 95%

Answer 2

B

Question 3

In practice, most tests in educational and employment contexts should have p-values that range from __________.

A) 25% to 75%
B) 10% to 90%
C) 50% to 90%
D) 0% to 100%

Answer 3

C

Question 4

Under Classical Test Theory (CTT) item analysis, the p-value for a given item is most likely to fluctuate based on ______.

A) who takes the test
B) how often they take the test
C) when the test is administered
D) where the test is administered

Question 5

Most item discrimination indexes correlate how a test taker responds to a given item with their ________.

A) overall GPA
B) total test score
C) score on another test
D) score on an alternate form of the same test

Question 6

The item discrimination index that correlates how a test taker answers (i.e., correct or incorrect) a given test item and their overall test score, while correcting for the fact that you are artificially dichotomizing performance on the item, is known as a ________ correlation coefficient.

A) Biserial
B) Point-biserial
C) Pearson
D) Tetrachoric

Question 7

Often times a multiple choice test item that has a negative item-total correlation may be improved by simply ________.

A) shortening the test
B) adding more alterative response options
C) allowing more time for the test taker to respond to the item
D) replacing a popular non-keyed response with a less attractive response

Question 8

One major drawback when using item-criterion data to construct a test is that the test tends to ___________________.

A) be much longer than usual
B) have low internal consistency reliability
C) be limited to educational testing environments
D) require more subject matter experts (SMEs) to evaluate the test

Question 9

If Tong has been tasked with developing a licensing exam for psychologists in her state, then she would be best off using a(n) __________ test development strategy.

A) ability
B) aptitude
C) norm-referenced
D) criterion-referenced

Question 10

The use of educational technology (e.g., clicker response pads) in large lecture hall classrooms for "real time" evaluation is most associated with ___________ assessment.

A) wholistic
B) preliminary
C) formative
D) summative

Question 11

Briefly describe the advantage of first examining the item difficulty statistic.

Question 12

Briefly describe and distinguish the different types of item discrimination statistics. Which is preferred in most situations?

Question 13

When is it preferable to have a lower average p-value (e.g., 50%) versus a bit higher (e.g., 75 to 80%)?

Question 14

In addition to examining individual item difficulty and discrimination indexes, it is also important to evaluate the overall test performance. What key statistics will help you evaluate the overall test?

Question 15

Shultz, Whitney, and Zickar (2021) note that, "Thus, every attempt should be made to revise an item before it is tossed out". Briefly describe some examples of appropriate ways to revise a test item to improve its perform in future administrations.