Deck 14: Scoring Tests

Question 1

Traditionally, one prominent way to reduce adverse impact when setting cut off scores is to use _______. However, the Civil Rights Act of 1991 mostly prohibits this practice for all practical purposes.

A) subgroup norming
B) contrasting groups
C) banding
D) the Delphi technique

Answer 1

A

Question 2

When using banding for setting standards, if you must exhaust (i.e., select, disqualify, or otherwise remove) all members within a given band before moving on to the next band, this would be considered ________ bands.

A) fixed
B) sliding
C) contrasting
D) exhaustive

Answer 2

A

Question 3

When we set a pass point on a test too low, we are most likely to impact the ______ of the assessment device the most.

A) validity
B) reliability
C) cost
D) utility

Answer 3

D

Question 4

If Benjamin was more concerned with issues such as the cost of testing and the size of the applicant pool when setting his cutoff score on a test, then he would be most focused on ________ issues.

A) legal
B) practical
C) edumetric
D) psychometric

Question 5

If an applicant has to pass one assessment before moving on to the subsequent assessment, then this is typically referred to as _________.

A) multiple tests
B) multiple hurdles
C) wholistic assessment
D) compensatory assessment

Question 6

One procedure mentioned in Module 14 in Shultz, Whitney, and Zickar (2021) that can be used to assess the quality of the SME ratings is __________.

A) fixed bands
B) sliding bands
C) the Delphi technique
D) generalizability theory

Question 7

A standard setting method that primarily uses subject matter expert (SME) judgements, but that also allows for feedback via a moderator summarizing the initial anonymous ratings and sharing them with the raters before they finalize their ratings, thus allowing them to change them based on this feedback, would be the _______ method.

A) Angoff
B) Bookmark
C) Delphi
D) Supplemental

Question 8

If Thomas asks Subject Matter Experts (SMEs) to determine the probability (0%-100%) of a minimally competent person (MCP) answering a given multiple-choice item correctly and then these probabilities are averaged across SMEs for each item and then summed across all items to determine the final cutoff score, then Thomas must be using the _______ method to set cutoff scores.

A) Ebell
B) Angoff
C) Nedelsky
D) Bookmark

Question 9

If Alisha asks Subject Matter Experts (SMEs) to examine each multiple-choice question and decide which alternatives a minimally competent person (MCP) could eliminate (e.g., option D really is not feasible). As a result, for a four-option (one correct response and three distracters) multiple-choice question, the only values possible are 25% if no distracters can be eliminated, 33% if one distracter can be eliminated, 50% if two distracters can be eliminated, or 100% if all three distracters can be eliminated. Again, these ratings (or judgments) are averaged across all SMEs for each question and then summed across items in order to set the pass point. Then Alisha must be using the _______ method to set cutoff scores.

A) Ebell
B) Angoff
C) Nedelsky
D) Bookmark

Question 10

Let's say that Alejandra takes all of the test items on a given measure and orders them from easiest to hardest. Next, the SMEs work their way through the items starting at the easiest and evaluating whether the item is equal to or above a given response probability (e.g., 70%) for the minimally competent person or below that standard. All of the early easy items should be above, so when the SME reaches the item that splits those items equal to or above the response probability and those below the response probability, they place a marker to create the cutoff score. Thus, Alejndra must be using the _______ method to set cutoff scores.

A) Ebell
B) Angoff
C) Nedelsky
D) Bookmark

Question 11

Many of the judgmental methods (e.g., Angoff, Ebel) of setting standard or cutoff scores require reference to a minimally competent person (MCP). How do raters define the MCP in order to determine their cutoff scores?

Question 12

Briefly describe how where we set the cutoff score may affect the validity and/or utility of a test in any given situation?

Question 13

Do we really even need to set cutoff scores? Why not just rank order all the test scores from highest to lowest and provide the valued outcome until it runs out?

Question 14

Briefly explain who can serve as a subject matter expert (SME) for setting cutoff scores under the judgmental methods?

Question 15

What are the major advantages and disadvantages of using a hybrid of judgmental and empirical procedures when determining cutoff scores?