Deck 8: Test Development

Question 1

A good item on a norm-referenced achievement test is an item that

A) demonstrates that the testtaker has met certain pre-specified criteria.
B) high scorers respond to correctly while low scorer respond to incorrectly.
C) both high and low scorers respond to correctly.
D) low scorers seek clarification regarding the meaning of the question.

Answer 1

B

Question 2

Consider the following sample True/False item: "I am going to ace this course in psychological testing and assessment." Circle TRUE or FALSE according to your own belief.This item is an example of an item that

A) is referred to as binary choice.
B) will, in all likelihood, never actually appear on my test.
C) can only be used when a dichotomous choice can be made without qualification.
D) All of these

Answer 2

D

Question 3

A test developer has created a pool of 30 items and is ready for a test tryout.At a minimum,how many subjects should the test be administered to?

A) 60
B) 120
C) 150
D) 180

Answer 3

C

Question 4

Test developers have at their disposal a number of statistical tools that may be applied when selecting items items for use on a test.In Chapter 8's Meet an Assessment Professional,Dr.Scott Birkeland made reference to two such techniques.One was a measure of item discrimination,and the other was a measure of item

A) reliability.
B) utility.
C) difficulty.
D) variance.

Question 5

As illustrated in the sample item-characteristic curve published in your textbook,the vertical axis on the graph lists the

A) values of the score on the test ranging from 0 to 100.
B) values of the characteristic of the items on a scale of 1 to 10.
C) heteroscedasity of the item curve in values ranging from 0 to infinity.
D) probability of correct response in values ranging from 0 to 1.

Question 6

One of the questions that the developer of a new test must answer is,"How will the test be administered?" The answer to this question may be

A) the test will be individually administered.
B) the test will be group administered.
C) the test will be individually or group administered.
D) None of these

Question 7

Test item writers must keep many considerations in mind.Which of the following is NOT typically one of those considerations?

A) Will the test be administered by a male or a female?
B) Which item format or formats should be employed?
C) How many items should be written in total?
D) What range of content should the items cover?

Question 8

A test item written in a multiple-choice format has three elements.Which of the following is NOT one of those elements?

A) foil
B) stem
C) leaf
D) correct option

Question 9

A test developer is designing a standardized test using a multiple-choice format.The final form of the test will contain 50 items.It would be advisable for the first draft of this test to contain,at least,how many items?

A) 50
B) 100
C) 150
D) 25

Question 10

So-called "smiley face" scales may be used with

A) young children.
B) adolescents who have limited language skills.
C) adults who have limited language skills.
D) All of these

Question 11

Which statement is TRUE regarding test development and testtaker guessing?

A) Methods have been designed to detect guessing.
B) Methods have been designed to statistically correct for guessing.
C) Methods have been designed to minimize the effects of guessing.
D) All of these

Question 12

Using the method of paired comparisons yields

A) nominal level data.
B) ordinal level data.
C) interval level data.
D) ratio level data.

Question 13

One of the advantages of computerized adaptive testing (CAT)is that

A) all test items are administered to all testtakers.
B) floor effects are reduced.
C) the ceiling has been removed.
D) the basement has been finished.

Question 14

A close friend,who is now a beauty school dropout,is heard to complain: "I spent all night studying "Shampoo" for the final examination and there was not a single question on that subject!" As a budding expert in testing and assessment you hear that complaint as:

A) "I have a problem with that test's content validity!"
B) "There was excessive error variance in the test administration procedures!"
C) "The instructor should have paid more attention to the test's construct validity!"
D) "Now I am going to have to reconsider a career as a tanning technician!"

Question 15

Pilot work refers to the

A) job of someone whose responsibility it is to fly an airplane, jet, or space vehicle.
B) preliminary research entailed in finalizing the form of a test.
C) efforts of the lead researcher on a test development team.
D) preliminary research conducted prior to the stage of test construction.

Question 16

An analysis of a test's item may take many forms.Thinking of the descriptions cited in your text,which is NOT one of those forms?

A) item validity analysis
B) item discrimination analysis
C) item tryout analysis
D) item reliability analysis

Question 17

One of the questions that the developer of a new test must answer is,"Should more than one form of the test be developed?" In answering this question,a primary consideration is

A) development costs.
B) test content.
C) test reliability.
D) item discrimination.

Question 18

The inspiration to create a new test may come from many varied sources.Thinking of the illustrative descriptions of inspiration cited in your text,which of the following is NOT a possible source of inspiration for the creation of a new test?

A) an emerging social phenomenon suggests the need for a psychological test
B) legislation has been passed ordering the creation of a new psychological test
C) a review of the literature suggests a need for a new psychological test
D) a test developers thinks "there is a need for this sort of test"

Question 19

The development of a criterion-referenced test usually entails

A) exploratory work with a group of testtakers who have mastered the material.
B) exploratory work with a group of testtakers who have not mastered the material.
C) both (a) and (b)
D) None of these

Question 20

If all raw scores on a test are to be converted to scores that range only from 1 to 9,the resulting scale is referred to as this type of scale:

A) a unidimensional scale
B) a stanine scale
C) a multidimensional scale
D) None of these

Question 21

Item analysis is conducted to evaluate

A) item reliability.
B) item validity.
C) item difficulty.
D) All of these

Question 22

An ADVANTAGE of applying item response theory (IRT)in test development is that

A) the principles underlying IRT make its application easy and appealing.
B) sample sizes used to test the utility of test items can be relatively small.
C) assumptions underlying IRT usage are weak.
D) item statistics are independent of the samples administered the test.

Question 23

Test items that contain alternatives with five points ranging from "strongly agree" to "strongly disagree" are characterized as using this approach to scaling

A) Guttman scaling.
B) Likert scaling.
C) Nielson scaling.
D) opinion scaling.

Question 24

An item bank is

A) a test item not currently in use.
B) the optimum combination of reliability and validity in an item.
C) a set of items from which a test can be constructed.
D) a statistical "Keogh plan" for data relating to high and low scorers on a test.

Question 25

Guttman scales:

A) are typically used with nominal categories.
B) typically are constructed so that agreement with one statement may predict agreement with another statement.
C) typically are constructed so that agreement with one statement should not be correlated with agreement with any other statement.
D) were originally developed by a Peace Corps task force.

Question 26

Ideally,the first draft of a test should include at least how many items as compared with the final version of the test?

A) about twice the number of the final version
B) about half the number of the final version
C) about three times the number of the final version
D) roughly the same number as the final version

Question 27

If 100 people take a test and 20 of those testtakers answer a particular item correctly,then the p value of the item is

A) .25.
B) .20.
C) .40.
D) .04.

Question 28

Item branching refers to

A) administering certain test items on a test depending on the testtakers' responses to previous test items.
B) the creation of alternate and parallel forms of tests based on a group of testtakers' responses to the original test.
C) statistical efforts to ensure that items translated into foreign languages are of the same difficulty.
D) re-using items in an original test that were originally developed for use in a parallel test.

Question 29

Scoring drift refers to

A) the tendency of scorers to give higher scores to testtakers with certain characteristics (such as age and gender) that is similar to themselves.
B) differences between the typical scoring of an item during standardization and subsequent, more authoritative scoring of an item.
C) a gradual decline in inter-scorer reliability after 95% of the examinations have been scored due to scorer fatigue.
D) a flexible method of scoring test items for populations other than that of the standardization sample.

Question 30

Which is an example of the selected-response item format?

A) a multiple-choice item
B) a fill-in-the-blank item
C) Both a and b
D) None of these

Question 31

According to your textbook,the minimum sample for a test tryout is

A) one-half of the number of testtakers in the standardization sample.
B) 25 testtakers.
C) 50 testtakers.
D) 500 testtakers

Question 32

According to the text,which statement is TRUE of scaling?

A) There is only one best approach to scaling and only one best type of scale.
B) Ratio scaling leads to the least scoring drift.
C) Ratio scaling was first developed in the Republic of Samoa.
D) None of these

Question 33

The elements of a multiple-choice item include

A) a stem.
B) a distractor.
C) a foil.
D) All of these

Question 34

Multiple-choice items draw primarily on which testtaker ability?

A) recognition.
B) organization.
C) planning.
D) perceptual-motor skills.

Question 35

A well-written true-false item

A) includes multiple ideas.
B) has a correct response that is either true or false, and not subject to debate.
C) typically contains irrelevant information as a distracter.
D) Both a and b

Question 36

Sorting techniques can be employed to develop

A) nominal scales.
B) ordinal scales.
C) interval scales.
D) All of these

Question 37

The idea for a new test may come from

A) social need.
B) review of the available literature.
C) common sense appeal.
D) All of these

Question 38

An example of a selected-response type of item is

A) a multiple-choice item
B) an essay item
C) a matching item
D) Both a and c

Question 39

An anchor protocol is

A) a previously developed test with known validity that can be used as a comparison for newly developed tests.
B) a statistical procedure in which weights are assigned to each item of a model test to maximize predictive validity.
C) a list of guidelines for a standardized test used to ensure that all testtakers are similar in key ways to the population of the original standardization sample.
D) a model for scoring and a mechanism for resolving scoring discrepancies.

Question 40

With regard to the test tryout phase of test development,

A) test conditions should be as similar to the actual administration as possible.
B) at least 500 subjects should be included to ensure accurate results.
C) the sample used must be nationally representative.
D) All of these

Question 41

An item-characteristic curve includes all of the following EXCEPT

A) information that can be used to judge item bias.
B) information that can be used to judge item fairness.
C) item-discrimination information.
D) item-difficulty information.

Question 42

What is the value of the item-discrimination index for an item answered correctly by an equal number of students in the higher- and lower-scoring groups?

A) -1
B) +1
C) .50
D) 0

Question 43

Which statement is TRUE regarding an item-discrimination index?

A) It has been used by e-Harmony.com and other dating sites for matchmaking.
B) There is more than one formula for calculating an item-discrimination index.
C) Tetrachoric correlation is most frequently used in any formula for an item-discrimination index.
D) All of these.

Question 44

An item-discrimination index typically compares

A) high scorers' performances with low scorers' performances on a particular item.
B) medium scorers' performances with low and high scorers' performances on a particular item.
C) low scorers' performances with lower scorers' performances on a particular item.
D) one group of scorers' performances on the item with any other groups of scorers' performances on the same item.

Question 45

Which statement best describes the relationship between item difficulty and a "good" item?

A) The difficulty level is not a factor in determining a "good" item.
B) An item with a high difficulty level is likely to be "good."
C) An item with a mid-range difficulty level is likely to be "good."
D) An item with a low difficulty level is likely to be "good."

Question 46

A negative item-discrimination index results for a particular item when

A) more high scorers than low scorers on a test get the item correct.
B) more low scorers than high scorers on a test get the item correct.
C) an item is found to be biased and unfair.
D) most testtakers do not enter the response keyed correct for the particular item.

Question 47

The higher the item-difficulty index,the ________ the item.

A) easier
B) harder
C) more robust
D) less robust

Question 48

What is the value of the item-discrimination index for an item that all the students in the higher-scoring group answered correctly but that no one in the lower-scoring group answered correctly?

A) -1
B) +1
C) .50
D) .25

Question 49

An item-endorsement index is most likely to be used in which type of test?

A) a cognitive test
B) an achievement test
C) a vocational aptitude test
D) a personality test

Question 50

The greater the magnitude of the item-discrimination index,the more testtakers in the higher-scoring group answered the item correctly,as compared to testtakers

A) who served as the non-test-taking control group.
B) in the lower-scoring group.
C) who participated in the test standardization.
D) None of these

Question 51

An item-difficulty index can range from

A) 0 to 1
B) .10 to .99
C) .25 to .75
D) 0 to 100

Question 52

What is the optimal item-difficulty level for a true-false item?

A) .500
B) .625
C) .755
D) 1.000

Question 53

The item-validity index is key in determining

A) construct validity.
B) criterion-related validity.
C) content validity.
D) All of these

Question 54

Item-discrimination indexes can range from

A) .001 to 1.00
B) -1 to +1
C) 0% to 100%
D) 1 to 100

Question 55

As a distribution of scores gets flatter,what happens to the optimal boundary line for determining higher- and lower-scoring groups for item-discrimination indices?

A) the optimal boundary line gets smaller
B) the optimal boundary line gets larger
C) the optimal boundary line does not change
D) the optimal boundary line ceases to be optimal

Question 56

To calculate an item-reliability index,one must have previously calculated

A) the correlation between the item score and the criterion.
B) the correlation between the item score and the total score.
C) the item-score standard deviation.
D) All of these

Question 57

An item-difficulty index of 1 occurs when

A) all examinees answer the item incorrectly.
B) all examinees answer the item correctly.
C) examinees are evenly divided between correct and incorrect responses.
D) None of these

Question 58

An item-reliability index provides a measure of a test's

A) test-retest reliability.
B) internal consistency.
C) stability.
D) All of these

Question 59

In item analysis,the term item endorsement refers to the percent of testtakers who

A) responded correctly to a particular item.
B) indicate that they agree with a particular item.
C) passed the item on a pass/fail test of ability.
D) consented to answer an optional item.

Question 60

It is needed to calculate the item-validity index.It is

A) the point-biserial correlation between the item score and the criterion score.
B) the mean of the item-score distribution.
C) the item-score standard deviation.
D) All of these

Question 61

In general,what can be said about an item analysis of a speeded test?

A) Results are often misleading and difficult to interpret.
B) Item-difficulty levels are higher toward the end of the test.
C) Item-discrimination levels are higher for later items.
D) All of these

Question 62

The best type of item yields an item-characteristic curve that

A) has a positive slope.
B) has a negative slope.
C) is leptokurtic.
D) has few, if any, outliers.

Question 63

Which is TRUE of cross-validation of a test after standardization has occurred?

A) Cross-validation creates confusion regarding the meaning of the original standardization data.
B) The cross-validation sample is composed of the same testtakers that participated in the original test standardization.
C) Cross-validation often results in validity shrinkage.
D) All of these

Question 64

Which statement is TRUE of guessing?

A) It occurs more often on achievement than personality tests.
B) It posts methodological problems for the testtaker.
C) Most testtakers guess based on little knowledge of the subject matter.
D) It poses methodological problems for the test developer.

Question 65

Ideally,psychological or educational tests are revised

A) every decade
B) when the test is no longer useful.
C) as a function of annual test sales
D) None of these

Question 66

All of the following are methods of evaluating item bias EXCEPT

A) noting differences between the item-characteristic curves.
B) noting differences in the item-difficulty levels.
C) noting differences in item-discrimination indexes.
D) noting differences in validity shrinkage.

Question 67

During the norming of a new intelligence test,a test publisher administers to all of the testtakers not only the new intelligence test,but a vision test using an eye chart.The publisher has engaged in

A) test conceptualization
B) cross-validation.
C) shared validation
D) None of these

Question 68

Generous time limits are typically associated with

A) speeded conditions.
B) power conditions.
C) untimed conditions.
D) hazardous conditions.

Question 69

Ability tests are typically standardized on a sample that is representative of the general population and selected on the basis of variables such as

A) age.
B) gender.
C) geographic region.
D) All of these

Question 70

A student makes the following complaint after taking an exam: "I spent all night studying Chapter 7 and there wasn't even one test question from that chapter!" From a psychometric perspective,this student is concerned about the exam's

A) error variance.
B) test-retest reliability.
C) rater error.
D) None of these

Question 71

As part of the test development process,a test revision may entail

A) re-wording, deletion, or development of new items.
B) development of a new edition of a test.
C) the reprinting of a test.
D) Both a and b

Question 72

A test manual for a commercially prepared test should ideally include

A) a description of the test development procedures used.
B) test-retest reliability data.
C) internal-consistency reliability data.
D) All of these

Question 73

Which is TRUE with regard to latent-trait models?

A) The latent trait is multidimensional.
B) The latent trait is unidimensional.
C) The latent trait cannot be measured by traditional models.
D) The latent trait surfaces before age 12.

Question 74

Co-validation is:

A) highly recommended and encouraged by test professionals.
B) also referred to as co-norming.
C) a strategy that can save time and money for the test publisher.
D) Both b and c

Question 75

With regard to the test revision process,it typically

A) takes about one year to complete.
B) includes all of the steps that the initial test development included.
C) is much less expensive than the original development of a test.
D) All of these

Question 76

A student complains that a midterm examination did not include items from a particular in-class lecture.From a psychometric perspective,the students is expressing concern about the midterm's

A) test-retest reliability.
B) internal consistency reliability.
C) content validity.
D) cross-validation.

Question 77

A student raises concern that a professor has given different grades to two essay answers that are very similar.From a psychometric perspective,the student is expressing concerns about

A) criterion-related validity.
B) rater error.
C) test-retest reliability.
D) parallel forms reliability.

Question 78

Which of the following conditions may lead to the decision to revise a psychological or educational test?

A) item content, including the vocabulary used in instructions and pictures, has become dated
B) test norms no longer represent the population for which the test is designed
C) reliability and validity of a test can be improved by a revision
D) All of these

Question 79

The term used to describe the decrease in item validities that typically occurs during cross-validation is

A) validity detriment.
B) validity decrement.
C) validity shrinkage.
D) cross-validation devaluation.

Question 80

Which is TRUE of item-characteristic curves?

A) They determine which items are fair.
B) They may be used as an aid in assessing whether or not items are biased.
C) They determine which items are most reliable under specified conditions.
D) They may be used as an aid in determining the kurtosis of a distribution of test scores.