Deck 7: Evaluation

Question 1

Evaluation is the process of careful appraisal and study to determine several aspects of something that is of interest. These aspects include:

A) significance
B) size
C) shape
D) personality
E) temperature

Answer 1

significance

Question 2

While an information system is developed, the evaluation of it should happen:

A) in planning stage
B) in entire life cycle
C) in analysis stage
D) in design stage
E) in implementation state

Answer 2

in entire life cycle

Question 3

Which of the following is a topic in Development Evaluations of an information system?

A) organizational changes after the system use
B) new employees recruiting
C) system functionality
D) enterprise resources distribution
E) work flow design for developing the system

Answer 3

system functionality

Question 4

Usability testing should be conducted in:

A) cost-benefit evaluations
B) profit evaluations
C) use and impact evaluations
D) development evaluations
E) user characteristics evaluations

Question 5

A primary purpose of the Use and Impact Evaluation of an information system is:

A) deciding functionality of the system
B) testing usability of the system
C) understanding user needs with the system
D) developing the module of the system
E) understanding system impacts on users' activities

Question 6

In the development lifecycle of an interactive system, when should the Use and Impact Evaluation be conducted?

A) system planning stage
B) system analysis stage
C) system design stage
D) system use stage
E) system abandoning stage

Question 7

Which of the following is a characteristic of the Use and Impact Evaluation of an information system?

A) It is conducted in the trial period of the system before release;
B) Its data are collected from HCI experts during the development phases;
C) It is taken in a strictly controlled experimental environment;
D) It is conducted on a paper-based prototype of the system;
E) It aims at finding out the influences on users and the organization from the use of this system.

Question 8

Which of the following could be a motivation for a user to interact with an information system (e.g. a computer, the Internet, a mobile phone, etc):

A) a tremendous change in weather
B) a leak in the kitchen basin
C) a bird singing on a tree
D) a personal meeting with colleagues
E) a need to send a message to family members

Question 9

Which of the following user needs should be considered when developing HCI support for an information system?

A) users' affective factors
B) organizational budget
C) formation of the system development team
D) location of system development team members
E) mission of an organization

Question 10

The uncomfortableness in users' eyes caused by too bright screen is an example of ___ concern in HCI.

A) protective
B) physical
C) psychological
D) cognitive
E) affective

Question 11

Within an organizational context, an example of the extrinsic motivation of using an information system could be:

A) a problem with the fridge at an employee's home
B) a love story movie broadcasted in a TV channel
C) the perceived usefulness of using this system to improve work performance
D) a product advertised on a poster
E) the problem that an employee has because of long-time computer use

Question 12

In the development of an information system, when should formative evaluation be conducted?

A) before the system is developed
B) when the system is being developed
C) when the system is finished in design
D) after the system is released for real use
E) throughout the system development life cycle

Question 13

The purpose of doing formative evaluation in a system development is to:

A) assess the "return on the investment" from implementing the system
B) examine the market share of this system
C) form or influence design decisions
D) ensure that every developer has done his/her job
E) check the system's impacts on users in real use

Question 14

Which of the following should be examined in summative evaluation of an information system?

A) the configuration of the system equipments
B) the schedule for developing the system
C) the rate of progress in developing the system
D) whether the system meets predefined requirements
E) whether the organizational operation is affected by the use of this system

Question 15

The summative evaluation in a system development happens:

A) in the real use of the system
B) when the system is being analyzed for requirements
C) after users feedback in real use is available
D) when the budget for developing this system is not approved
E) when the system is ready in design before release

Question 16

For a longitudinal evaluation of an information system, the data collection is:

A) a one-time process
B) a repeated process
C) on the measure of just one variable
D) always on the measures of multiple variables
E) a snapshot-based approach

Question 17

Which is an appropriate statement about longitudinal evaluation ?

A) Longitudinal evaluation examines just one subject over time;
B) Longitudinal evaluation is a one-time process that happens on a set of subjects;
C) Longitudinal evaluation of an information system has the ability to reveal its dynamic natures;
D) The data collected in longitudinal evaluation of an information system only reflect the current state of the system;
E) Longitudinal evaluation is the least expensive one among all system evaluation methods.

Question 18

In the following statements about formative and summative evaluation, which one is correct?

A) Formative evaluation influences the design of the target system;
B) Summative evaluation influences the design of the target system;
C) Formative evaluation is conducted during the actual use of the product by real users in real context;
D) Summative evaluation is conducted during the development of the product
E) Formative and summative evaluations both influence the design of the target system

Question 19

The usability of a system refers to its quality with respect to aspects such as:

A) ease of purchasing
B) ease of learning
C) ease of selling
D) ease of complaining
E) ease of becoming outdated

Question 20

The usability of a system is a concept that should be discussed within given context, which depends on factors such as:

A) specified budget
B) specified power
C) specified users
D) specified color
E) specified music

Question 21

According to Jakob Nielsen, usability is a multi-dimensional property of a system. One of these multiple attributes of usability is:

A) utility
B) learnability
C) acceptability
D) usefulness
E) reliability

Question 22

Usefulness of an information system is an issue about:

A) whether the system can be used to achieve some desired goal;
B) whether the functionality of this system in principle can do what is needed;
C) how well users can use the system's functionality;
D) whether the system is easy to learn;
E) whether the system is easy to use.

Question 23

Which of the following is an example characteristic that an information system with good usefulness could have?

A) the interface of this system is aesthetically pleasing
B) the interface of this system has audible features
C) the system can help users do a task that was not possible without it
D) the system is safe to use
E) the system is fun to use

Question 24

Usability engineering is a process that

A) is similar to physical engineering
B) is similar to cognitive engineering
C) is similar to affective engineering
D) happens only in the early stage of a system's development
E) happens throughout a system's development

Question 25

In usability engineering, what is the task that should be done throughout a system's development process?

A) specifying user characteristics
B) specifying usability characteristics
C) specifying tasks that will be performed with the system
D) measuring the system's usability characteristics
E) measuring the profit of a company within a specific period

Question 26

Universal usability is a usability issue that pertains to:

A) the price-influencing factors on the market
B) the global weather change
C) employing cheaper labors for information systems development
D) global collaboration in information systems development
E) the diverse user population of an information system across the world

Question 27

Which of the following is a question to be addressed in dealing with universal usability of an information system?

A) What are the cultural differences exist in all potential users of the system?
B) What is the deadline that the system should be released on the market?
C) How should the equipments of the system be delivered to various locations?
D) How the advertisements for this system should be designed for different markets?
E) How the pricing policy should be developed for this system in different markets?

Question 28

In the following evaluation methods for information systems, which one requires data from users?

A) guideline review
B) field experiment
C) heuristic evaluation
D) cognitive walkthrough
E) TSSL-guided evaluation

Question 29

Field experiment is an evaluation method that:

A) is guided by a set of higher level design principles and evaluate whether interface elements conform to the principles;
B) is analytical instead of empirical in nature
C) draws conclusion based on empirical data;
D) is performed through guideline review;
E) is utilized in settings concocted for research purpose.

Question 30

In the following statements about analytical and empirical evaluations, which one is correct?

A) Analytical evaluations draw conclusions based on field data
B) Empirical evaluations only rely on quantitative data to draw conclusions
C) Evidence collected from evaluators is a must-have for analytical evaluations
D) Evidence collected from users is required for empirical evaluations
E) Empirical evaluations only rely on qualitative data to draw conclusions

Question 31

Heuristic evaluation method has the characteristic of:

A) applicable only in late stages of a system design
B) having deep learning curve for evaluators
C) high cost
D) requiring professional evaluators in implementation
E) easy to apply

Question 32

As a heuristic evaluation method, the HOMERUN heuristics are good for the evaluation of usability of:

A) documentations of hardware
B) documentations of software
C) commercial websites
D) hard copy manuals
E) online copy manuals

Question 33

Guideline review is a ___ method to evaluate information systems.

A) analytical
B) empirical
C) experimental
D) field
E) real-user-based

Question 34

Guideline review requires participation of __ in its implementation.

A) executives of an organization
B) system designers
C) payroll personnel
D) office assistants
E) the marketing department

Question 35

Cognitive walk-through is a system evaluation method that:

A) lets users recall their problem-solving steps in using the system
B) defines the cognitive steps that users should following in using the system
C) simulates the steps of a user's problem-solving process
D) invites users to walk through all the offices designated for developing the system
E) relies on users' cognitive responses after doing designed tasks

Question 36

Which of the following description about cognitive walk-through is appropriate?

A) Cognitive walk-through has a broad coverage of tasks supported by an information system;
B) Cognitive walk-through requires input from real users
C) Cognitive walk-through is usually implemented when the design of a system is finished;
D) Before doing cognitive walk-through, assumptions about the user population of an information system should be defined;
E) Cognitive walk-through is good for identifying general problems instead of detailed ones in the use of a system.

Question 37

Focus Group is a form of:

A) questionnaire
B) field observation
C) experiment
D) survey
E) interview

Question 38

When the focus group method is utilized for the evaluation of a system, questions for respondents are administered by:

A) evaluator
B) system user
C) respondent
D) company supplier
E) customer

Question 39

In the following statements about survey, which one is correct?

A) Survey is a good method to reveal dynamic nature of all phenomena;
B) Respondents are always motivated to answer questions;
C) Rationalization of actions from respondents is always proper and trustworthy;
D) The anonymity of respondents in survey can encourage more candid and honest responses;
E) Respondents that are selected based on convenience can truly represent the intended population.

Question 40

"The reliability of results depends on the truthfulness of respondents' self-report" is a problem in collecting data from computer users. This problem usually happens to the method of:

A) survey
B) experiment
C) guideline review
D) cognitive walkthrough
E) heuristic evaluation

Question 41

The advantage of the survey method includes:

A) the reliability of survey results depends on the truthfulness of respondents' self-report
B) respondents may be unmotivated to answer questions
C) respondents may over-rationalize their actions
D) the results reveal little of the dynamic nature of certain phenomena
E) respondents are encouraged to provide honest responses because of the anonymity in a survey

Question 42

Lab Experiment is a good method for system evaluation when:

A) evaluators have a clear focus on specific consequences of design features on users performance;
B) evaluators want to obtain an understanding about the system use in real setting;
C) subjects for the evaluation are hard to recruit;
D) data about the system use from the whole user population are needed;
E) a company is working on the budget for developing an information system.

Question 43

Which of the following is a step that should be included in the lab experiment on the evaluation of an information system?

A) forming the system developing team
B) developing theory-driven hypotheses to be tested
C) defining responsibilities of project team members
D) analyzing the working efficiency curve of each system developer
E) estimating the potential benefits from implementing the system

Question 44

Field study differentiates itself from other empirical evaluation methods in:

A) the evaluators
B) the type of collected data
C) the evaluation settings
D) the system being evaluated
E) the user population of the system being evaluated

Question 45

When evaluators want to understand actual user experience in real-world situations with minimum interruption to them, which of the following methods would be good to use?

A) interview
B) questionnaire
C) experiment
D) ethnographic observation
E) cognitive walk-through

Question 46

Standards are applied for:

A) achieving consistency across the same type of products;
B) amusing users of the same type of products;
C) eliminating the competition among the same type of products;
D) creating new technology to improve current products;
E) distributing products from manufactures to retailers.

Question 47

An information system developed based on standards has the ability to indicate its:

A) power consumption
B) developing cost
C) ultimate success in application
D) evaluation methods
E) quality

Question 48

CIF stands for:

A) Common Information Forms
B) Centralized Information Framework
C) Common Industry Format
D) Critical Influencing Factor
E) Computerized Interaction Flow

Question 49

Common Industry Format is used for:

A) summarizing characteristics of the user population of an information system
B) reporting the development process of an information system
C) comparing similar products from different companies
D) reporting summative usability test findings about an information system
E) analyzing the sales trend of a specific product

Question 50

Which of the following are concerned with Common Industry Format?

A) customers of an organization
B) external service providers of a company
C) constructors of an organization's facilities
D) family members of an organization's employees
E) usability professionals