Deck 14: Exploring Performance Evaluation and Benchmarking in Computer Systems

Question 1

Which of the following is not a recent trend affecting performance evaluation?

A) Software complexity is increasing.
B) The computing community is approaching a consensus on the validity of various standard performance evaluation tools and methods.
C) Raw and potentially misleading measures, such as processor clock speed, have become influential in the consumer market.
D) Industry-standard benchmarks and synthetic programs are emerging.

Answer 1

B

Question 2

Why is it more important today than decades ago to evaluate the performance of software?

A) Hardware is so fast that the real limiting factor in a computer is the software.
B) Hardware is becoming increasingly similar, even between systems constructed by different vendors, so the software affects performance the most.
C) Software is becoming increasingly complex so it is harder for programmers to determine if their code is efficient by just looking at the code.
D) Software is becoming increasingly complex and often hides the hardware from the user, creating a virtual machine. Cumbersome software can cause a system to execute slowly, even with the most powerful hardware.

Answer 2

D

Question 3

All of the following are common purposes for performance evaluation except __________.

A) performance validation
B) performance projection
C) selection evaluation
D) performance monitoring

Answer 3

A

Question 4

Performance projection would be useful for __________.

A) deciding whether to purchase a particular computer system
B) system tuning
C) estimating the performance of a system that does not yet exist
D) finding bottlenecks in a system

Question 5

All of the following are common measures of performance except __________.

A) throughput
B) reliability
C) processor utilization
D) ease of use

Question 6

Variance in response times measures __________.

A) the typical response time the user expects
B) the range of response times in the system
C) predictability-how response times differ from the mean response time
D) how the response time changes based on changes in the system's workload

Question 7

Evaluators use traces to __________.

A) simulate the user environment a system is intended to execute
B) determine how well a system executes the programs it is intended to run
C) determine which operating system primitives are implemented inefficiently
D) all of the above

Question 8

Why has little progress been made in developing industry-standard traces?

A) Traces are typically proprietary and they rarely are distributed to vendors or to the research community.
B) Trace data is not portable between installations because even small differences in user activity can affect the validity of a trace.
C) For many users, such as those who use personal computers, traces do not yield meaningful performance evaluations.
D) all of the above

Question 9

What is a microbenchmark?

A) a real program, whose executions on various systems are evaluated to make comparisons between systems
B) a small program or function that measures the time required for a system to perform a specific operating system primitive or system operation
C) a program that is designed to evaluate the performance of a specific component or subsystem, such as the file system
D) a measure of "raw horsepower," used for making quick comparisons between hardware

Question 10

Which of the following is not a useful application of microbenchmarks?

A) determining the speed of a particular operating system primitive that is known to be used heavily
B) understanding the relationship between an operating system and a system's underlying hardware
C) evaluating a system's overall performance
D) making comparisons between similar systems, such as different flavors of UNIX

Question 11

An application-specific evaluation is designed to enable evaluators to __________.

A) determine which applications a system executes the best
B) determine which system will execute an application (or group of applications) the best, regardless of how well the system executes other tasks
C) compare the overall performance of two different systems
D) both a and c

Question 12

The hybrid methodology would be more appropriate than the vector-based methodology for evaluating __________.

A) the performance of an application for a database system
B) how well a particular architecture executes system programs
C) how well a particular application will run on a system that does not yet exist
D) the performance an antivirus application

Question 13

Which of the following is/are true of analytical models?

A) Analytical models can be constructed using queuing theory and Markov processes.
B) They are mathematical models of computer systems.
C) A large body of mathematical results exists that evaluators can apply in order to build relatively accurate models for new systems.
D) all of the above

Question 14

Which of the following statements best describes why an evaluator must validate the results of an evaluation performed using analytical models?

A) Systems are often too complex to model accurately, so the results of analytical models are often erroneous.
B) The evaluator needs the results of other performance evaluations to build the model in the first place.
C) Analytical models are useful in the development process, but many consumers distrust or do not understand analytical models. A vendor must validate the results of an analytical model before presenting the results to potential customers.
D) In general, it is a good idea to use multiple techniques to test a system. Additionally, because an evaluator often must make simplifying assumptions to model complex systems, other evaluations should be done to verify that these assumptions are valid.

Question 15

Which of the following is not generally a characteristic of a well-designed benchmark?

A) Multiple iterations of the benchmark on the same system should produce nearly the same results.
B) The benchmark should be able to test for "what-if" scenarios, such as how adding a new Web site will affect a Web server's performance.
C) The benchmark should accurately reflect the types of applications that are intended to run on the given system for which the benchmark is being used to evaluate.
D) The benchmark should be widely used to provide a good basis of comparison.

Question 16

The rationale for developing industry-standard benchmarks is that __________.

A) benchmarks are difficult to develop and it is more efficient to use already-constructed industry-standard benchmarks.
B) industry-standard benchmarks typically reflect the types of applications that are intended to run at a given installation better than a benchmark made at the installation or a production program.
C) benchmarks most often are used as a basis of comparison between systems constructed by different vendors. Industry-standard benchmarks are fairer for this comparison than those developed by individual vendors.
D) all of the above

Question 17

A synthetic program is most useful for __________.

A) thoroughly evaluating a system
B) determining which of two file systems performs better on a particular system
C) isolating which operating system primitives are causing bottlenecks in the system's daily use
D) predicting the performance of a system that has not yet been built

Question 18

Why is an evaluation that uses only synthetic programs inadequate for predicting performance?

A) In general, it is a good idea to use more than one evaluation technique to confirm the results obtained from each type of evaluation.
B) Synthetic programs are not real programs and typically target only a specific subsystem so they do not evaluate how the overall system will perform while executing real programs.
C) Synthetic programs provide valuable insight into which components are creating bottlenecks, but they do not always indicate the precise cause of a component's inefficiency.
D) all of the above

Question 19

What is the primary difference between event-driven simulators and script-driven simulators?

A) Event-driven simulators are controlled by events that occur according to probability distributions, whereas script-driven simulators are controlled by data carefully manipulated to reflect the system's intended execution environment.
B) Event-driven simulators are controlled by events made to occur to reflect the system's intended execution environment, whereas script-driven simulators are controlled by data carefully manipulated based on probability distributions.
C) Event-driven simulators are controlled by traces of events that occurred in other systems built for a similar execution environment, whereas script-driven simulators are controlled by a script of events that evaluators predict is similar to one that would occur in the system's intended execution environment.
D) The only difference is that event-driven simulators are controlled by events, whereas script-driven simulators are executed without any input regarding events.

Question 20

A simulation is most effective for __________.

A) deciding whether it is advisable to build a proposed system
B) evaluating the performance of a system available for customers to purchase
C) determining which of two systems executes a particular application or group of applications most efficiently
D) system tuning

Question 21

Performance monitoring is best used to __________.

A) project the performance of a system that has not yet been built
B) evaluate which of several systems to purchase
C) decide how to tune a system to increase its performance at a given installation
D) determine whether it is worthwhile to upgrade a specific component

Question 22

__________ are most appropriate for performance monitoring.

A) Profiles
B) Analytical models
C) Benchmarks
D) Simulations

Question 23

A bottleneck occurs when a __________.

A) system is operating below peak performance
B) resource hinders the progress of the system because it cannot do its job efficiently
C) process interferes with another process because it is competing for a shared resource
D) resource's service rate is higher than its arrival rate

Question 24

Which of the following statements about bottlenecks and saturation is false?

A) Removing a bottleneck will not always result in increased performance.
B) When a resource becomes saturated, processes interfere with each other because they must wait for other processes to complete using the resource.
C) A virtual memory system that is thrashing is an example of a saturated resource.
D) Saturation cannot occur if the system contains sufficient resources to fulfill all incoming requests.

Question 25

Which of the following would produce negative feedback?

A) a processor in a system whose poor performance is due to thrashing
B) an overloaded Web server
C) a disk in a RAID array whose queue of pending accesses is short
D) an idle print server

Question 26

Positive feedback can be misleading __________.

A) When a resource has excess capacity because another resource with which it cooperates is a bottleneck.
B) When a resource has excess capacity because the system is sending requests to other equivalent resources, all of which are overloaded.
C) When a resource must remain idle for quick responses to critical requests.
D) When a resource has a small (or zero) queue length, but is slowly servicing requests.

Question 27

CISC ISAs do not __________.

A) incorporate single instructions that perform complex functions
B) take the same amount of time to execute most or all instructions
C) contain variable-sized instructions
D) contain a small number of registers

Question 28

What was/were the original motivation(s) behind the CISC processor-design philosophy?

A) CISC instruction sets could be implemented with simple hardware, decreasing hardware-design cost.
B) Programs written for CISC processors executed quickly because they took full advantage of pipelines.
C) CISC instruction sets made it easy to code in assembly language, and programs consumed less memory.
D) Complex instructions were more-frequently used than simple instructions.

Question 29

RISC processors __________.

A) contain a small instruction set
B) exhibit a uniform execution time for most or all instructions
C) contain fixed-size instructions
D) decode instructions using microcode

Question 30

__________ is not a tenet of the RISC processor-design philosophy.

A) Making the common case fast
B) Reducing context switch latency
C) Reducing the number of register-memory transfers
D) Keep hardware simple

Question 31

Today's CISC processors are similar to RISC processors in that they include __________.

A) multiple execution units for superscalar execution
B) on-chip floating-point and vector coprocessors
C) an optimized core subset of simple instructions to which all complex instructions are decoded
D) infrequently used instructions

Question 32

In what ways are today's CISC and RISC processor designs divergent?

A) RISC processor instruction sets still only contain fixed-length instructions that generally execute in one cycle, whereas CISC processors do not.
B) RISC processor instruction sets still are smaller than CISC instruction sets.
C) RISC processors still contain simple hardware, whereas CISC processors contain complex hardware.
D) Although many processors that are labeled "RISC" actually have adopted typical CISC features, there are still many other processors that remain true to the original RISC characteristics.

Question 33

EPIC processors attempt to provide higher performance than so-called post-RISC processors by __________.

A) including substantially more execution units than do conventional superscalar architectures
B) branch predication
C) speculative loading
D) all of the above

Question 34

__________ is not a tenet of the EPIC processor-design philosophy.

A) Allowing the compiler to choose the path of execution
B) Incorporating techniques from both VLIW and superscalar designs
C) Allowing the processor to reorder instructions
D) Simplifying processor hardware