"A More Functional Approach to Performance Analysis"
By Doug Peterson
"We've become hypnotized by data," Edward S. Davidson told a full house at the University of Illinois Computer and Systems Research Lab - home base for CSL.
If we hope to gain insights into the behavior of today's computers, which are highly decoupled with many autonomous engines, Davidson said it's going to take more than reams of data. It's going to take a deeper analysis of the data and some considerable creative thought, functional modeling, and validation.
As the first speaker in CSL's series of Golden Anniversary lectures, Davidson said it's easy to fall into the trap of a strictly quantitative approach.
With this approach, he said, you typically come up with many possible solutions to a particular problem, you simulate them over some range of machines and applications, and you plot the results on bar charts resembling the Manhattan skyline. Then you observe which solution seems to work better on average and claim that it is a superior technique in general.
"I'm concerned about that type of research," said Davidson, a former CSL faculty member (1973-87) and professor emeritus at the University of Michigan. "It's publishable today and it's very quick to do. But it is misleading, and people who believe it are dangerous."
According to Davidson, simulations provide data that are accurate only within certain boundaries and specific "regions of validity."
"That's all simulations give you," he said. "But many people today in our field don't realize that they are taking results and applying them far outside their region of validity."
A quantitative approach is certainly better than a non-quantitative approach, Davidson noted, but "we've gone too far. It's time we swing the pendulum back to where we can use the wealth of quantitative data that we can get so easily today to do some serious analysis."
Taking measurements is not performance analysis, he stressed.
Measurements tell us what is happening, but not why. Each data point is based on an array of underlying assumptions. And with a "more functional approach to computer performance analysis," he said you can dig beneath those data points to discover when the assumptions are valid and when they are not.
To drive this point home, Davidson used an assortment of examples, including some work from around 1986 that Pen Yew did at the University of Illinois on an 8-processor Alliant computer with a shared cache. In this work, Yew found that the Alliant was getting cache miss rates of 40 percent for typical large-scale applications.
However, when his paper was submitted for publication, the reviewer rejected it with this reasoning: Everyone knows that cache miss rates are less than 5 percent, so there must be something wrong with the analysis.
Davidson said the reviewer didn't understand that data showing miss rates less than 5 percent applied only within a certain region of validity. The 5-percent figure didn't apply to large-scale, scientific applications; and it didn't apply to shared caches that were sized according to prevailing quantitative rules of thumb for private caches in minicomputers.
Unfortunately, the reviewer was too mesmerized by quantitative folk wisdom to accept this contradictory analysis, so that research paper and the important warning that it raised didn't get published for quite some time.
"I submit that if the paper had said the cache miss rates had been 5 percent, it would have been a totally uninteresting paper but probably would have been accepted for publication," Davidson contended. "But if it says it's 40 percent and the work is valid, it's very interesting but hard to publish.
"As a review community, let's accept more interesting conference papers and journal articles. We don't need more data about old ideas without serious analysis."
By Doug Peterson
"We've become hypnotized by data," Edward S. Davidson told a full house at the University of Illinois Computer and Systems Research Lab - home base for CSL.
If we hope to gain insights into the behavior of today's computers, which are highly decoupled with many autonomous engines, Davidson said it's going to take more than reams of data. It's going to take a deeper analysis of the data and some considerable creative thought, functional modeling, and validation.
As the first speaker in CSL's series of Golden Anniversary lectures, Davidson said it's easy to fall into the trap of a strictly quantitative approach.
With this approach, he said, you typically come up with many possible solutions to a particular problem, you simulate them over some range of machines and applications, and you plot the results on bar charts resembling the Manhattan skyline. Then you observe which solution seems to work better on average and claim that it is a superior technique in general.
"I'm concerned about that type of research," said Davidson, a former CSL faculty member (1973-87) and professor emeritus at the University of Michigan. "It's publishable today and it's very quick to do. But it is misleading, and people who believe it are dangerous."
According to Davidson, simulations provide data that are accurate only within certain boundaries and specific "regions of validity."
"That's all simulations give you," he said. "But many people today in our field don't realize that they are taking results and applying them far outside their region of validity."
A quantitative approach is certainly better than a non-quantitative approach, Davidson noted, but "we've gone too far. It's time we swing the pendulum back to where we can use the wealth of quantitative data that we can get so easily today to do some serious analysis."
Taking measurements is not performance analysis, he stressed.
Measurements tell us what is happening, but not why. Each data point is based on an array of underlying assumptions. And with a "more functional approach to computer performance analysis," he said you can dig beneath those data points to discover when the assumptions are valid and when they are not.
To drive this point home, Davidson used an assortment of examples, including some work from around 1986 that Pen Yew did at the University of Illinois on an 8-processor Alliant computer with a shared cache. In this work, Yew found that the Alliant was getting cache miss rates of 40 percent for typical large-scale applications.
However, when his paper was submitted for publication, the reviewer rejected it with this reasoning: Everyone knows that cache miss rates are less than 5 percent, so there must be something wrong with the analysis.
Davidson said the reviewer didn't understand that data showing miss rates less than 5 percent applied only within a certain region of validity. The 5-percent figure didn't apply to large-scale, scientific applications; and it didn't apply to shared caches that were sized according to prevailing quantitative rules of thumb for private caches in minicomputers.
Unfortunately, the reviewer was too mesmerized by quantitative folk wisdom to accept this contradictory analysis, so that research paper and the important warning that it raised didn't get published for quite some time.
"I submit that if the paper had said the cache miss rates had been 5 percent, it would have been a totally uninteresting paper but probably would have been accepted for publication," Davidson contended. "But if it says it's 40 percent and the work is valid, it's very interesting but hard to publish.
"As a review community, let's accept more interesting conference papers and journal articles. We don't need more data about old ideas without serious analysis."
Edward S. Davidson
Professor Emeritus, Electrical Engineering and Computer Science
University of Michigan
Areas of Specialization: Computer architecture, supercomputing, parallel and pipeline processing, performance modeling, application code assessment and tuning, intelligent caches
Presentation Title: "A More Functional Approach to Performance Analysis"
Delivered:
February 13, 2001
Professor Emeritus, Electrical Engineering and Computer Science
University of Michigan
Areas of Specialization: Computer architecture, supercomputing, parallel and pipeline processing, performance modeling, application code assessment and tuning, intelligent caches
Presentation Title: "A More Functional Approach to Performance Analysis"
Delivered:
February 13, 2001
