Evaluating Research Productivity
no. 105 - June 2001

Panel of Researchers

STATISTICAL ANALYSES OF ECONOMICS NRC SCORES

Michael Podgursky
Professor of Economics
University of Missouri - Columbia

Economists are obsessive about measuring things.  Thus it was inevitable that when the rankings by the National Research Council (NRC) were released, particularly given their "high stakes" character, economists would start doing statistical analyses of them.  There have been several good studies of these rankings.  Today I am going to focus on what I consider the most thorough of these - a recently published study by a well-known econometrician at Purdue (Thursby, 2000).  This study was published in the Journal of Economic Literature, one of the "flagship" professional journals.

Thursby actually had access to the underlying individual survey responses.  He did quite a bit of reliability checking and explored a variety of hypotheses concerning the rankings.  I will cover some of the high points of his study.  Here are a couple of the questions that were addressed.

1.  Are the NRC rankings measuring something real?

The answer to this seems to be "yes."

As all of you are aware, the NRC rankings are based on a subjective survey of faculty concerning the quality of departments and Ph.D. program effectiveness.  On the whole, economists are wary of subjective survey data.  We usually prefer "hard" data like prices and quantities.  And skepticism is not just limited to economists.  When these rankings were first released I heard grumbles from my fellow department chairs in the College of Arts and Sciences that this was a "beauty contest" and similar disparaging remarks.

However, these NRC rankings in economics are strongly associated with objective measures of productivity such as total citations or total pages in refereed journals - measures that economists take seriously.  If the NRC score is the dependent variable, publications and citations - current and lagged - explain about 90 percent of the variation in NRC scores.

There are a couple of things to note about the statistical model that generates this good fit.  First, the effect of citations and publications is non-linear.  Specifically, these variables demonstrate diminishing returns. That is, it takes progressively more and more citations or journal pages for your NRC score to move up as you get closer to the top.  For example, it might take 30 journal pages to move you from ninety-eight to ninety-seventh rank, but 250 pages to move a department from twenty-six to twenty-fifth rank. 

In addition, the effect of citations and publication is different depending on their age. Current journal pages have a bigger effect than older pages.   Citations to older articles count more than citations to current articles.

So, these subjective NRC rankings are strongly associated with objective measures of departmental productivity in ways that make sense.

Let me note, however, that this strong association holds for total department productivity.  The relationship is much weaker when we consider per capita productivity.  When per capita measures of journal pages and citations are used in the model instead of total pages and citations, the percent of variation explained falls from 90 percent to just 60 percent.  So, size clearly matters in the NRC rankings. 

2. What factors produce higher rankings?

If we think of NRC scores as outputs, what inputs produce a yield with higher output?  Thursby again fits a non-linear statistical model to these data.  He found that total faculty size matters.  No surprise here.  The proportion of full professors, external grants, library expenditures, research assistants per faculty member, and the number of faculty per undergraduate student all mattered.  The latter variable, by the way, fully explained the public-private difference in rank.  The private schools had far more faculty per undergraduate student.  As with the former analysis, most of these inputs displayed diminishing returns.

The link between size of department and rank is clearly visible in even casual inspection of the NRC data.  The average department size in the lowest quartile of departments was 17.4 whereas the average size in the highest quartile was 36.1.

More recent evidence on the effect of size may be found in the U.S. News and World Report top 50 rankings. We counted the number of regular faculty based on information on the web sites of these departments.  The chart below shows the number of regular faculty in the top 50 economics Ph.D. programs according to the most recent U.S. News ranking (see figure 1).  [I omitted  two departments: CalTech and Claremont-McKenna.  Caltech doesn't have an economics department; it has a quantitative social science department.  Claremont-McKenna draws on faculty from a variety of private colleges in the area.]  For background, let me indicate the size of economics departments for some of the Midwestern public universities represented at this conference:  the University of Missouri-Columbia 17, the University of Nebraska 17, the University of Kansas 17, and Kansas State University, 16.  (None of these is in the top 50.)  The department at Iowa State is in the top 50, and has 29 faculty.

There are a couple of things to note here.  First, the average size in the top 50 is 32.  Furthermore, as you can see from the ranking, there is no department in the top 50 that is as small as my department. 

If these were data on firms in an industry, the interpretation would be clear.  There are "economies of scale" in this market and if you want to be competitive you need to be larger than a minimum size threshold.

Several years ago, a previous Chancellor at the University of Missouri urged departments to pursue what he termed a "unique niche" strategy.  He believed that a department could pull itself up and gain high professional rank by  "putting all its eggs in one basket," finding a niche, and becoming the best around in that niche area.  Aside from the fact that this is a very risky strategy (suppose a physics department picked "cold fusion"), there is no evidence in these data to suggest it works in raising NRC rankings.   So far as I'm aware, none of these departments is highly specialized.  On the contrary, they tend to be fairly diversified, with faculty in a variety of fields.  To be sure, some departments stand out in some fields more than others, but all of them have high quality productive faculty in a variety of fields.  They all produce dissertations in all or most of the major fields in the profession.

In short, I see no quick, inexpensive gimmick that will move a department up in these rankings.  You need to be big and you need to be good.

References

Thursby, Jerry G. (2000, November).  What Do We Say about Ourselves and What Does It Mean?  Yet Another Look at Economics Department Research.  Journal of Economic Literature, 33, 383-404.