Evaluating Research Productivity
no. 105 - June 2001

Panel of Researchers

A PASSION FOR REPUTATION, A TASTE FOR ORIGINALITY

When Mabel Rice first invited me to this retreat on "Evaluating Research Productivity," I dashed off to look up the topic, where else, on the Internet. I found lists of rankings, lists of productivity indicators, and policy statements designed to increase research productivity.  I even read some of this stuff, learning much more than I previously knew about citation indices and their limitations, per capita publication counts, the research productivity ranking of Hong Kong University's economics department, and biases of impact and reputation rankings.  I acquired a few, new buzzwords and phrases.  I particularly like the notion of  "streamlining the administrative process" [note: not streamlining the administration] as one key to increasing research productivity and the distinction between measuring the "vitality" of the institution, not its "quality of research." 

I found little of this sort of thing to be helpful to me, professionally, and certainly not something I could pass along to my students as sage advice.   It seems to me that the researcher has gotten lost in all these discussions of evaluating research productivity - the researcher, the one who writes the books, referees the articles, and delivers the keynote addresses.  It seems to me that what is a missing is a bridge between the formal metrics for evaluating research productivity (citation counts, peer evaluations, dollar cost/benefit analyses), applied at the unit level, and the activities of individual researchers. 

I take it that the goal of evaluating research productivity is to better help the university manage its resources: resources flow to units with greater productivity, and improvements in productivity reduce costs and increase quality.  Measuring research productivity involves establishing appropriate benchmarks or indicators, such as counts of publication, citations, and patents, and scaling each unit relative to appropriate professional, national or international standards.  

While there appears to be an extensive literature on "bibliometrics" and "scientometric," there is little that relates to the activities of the individual researcher - with one exception.  The only discussions I uncovered that relate such measures of research productivity to individual faculty members concerned tenure disputes; I found citation counts evoked by those denied tenure as an objective measure of impact.  This to me signals a disturbing trend, one of substituting unit-level aggregations of quantifiable data - this new science of bibliometrics- for more qualitative reviews of individual researchers' productivity.  

I next turned to my collection of "faculty survival guides" and found much advice regarding that "pesky tenure problem" as well as networking, socialization, dealing with sexist comments, preparing course syllabi, devoting time to writing, and choosing committee assignments.  But again, there is surprisingly little advice on the evaluation of research productivity at the level of the individual researcher. 

After logging a lot of time web surfing and even a few hours in the library, I have found one source that I think is helpful:  Advice for a Young Investigator by Santiago Ramón y Cajal, the pioneering Spanish neuroanatomist who is credited with the discovery of the synapse and shared the 1906 Nobel Prize in Physiology.   Writing in 1897, he offered a wide range of advice on many matters pertaining to evaluating research productivity.

I'll spare you his advice on choosing a wife - one key to research productivity - and turn to his typology of "diseases of the will" that I found illuminating:

"We have all seen teachers who are wonderfully talented and full of energy and initiative - with ample facilities at their disposal- who never produce any original work and almost never write anything.  Their students and admirers wait anxiously for the masterpiece worthy of the lofty opinion they have formed of the teacher.  But the great work is never written, and the teacher remains silent. Let us not be deceived by optimism and good intentions.  Despite their exceptional merit, and the zeal and energy they display in the classroom, such teachers suffer from a disease of the will - their students and friends may nevertheless consider them abnormal and suggest some adequate form of spiritual therapy, with all due respect to their fine intellectual abilities" (p. 75).

Cajal classifies these diseases of the will as "the dilettantes or contemplators; the erudite or bibliophiles; the instrument addicts; the megalomaniacs; the misfits; and the theory builders" (p. 75).  He is most dismissive of the contemplators as "likeable for their juvenile enthusiasm and piquant and winning speech as they are ineffective in making any real scientific progress" (p. 77) and he recognizes that  "cold-hearted instrumental addicts cannot make themselves useful" (p. 82) and he notes that the misfits, who occupy a professorship  "simply to collect the salary, and to enjoy the incidental pleasure of excluding the competent," are "hopelessly ill" (pp. 82-83).

For the rest, Cajal has some recommendations regarding evaluating research productivity at the individual level.  Cajal's recommendations ring as true today as they did in 1897.  Cajal reminds the bibliophile that "We render a tribute of respect to those who add original work to a library, and withhold it from those who carry a library around in their head" (p. 78).  He advises the megalomaniac to "tackle a small problem first "[an approach which] may not always lead to fame but [to] the esteem of the learned and the respect and consideration of our colleagues" (p. 80).  He notes that rather than bemoaning the lack of able assistants, or laboratory equipment, or government funding, that "dreamers do not work hard enough" (p. 80).  And he reminds the theorist that "Theories desert us, while data defend us" (p. 86).  

For all, Cajal cautions that independent judgment, intellectual curiosity, perseverance, and concentration are required. He also recognizes that excellence in research and teaching are complementary, requiring similar intellectual traits.  He notes that time constraints and competing time demands erode the scientist's devotion to research.

Cajal foresaw the current debates over basic versus applied science as a "false distinction" since "nothing in nature is useless" and applications "will always come, whether in years or perhaps even in centuries" although he does bemoan that "the struggle for patents, and the fever of industrial competition, have disturbed the august calm in the temple of Minerva."

And he called for public support of science, pointing out that "today's statesmen undoubtedly have limitations, one of which is not realizing - that the greatness and might of nations are the products of science, and that justice, order, and good laws are important but secondary factors in prosperity" (p. 91).  

Beyond these prerequisites, Cajal emphasizes that research productivity results from a "passion for reputation, for approval and applause," and a "taste for originality, the gratification associated with the act of discovery itself."  So in evaluating research productivity, our challenge is to evaluate this "passion for reputation" and this "taste for originality" that then lead to publication lists, citations counts, and impact assessments for unit-level aggregations.

I'd like to now introduce a bit of gerontology into our discussion.  In the book Age and Achievement, Lehman (1953) evaluated research productivity in a number of different fields.  He identified, using standard sources, the most significant achievements in those fields, often recognized years later by Nobel Prizes or other awards.  And he determined the age of the individual at the time these contributions were made.  He published a series of graphs showing striking agreement: research productivity peaks in the 30s, and rapidly declines thereafter - in medicine, in mathematics, or in German grand opera.

Recently Simonton (1990) has reanalyzed Lehman's data, drawing a different conclusion. Simonton has shown that research productivity is a function, not of chronological age, but of time-in-profession.  Following a 4 to 8 year period of training and apprenticeship, research productivity peaks and then rapidly declines.  What Simonton's analysis makes clear, is that one is "over the hill" professionally not by age 40 or so but approximately 15 years after entering the profession.  Simonton also makes clear, and illustrates with numerous examples, that one can experience many productivity peaks by shifting careers, frameworks, and methodologies at the price of investing 8 to 12 years in a new period of training and apprenticeship.

So my point is that our system of evaluating research productivity assumes that productivity is a monotonically increasing function.  It is not, and the successful researcher who is able to sustain productivity over a long career may experience multiple peaks (and valleys) as he or she invests additional time in acquiring new skills and competencies in order to develop new lines of investigation. 

So my survey has identified two dissociations between the unit-level measures of research productivity and the efforts of individual researchers.  First, productive researchers are distinguished from less productive dilettantes, bibliophiles, instrument addicts, megalomaniacs, misfits, and theory builders by their "passion for reputation" and "taste for originality."  These traits do not appear among the five factors of personality modeled by Costa and McCrae (1998), nor do they map only McClelland's (1953) achievement motive, and they certainly don't correspond to any of the profiles on the MMPI (Hathaway and Meehl, 1967).  My concern is that these qualities are also overlooked by bibliometrics.  Counting publications, estimating impact, and assessing costs/benefit ratios may be appropriate for unit-level aggregations of researchers but not for the evaluation of research productivity of individual researchers. Individual researchers are motivated, not by their count of publications or listing of citations, but by their "passions" and "tastes."

Second, research productivity is not linear but is best described by Simonton's exponential function of "investment time."  Our system for evaluating research productivity at the unit-level must reflect this non-linear career trajectory at the individual level.  We must encourage and allow individual researchers, who have demonstrated their productivity, to seek out new venues.  We must grant them the time to invest in new skills and competencies if we wish to sustain their productivity over a long career.  Bibliometric practices that assume productivity is monotonic will discourage those very investments that promote research productivity. 

Thus, I think no discussion of "evaluating research productivity" at the institutional level can be complete without a discussion of "evaluating research productivity" at the level of individual researchers.  Whenever we aggregate data to examine trends in research productivity or to look at the leveraging of state funds, we are in danger of overlooking the variability of individual research careers.  The unit level aggregations will average over nonproductive dilettantes, bibliophiles, instrument addicts, megalomaniacs, misfits, and theory builders.   Aggregations at the unit-level will also mask the peaks and valleys of individual research careers. We need to question how bibliometrics affects not only institutional research planning but how it may impact the very "passion for reputation" and "taste for originality" that motivate researchers to be productive and to sustain their productivity over a long career.

References

Ramón y Cajal, S. (2000). Advice for a Young Investigator (N. Swanson and L. W. Swanson, Trans.). Cambridge, MA: MIT Press.

Costa, P. T., and R. R. McCrae (1998).  Emerging Lives;  Enduring Dispositions:  Personality in Adulthood. Boston:  Little, Brown, & Co.

McClelland, D. C. (1953).  The Achievement Motive.  New York:  Appleton-Century-Crofts, 1953.

Hathaway, S. R., and Meehl, P. E. (1967).  Atlas for the Clinical Use of the MMPI.  Minneapolis, MN:  University of Minnesota Press.

Lehman, H. C. (1953). Age and Achievement. Princeton, NJ: Princeton University Press.

Simonton, D. K. (1990). Creativity and Wisdom in Aging. In J. E. Birren and K. W. Schaie (Editors.), Handbook of the Psychology of Aging (pp. 320-329). New York: Academic Press.