Evaluating Research Productivity
no. 105 - June 2001

Second Panel of Research Administrators

JUDGING RESEARCH PRODUCTIVITY ON AN ENTREPRENEURIAL CAMPUS

James R. Bloedel
Vice Provost for Research and Advanced Studies
Iowa State University of Science and Technology

One of the key benchmarks in judging the academic achievements and excellence of faculty is the quality and quantity of their scholarship. For individuals in the social, biological, engineering, physical, and mathematical sciences, research productivity often serves as one of the primary criteria for making this judgment.  At Iowa State University this criterion is related to the broader area of discovery, one of the three major tenets of the University's strategic plan.

Traditionally, the assessment of research productivity has been based on the number of research publications in high quality journals as well as the level and consistency of research funding acquired from competitive sources.  Despite this well-established practice on many campuses, it is becoming progressively more difficult to utilize only these norms for judging research productivity.  The faculty are becoming appreciably more diverse in the exercise of their scholarship. In addition, there is an evolving emphasis on fostering the scholarship of teaching among faculty.  Perhaps most important, campuses are becoming much more entrepreneurial.  Universities are seeking partnerships with industry, and, as an integral part of those partnerships, entrepreneurial activity among its faculty is encouraged.  As a consequence, many are actively engaged in research and the development of intellectual property that can lead to patent applications and the execution of licenses and options. The more aggressive faculty also are becoming involved in the establishment of start-up companies that utilize the intellectual property they developed.  Given the time constraints under which all faculty operate, this type of entrepreneurial activity may at least partially displace the more traditional scientific pursuits, such as publishing in scientific journals and acquiring grants from foundations and agencies.

Since professional diversity is now encouraged on our campuses, it may be time to reassess how the scientific productivity of our faculty is to be judged.  This presentation focuses on the development of a format for evaluating scientific faculty for promotion and tenure at an entrepreneurial institution. It should be emphasized, however, that the framework suggested here could be modified for use in the annual assessment of faculty achievement. 

Two faculty profiles illustrate the type of scholarship activity that could characterize the research portfolio of young scientists who have been particularly active in entrepreneurial endeavors. The first is an assistant professor of electrical engineering who has served on the faculty for six years.  Although having published only eleven manuscripts in reputable journals, this faculty person also had four patent applications.  Furthermore, the applied research of this individual was particularly excellent, one project leading to an R&D 100 Award.  Grant activity consisted of one grant from a private company that resulted in four years of continuous funding at a level that adequately funded the laboratory's entire research program.  Letters from five references were very good, and the teaching assessment of this faculty person indicated an above-average, but not exceptional, performance. The second faculty profile characterizes an assistant professor in mathematics who also has served on the faculty for six years.  Although only seven publications in refereed journals were produced, three widely acclaimed web-based courses were planned, organized, formulated and executed under the leadership of this individual.  Furthermore, all of the courses are under consideration for copyrights.  This faculty member was judged to be an excellent teacher, and the letters of support were very good.  Grant activity consisted of one three-year grant supporting the generation of web-based educational materials in mathematics. 

To assess these two faculty and their somewhat diverse areas of scholarship, we should begin by establishing a definition of research that can be useful in their applications for promotion and tenure.  Clearly a broader definition is required.  In general, I favor defining research as objective-driven scholarship.  This definition is applicable to research activity across many disciplines, including the arts and humanities as well as the sciences.  Furthermore, when appropriate, objective-driven scholarship can apply to educational initiatives as well as extension activities. Consequently, it is feasible to use this definition for evaluating faculty whose emphasis has been in areas outside of traditional research, namely learning and engagement.  At Iowa State University, these areas receive significant emphasis in the promotions and tenure process.  This broader definition of research certainly includes the more entrepreneurial activities of the faculty mentioned above.

Once a working definition of research is established, it is necessary to develop an approach to evaluating faculty with a diversity of achievements and contributions.  Based on the examples I have given, this evaluation should recognize some degree of parity among research contributions resulting in journal articles, patent applications and disclosures, and/or intellectual property related to the educational mission of the university.  Similarly, a broader view in evaluating laboratory funding may be necessary.  For example, although a more classic research career may utilize funding from the National Institutes of Health (NIH) and the National Science Foundation (NSF) predominantly, successful entrepreneurial faculty may derive their funding from private sources very interested in the intellectual property being developed.  In some instances, this can be highly competitive, at least at the higher funding levels.  Based on this view, the critical issue is the adequacy of funding for supporting both the quality and the quantity of the laboratory's scholarship, not the specific source of research support.  

If one accepts the premise that a significant degree of heterogeneity exists among the scholarship activities of scientists on many campuses, it is necessary to derive a common denominator by which research productivity can be judged fairly across the diverse research programs of faculty competing for promotion and tenure in a Research I institution.  One criterion capable of meeting this objective is the impact of the faculty person on his/her field.  This criterion, impact on the field, can be applied to any discipline and any area of scholarship. To meet this standard, the faculty must demonstrate a set of contributions that has impacted a field in a way that modified thinking and/or trends among other scholars in the same area. Almost by definition, implementing this criterion requires the utilization of external peer reviewers; a traditional promotion and tenure committee could not adequately assess the faculty based on "impact on the field" because it would not have the required expertise.  Impact assessment is best judged by individuals who are working in the same area and have had multiple years of experience assessing the impact of new ideas and new findings on their field of expertise.  External experts could provide an unbiased evaluation of a specific person in the context of other faculty at the same institution, and in comparison with individuals throughout the discipline.   

Given the importance of external evaluations in formulating judgments regarding the impact of faculty on their field, I strongly advocate a method of assessment similar to that used in the evaluation of grants and contracts for funding agencies such as NIH and NSF.  This new system utilizes an approach analogous to the study section/research council system with which we are all familiar.  Iowa State University currently considers applications for promotion and tenure from approximately 70-80 faculty each year. Given the areas of scholarship represented across these faculty, the initial reviews could be performed by study sections in seven areas:  engineering; chemistry and physics; math, statistics, bioinformatics and related disciplines; language and literature; fine and performing arts; humanities and social sciences; and medical/veterinary sciences.  Study sections could be comprised of two to three invited external experts approved by colleges for assessing their faculty.   The exact number would depend on the number of applications that a given panel was going to consider.  

Invited participants would be provided with a $300-500 honorarium for their services.  The members could be invited to the University for deliberations over one day, or the members could discuss each of the applicants using a conference call format.  The latter approach clearly would result in minimizing the expenses of the overall process. If visits to the campus were considered preferable, assuming a $1000 travel allotment for each person and an average number of members per section of 2.5 across the seven areas to be reviewed, the maximum cost per year would total  $25,850. The use of conference calls could accomplish the review for a fraction of this cost ($8350). 

Following the assessment of the "study sections," a "council" consisting of the Provost's team (consisting of the associate provosts and vice provosts) would then review the pending recommendations for consistency and fairness. The council would have the responsibility for modifying recommendations should it be necessary.

In summary, this presentation illustrates a progressively more common profile of scientific faculty at an entrepreneurial institution and provides a practical suggestion for fairly and adequately addressing the evaluations required for their tenure and promotion.  Although it could be argued that the faculty profiled above are not worthy of tenure and promotion based on any criteria, there is no question that the scholarship of the scientific faculty at our institutions has become much more diverse.  Since this trend often parallels a related change in the priorities of the university, there is little doubt that a broader perspective is required for faculty evaluation than has been employed in the past.   In fact, on some campuses, chairs and deans have viewed entrepreneurial activity negatively.  This must change if campuses are to attract and retain the best of the new breed of faculty.  Many of these individuals are interested in the wide-range of experiences that result from entrepreneurial activities, not as a substitute for their more traditional scholarship activities, but rather as a complement to their professional experiences while serving our institutions.  Their contributions not only add to the research culture on our campuses, they also provide unique training opportunities for our undergraduate and graduate students. These training opportunities support current trends in graduate education that emphasize the importance of meeting the needs of students interested in careers in industry.  The best programs and the most progressive campuses will be those that can accommodate to the new trends and maintain excellence in programs that continue to train students for academic careers.