Evaluating Research Productivity
no. 105 - June 2001

KEYNOTE ADDRESS
Joan F. Lorden
Associate Provost for Research and Dean of the Graduate School,
University of Alabama at Birmingham

• Evaluation is useful to set benchmarks, to recognize excellence and to promote improvement.  We become what we measure, so it is important to choose wisely.
• The study released by the National Research Council in 1995 on research doctoral programs in the United States achieved some important goals.  It provided broad coverage in terms of the number of universities. The report was derived from national datasets using a process of uniform data collection. Unlike the U.S. News and World Report survey, it provided in-depth analysis, and the rankings were done by scholars in the field of study. The primary features of the National Research Council study were:  rankings, a reputational survey, longitudinal comparison, institutional information and objective measures of performance.

• The Council on Research Policy and Graduate Education (CRPGE) provided feedback to the National Research Council and suggested several changes for the next study, critiquing in particular the undue emphasis on ranking programs based on reputation. The CRPGE suggested that reputational rankings don't reflect the tremendous change some fields of study have experienced in the last 20 years because reputations are slow to change.  The CRPGE is a group established by the National Association of State Universities and Land Grant Colleges.

• The survey portion of the study exemplifies many of the problems encountered when measuring research quality, but it successfully demonstrated the link between graduate education and research.
• Other questions to consider about rankings include: Is it valid that the top-ranked programs consistently have a larger faculty and more students?  How can we evaluate niche programs? 
• According to the survey, high-ranked programs: have a large faculty; are well funded; publish successfully; and give their graduate students research assistantships.  The count of per capita publications tends to correlate with ranks, but awards and honors are the marker of significance for the arts and humanities in the top quarter. In science and engineering, federal funding is the highest in the top quarter rankings.
• In general, the rankings don't tell us much about the experience of students or outcomes of graduate education.  We do see that students from the higher ranked programs are supported more often on research assistantships, whereas students from lower-ranked programs are supported more often on teaching assistantships.  We also see that time to degree increased more in the lower-ranked schools.  We would benefit from knowing what happens with graduate students while on campus and the outcomes of their education.
• The University of Alabama at Birmingham determined that funding is an important measure.  Most areas that have experienced funding success with the National Institutes of Health have been interdisciplinary. To provide incentives, the University of Alabama at Birmingham established an umbrella operation for interdisciplinary centers and guidelines, and also invests in targeted areas. 
• When choosing measures for the future, we should ask:  What are the goals? Who is the audience? Do the measures reflect our values?  Do we understand their limitations?  How will the measures be used?

• We should select and promote measures that reflect the values we think are important.
• The National Science Foundation annually collects data on the federal Research and Development expenditures in science and engineering.  This information has become our "gold standard" for national comparisons.  Rankings of this type also provide a surrogate for market share in terms of the percentage of the federal Research and Development funds obtained by a given university. Although federal expenditures in Research and Development measure national research competitiveness, this statistic underestimates the local impact of research.  The University of Kansas (KU) uses the same methodology, but extends it to include fields outside of science and engineering and to include research training grant expenditures. This is a measure of RDandT - research, development, and training expenditures.  KU's expenditures for Research, Development and Training rose 15% from fiscal year 1999 to 2000.
• The United States Department of Commerce estimates that in Kansas each $1 million in Research and Development funding creates 40.6 jobs.  The three Kansas research universities had $335.2 million in Research, Development and Training expenditures in fiscal year 2000, which implies that more than 13,600 jobs are due to this source of funding.  The average salary in these jobs exceeds the average salary in our state.
• Graduates are the largest form of technology transfer from research universities.  The annual income of the alumni of the three Kansas research universities, who currently reside in the state, is $9 billion.  About 1/3 of this total, or $3 billion, is due to the increased salaries they earn due to their degrees.  The state tax paid by these graduates is $700 million annually, a figure that exceeds the annual state appropriation to the three universities of $400 million.
• To maximize research productivity, we must minimize internal competition between academic departments and research centers. KU uses a multiple credit algorithm to accomplish this; expenditures are recorded in two lists, one according to departments and one according to centers. 

• To maintain research productivity, it is important to encourage participation at all levels:  faculty at all stages of their development; pre-doctoral and postdoctoral training programs; and mentoring and support of the most productive faculty so we don't lose them.
• Some research supports the notion that productivity is greater in larger institutions and departments because of the "intellectual synergy." Other factors to consider at the departmental level are: workloads, availability of leave-time and travel funds, the number of students on research support, availability of non-governmental funds, and availability of star faculty.
• Senior faculty are motivated to remain active as scholars by the intrinsic rewards of mentoring their graduate students.  They also thrive on public recognition of their contributions to the profession.  Interdisciplinary teams can energize faculty by creating opportunities and stimulating new research.
• The National Research Council rankings in economics are strongly associated with objective measures of productivity such as total citations or total pages in refereed journals.  NRC data also establishes a link between size of department and rank.  The large departments in the top 50 tend to have faculty in a variety of fields, which would seem to discredit the strategy of building a "unique niche."
• The individual faculty member is motivated to be productive because of his/her "passion for reputation" and "taste for originality."  The challenge is to find how these attributes then lead to publications, citations and impact assessments at the unit-level.  "Bibliometrics" is not helpful in actually fostering productivity.
• The researcher who is productive over a long career may experience multiple peaks and valleys as he/she invests additional time in acquiring new skills and competencies in order to develop new lines of investigation.  The system for evaluating research productivity at the unit-level must reflect this non-linear career trajectory at the individual level.

• One departmental model for faculty evaluation is, in a sense, "Darwinian."  It quantifies annual research productivity for all investigators; derives an average; compares each investigator with the average investigator; and then distributes rewards accordingly.  Investigators who do poorly over time are weeded out, whereas the strongest, most adaptable in the department, thrive. Whether the evaluation system is "Darwinian" or "Egalitarian," faculty members are not completely satisfied; however, it seems that faculty in departments with a major Darwinian component are more satisfied than they are in circumstances where the chair gives a highly subjective, or no, evaluation.
• We haven't determined how to recognize the quality or productivity of activities that are not for an academic audience.  In evaluating the outcomes of graduate education, how do we judge the placement of chemistry students who go into non-academic institutions?  Should the standard be placement in Fortune 500 companies?  Do we count the number of students who start their own companies?  Do we count patents?  Is a number an adequate indicator of productivity, or do we attach a dollar value?  As universities move in the direction of increased collaboration with industry, with increased public accountability, and respect for the wide range of career opportunities for our doctoral degree recipients, it will become more important to develop assessment and evaluation strategies that align with the values and goals of our non-academic audiences. 
• We must be careful to develop appropriate measures of quality and impact in the arts and humanities or we may erode the position of these disciplines at our institutions, especially when measures of "impact" drive resource allocation models in the future.  By intention or happenstance, our support of the arts and humanities will be an important statement about our institutional values.
• It is difficult to convince faculty that they should be interested in assessment as a strategy for improving the things they care about, i.e., the preparation of the next generation of scholars and researchers; faculty often believe that administrators actually want a quick and efficient way of allocating - or more frightening still, reallocating resources.
• For biomedical institutions, the total award by the National Institutes of Health (NIH) is a measure that meets key characteristics:  it can be measured in a simple, easily understood and goal-directed manner.  It has a strong association with other markers of research productivity and it is a clear outcome.  However, the use of a productivity index must not be confused with the goals and values of the institution, which include scholarship, clinical care, education and service.
• To make NIH funding the gold standard in an institution, each school must have its own mandate to increase NIH funding and create a strategic research plan for a 5-7 year period, with award targets as their goal.  Administrators who set the goals and oversee the process should be held accountable, using NIH awards as the productivity measure of the programs in their area of responsibility.

• Those who are concerned about the University's return on investment are: governing boards, accrediting bodies, funding agencies, state legislators, taxpayers, prospective students, employers of students and bill-paying parents.  Education is a value-added product.  It creates a significant difference in income for the student, particularly when comparing the salaries of high school graduates with those who earned a masters degree.  It also makes a difference to society; we have estimated that Kansas State University alumni paid $250 million in taxes from the earnings garnered from a college education. In agriculture, the value of research can be measured in dollars; for example, KSU high-yield wheat has been shown to generate $64 million more in income for farmers.
• Increasingly we must look at technology transfer and create new productivity measures: licensing income; licensing-linked research funding; companies launched and jobs created.  In a document published by Kansas State University in 1998 on the economic impact of research and teaching, we estimated that the return on investment is $17 for every dollar.  
• The master campus plan at the University of Missouri - Columbia involves these goals: maximizing internal resources and communications; enhancing research compliance; providing grant assistance; nurturing technology development; expanding external partnerships; and fostering governmental relations.  To make better use of internal resources, MU has established a campus network of 55 grant writers and a grant information system. Through its office of Technology and Special Projects, MU provides mentoring on technology transfer and develops relationships with economic development entities as it encourages entrepreneurship on campus.
• The scholarship of the scientific faculty is now more diverse. Many individuals are interested in the wide-range of experiences that result from entrepreneurial activities, not as a substitute for their more traditional scholarship activities, but rather as a complement to their professional experiences while serving our institutions.  Their contributions not only add to the research culture on our campuses, they also provide unique training opportunities for our undergraduate and graduate students. These training opportunities support current trends in graduate education that emphasize the importance of meeting the needs of students interested in careers in industry. 
• If we are to attract and retain faculty who are entrepreneurial, a broader definition of productivity is needed; "objective-driven scholarship" can apply to educational initiatives as well as extension activities.  Evaluation would then be based on "impact on the field."  To meet this standard, the faculty must demonstrate a set of contributions that has impacted a field in a way that modified thinking and/or trends among other scholars in the same area.  For Promotion and Tenure, I suggest setting up external study sections with experts chosen on the basis of their capacity to assess the impact of research.  This process would parallel the one established by NIH and NSF for evaluating grants and contracts. 

• A market model of evaluating our productivity does not work unless universities can demonstrate that they are covering the full cost - and yet all our activities are subsidized.  This is why we fall short in using measures such as the quantity of external funding.
• Our arguments about state funding and higher education may not succeed because the public knows our contributions are not unique.  Higher education does improve an individual's income, but if our state did not support universities, students would be able to seek an education elsewhere.  Likewise, the public may not accept the argument that our institutions give a good rate of return on monies invested because we cannot say that the leverage we provide for investment in higher education is really better than the benefits derived from money spent on traffic safety or early childhood education, etc.
• The public relies on U.S. News and World Report for information on college rankings, not the National Research Council.  The value of the process is further jeopardized if our evaluation schemes do not measure up to our ideals, and our faculty don't believe in the process.

• There are many differences between the culture of academe and that of the legislature. It is difficult to understand the way compromise is reached in the legislature, but it is critical to accomplishing their objectives. 
• We need to spend more time thinking about what it is we are doing in academia and how our work can be cast into an appropriate form.
• The universities need a focused message in communicating with the legislature.  Too often, we find ourselves espousing our own individual needs and positions.  It is important to unify our voices and if we do this, it makes it easier for the press to espouse our position and for the legislature to accomplish our objectives.
• Our long-term work with legislators should combine "friend raising" with fund raising.

• Academics in the developed world have contented themselves for many centuries with the same approaches to education as they themselves experienced.  Few scholars have read the literature on learning styles and best practices for teaching.
• We are likely to experience increasing difficulties in attracting public and private support for our institutions unless we change the way that we reward academic scholarship. Scholarship can be viewed from a broad perspective. It is integral to all three components of higher education:  learning, discovery and engagement.  Scholarship can be communicated through: teaching materials and methods, classes and curricula as well as publications, presentations, exhibits, performances, and also patents, copyrights and the web.  We need to place scholarship in the context of the institution its serves, not just the discipline it supports.
• Applied science is what John Maddox says has dramatically changed and improved the lives of people in the twentieth century.  Iowa State University has invested in applied science via the Plant Sciences Institute.
• Funding patterns for higher education in Kansas reflect more than 100 years of decisions to foster a high participation rate via community colleges and technical schools as well as the regional and research universities.  In an economic development context, we see a difference in funding priorities in Nebraska, Iowa, and Missouri where more investment is being made in research universities
• While every research university works to the limits of its ability to expand research and development, this happens in a context in which education retains primacy.  The federal agenda for academia is focused on research, and the state agenda is founded and evaluated primarily on the basis of undergraduate education.  These two forces frequently are in conflict.
• The standard model of the complete scholar is too constraining to be affordable.  Not everyone is able to maintain a research output that is nationally competitive, and even fewer can establish and maintain a national reputation.  While it is in everyone's best interest to celebrate and capitalize upon those who can produce optimally in teaching, research and service, we should recognize that not everyone can do this over the entire course of a career.  It is most effective to create flexibility in roles and rewards so that work can be allocated according to an individual's strengths, especially during the post-tenure period.