Joan F. Lorden
Associate Provost for Research and Dean of the Graduate School,
University of Alabama at Birmingham
RESPONSE TO THE KEYNOTE ADDRESS
- Evaluation is useful to set benchmarks, to
recognize excellence and to promote improvement. We become what
we measure, so it is important to choose wisely.
- The study released by the National Research
Council in 1995 on research doctoral programs in the United States
achieved some important goals. It provided broad coverage in
terms of the number of universities. The report was derived from
national datasets using a process of uniform data collection.
Unlike the U.S. News and World Report survey, it provided
in-depth analysis, and the rankings were done by scholars in the
field of study. The primary features of the National Research
Council study were: rankings, a reputational survey, longitudinal
comparison, institutional information and objective measures of
The Council on Research Policy and Graduate
Education (CRPGE) provided feedback to the National Research
Council and suggested several changes for the next study, critiquing
in particular the undue emphasis on ranking programs based on
reputation. The CRPGE suggested that reputational rankings don't
reflect the tremendous change some fields of study have experienced
in the last 20 years because reputations are slow to change.
The CRPGE is a group established by the National Association
of State Universities and Land Grant Colleges.
- The survey portion of the study exemplifies
many of the problems encountered when measuring research quality,
but it successfully demonstrated the link between graduate education
- Other questions to consider about rankings
include: Is it valid that the top-ranked programs consistently
have a larger faculty and more students? How can we evaluate
- According to the survey, high-ranked programs:
have a large faculty; are well funded; publish successfully; and
give their graduate students research assistantships. The count
of per capita publications tends to correlate with ranks, but
awards and honors are the marker of significance for the arts
and humanities in the top quarter. In science and engineering,
federal funding is the highest in the top quarter rankings.
- In general, the rankings don't tell us much
about the experience of students or outcomes of graduate education.
We do see that students from the higher ranked programs are supported
more often on research assistantships, whereas students from lower-ranked
programs are supported more often on teaching assistantships.
We also see that time to degree increased more in the lower-ranked
schools. We would benefit from knowing what happens with graduate
students while on campus and the outcomes of their education.
- The University of Alabama at Birmingham determined
that funding is an important measure. Most areas that have experienced
funding success with the National Institutes of Health have been
interdisciplinary. To provide incentives, the University of Alabama
at Birmingham established an umbrella operation for interdisciplinary
centers and guidelines, and also invests in targeted areas.
- When choosing measures for the future, we
should ask: What are the goals? Who is the audience? Do the measures
reflect our values? Do we understand their limitations? How
will the measures be used?
Robert E. Barnhill, Vice Chancellor
for Research and Public Service, University of Kansas
PANEL OF RESEARCHERS
- We should select and promote measures that
reflect the values we think are important.
- The National Science Foundation annually
collects data on the federal Research and Development expenditures
in science and engineering. This information has become our "gold
standard" for national comparisons. Rankings of this type also
provide a surrogate for market share in terms of the percentage
of the federal Research and Development funds obtained by a given
university. Although federal expenditures in Research and Development
measure national research competitiveness, this statistic underestimates
the local impact of research. The University of Kansas (KU) uses
the same methodology, but extends it to include fields outside
of science and engineering and to include research training grant
expenditures. This is a measure of RDandT - research, development,
and training expenditures. KU's expenditures for Research, Development
and Training rose 15% from fiscal year 1999 to 2000.
- The United States Department of Commerce
estimates that in Kansas each $1 million in Research and Development
funding creates 40.6 jobs. The three Kansas research universities
had $335.2 million in Research, Development and Training expenditures
in fiscal year 2000, which implies that more than 13,600 jobs
are due to this source of funding. The average salary in these
jobs exceeds the average salary in our state.
- Graduates are the largest form of technology
transfer from research universities. The annual income of the
alumni of the three Kansas research universities, who currently
reside in the state, is $9 billion. About 1/3 of this total,
or $3 billion, is due to the increased salaries they earn due
to their degrees. The state tax paid by these graduates is $700
million annually, a figure that exceeds the annual state appropriation
to the three universities of $400 million.
- To maximize research productivity, we must
minimize internal competition between academic departments and
research centers. KU uses a multiple credit algorithm to accomplish
this; expenditures are recorded in two lists, one according to
departments and one according to centers.
Carol Shanklin, Professor of Institutional
Management and Dietetics, Kansas State University
Michael Podgursky, Professor of
Economics, University of Missouri - Columbia
Susan Kemper, Distinguished Professor
of Psychology and Gerontology, University of Kansas
FIRST PANEL OF RESEARCH ADMINISTRATORS
- To maintain research productivity, it is
important to encourage participation at all levels: faculty at
all stages of their development; pre-doctoral and postdoctoral
training programs; and mentoring and support of the most productive
faculty so we don't lose them.
- Some research supports the notion that productivity
is greater in larger institutions and departments because of the
"intellectual synergy." Other factors to consider at the departmental
level are: workloads, availability of leave-time and travel funds,
the number of students on research support, availability of non-governmental
funds, and availability of star faculty.
- Senior faculty are motivated to remain active
as scholars by the intrinsic rewards of mentoring their graduate
students. They also thrive on public recognition of their contributions
to the profession. Interdisciplinary teams can energize faculty
by creating opportunities and stimulating new research.
- The National Research Council rankings in
economics are strongly associated with objective measures of productivity
such as total citations or total pages in refereed journals.
NRC data also establishes a link between size of department and
rank. The large departments in the top 50 tend to have faculty
in a variety of fields, which would seem to discredit the strategy
of building a "unique niche."
- The individual faculty member is motivated
to be productive because of his/her "passion for reputation" and
"taste for originality." The challenge is to find how these attributes
then lead to publications, citations and impact assessments at
the unit-level. "Bibliometrics" is not helpful in actually fostering
- The researcher who is productive over a long
career may experience multiple peaks and valleys as he/she invests
additional time in acquiring new skills and competencies in order
to develop new lines of investigation. The system for evaluating
research productivity at the unit-level must reflect this non-linear
career trajectory at the individual level.
Thomas H. Rosenquist, Vice Chancellor
for Research, University of Nebraska Medical Center
Suzanne Ortega, Dean of the Graduate
School and Vice Provost for Advanced Studies, University of Missouri
K. Michael Welch, Vice Chancellor
for Research, University of Kansas Medical School
SECOND PANEL OF RESEARCH ADMINISTRATORS
- One departmental model for faculty evaluation
is, in a sense, "Darwinian." It quantifies annual research productivity
for all investigators; derives an average; compares each investigator
with the average investigator; and then distributes rewards accordingly.
Investigators who do poorly over time are weeded out, whereas
the strongest, most adaptable in the department, thrive. Whether
the evaluation system is "Darwinian" or "Egalitarian," faculty
members are not completely satisfied; however, it seems that faculty
in departments with a major Darwinian component are more satisfied
than they are in circumstances where the chair gives a highly
subjective, or no, evaluation.
- We haven't determined how to recognize the
quality or productivity of activities that are not for an academic
audience. In evaluating the outcomes of graduate education, how
do we judge the placement of chemistry students who go into non-academic
institutions? Should the standard be placement in Fortune 500
companies? Do we count the number of students who start their
own companies? Do we count patents? Is a number an adequate
indicator of productivity, or do we attach a dollar value? As
universities move in the direction of increased collaboration
with industry, with increased public accountability, and respect
for the wide range of career opportunities for our doctoral degree
recipients, it will become more important to develop assessment
and evaluation strategies that align with the values and goals
of our non-academic audiences.
- We must be careful to develop appropriate
measures of quality and impact in the arts and humanities or we
may erode the position of these disciplines at our institutions,
especially when measures of "impact" drive resource allocation
models in the future. By intention or happenstance, our support
of the arts and humanities will be an important statement about
our institutional values.
- It is difficult to convince faculty that
they should be interested in assessment as a strategy for improving
the things they care about, i.e., the preparation of the next
generation of scholars and researchers; faculty often believe
that administrators actually want a quick and efficient way of
allocating - or more frightening still, reallocating resources.
- For biomedical institutions, the total award
by the National Institutes of Health (NIH) is a measure that meets
key characteristics: it can be measured in a simple, easily understood
and goal-directed manner. It has a strong association with other
markers of research productivity and it is a clear outcome. However,
the use of a productivity index must not be confused with the
goals and values of the institution, which include scholarship,
clinical care, education and service.
- To make NIH funding the gold standard in
an institution, each school must have its own mandate to increase
NIH funding and create a strategic research plan for a 5-7 year
period, with award targets as their goal. Administrators who
set the goals and oversee the process should be held accountable,
using NIH awards as the productivity measure of the programs in
their area of responsibility.
R.W. Trewyn, Vice Provost for Research
and Dean of the Graduate School, Kansas State University
Jack O. Burns, Vice Provost for Research,
University of Missouri at Columbia
James R. Bloedel, Vice Provost for
Research, Iowa State University
A REFLECTION ON A DAY SPENT DISCUSSING
- Those who are concerned about the University's
return on investment are: governing boards, accrediting bodies,
funding agencies, state legislators, taxpayers, prospective students,
employers of students and bill-paying parents. Education is a
value-added product. It creates a significant difference in income
for the student, particularly when comparing the salaries of high
school graduates with those who earned a masters degree. It also
makes a difference to society; we have estimated that Kansas State
University alumni paid $250 million in taxes from the earnings
garnered from a college education. In agriculture, the value of
research can be measured in dollars; for example, KSU high-yield
wheat has been shown to generate $64 million more in income for
- Increasingly we must look at technology transfer
and create new productivity measures: licensing income; licensing-linked
research funding; companies launched and jobs created. In a document
published by Kansas State University in 1998 on the economic impact
of research and teaching, we estimated that the return on investment
is $17 for every dollar.
- The master campus plan at the University
of Missouri - Columbia involves these goals: maximizing internal
resources and communications; enhancing research compliance; providing
grant assistance; nurturing technology development; expanding
external partnerships; and fostering governmental relations.
To make better use of internal resources, MU has established a
campus network of 55 grant writers and a grant information system.
Through its office of Technology and Special Projects, MU provides
mentoring on technology transfer and develops relationships with
economic development entities as it encourages entrepreneurship
- The scholarship of the scientific faculty
is now more diverse. Many individuals are interested in the wide-range
of experiences that result from entrepreneurial activities, not
as a substitute for their more traditional scholarship activities,
but rather as a complement to their professional experiences while
serving our institutions. Their contributions not only add to
the research culture on our campuses, they also provide unique
training opportunities for our undergraduate and graduate students.
These training opportunities support current trends in graduate
education that emphasize the importance of meeting the needs of
students interested in careers in industry.
- If we are to attract and retain faculty who
are entrepreneurial, a broader definition of productivity is needed;
"objective-driven scholarship" can apply to educational initiatives
as well as extension activities. Evaluation would then be based
on "impact on the field." To meet this standard, the faculty
must demonstrate a set of contributions that has impacted a field
in a way that modified thinking and/or trends among other scholars
in the same area. For Promotion and Tenure, I suggest setting
up external study sections with experts chosen on the basis of
their capacity to assess the impact of research. This process
would parallel the one established by NIH and NSF for evaluating
grants and contracts.
David Shulenburger, Provost,
University of Kansas
HIGHER EDUCATION ADVOCACY: THE INTERFACE
OF TWO CULTURES
- A market model of evaluating our productivity
does not work unless universities can demonstrate that they are
covering the full cost - and yet all our activities are subsidized.
This is why we fall short in using measures such as the quantity
of external funding.
- Our arguments about state funding and higher
education may not succeed because the public knows our contributions
are not unique. Higher education does improve an individual's
income, but if our state did not support universities, students
would be able to seek an education elsewhere. Likewise, the public
may not accept the argument that our institutions give a good
rate of return on monies invested because we cannot say that the
leverage we provide for investment in higher education is really
better than the benefits derived from money spent on traffic safety
or early childhood education, etc.
- The public relies on U.S. News and World
Report for information on college rankings, not the National
Research Council. The value of the process is further jeopardized
if our evaluation schemes do not measure up to our ideals, and
our faculty don't believe in the process.
Kim A. Wilcox, Executive Director,
Kansas Board of Regents
- There are many differences between the culture
of academe and that of the legislature. It is difficult to understand
the way compromise is reached in the legislature, but it is critical
to accomplishing their objectives.
- We need to spend more time thinking about
what it is we are doing in academia and how our work can be cast
into an appropriate form.
- The universities need a focused message in
communicating with the legislature. Too often, we find ourselves
espousing our own individual needs and positions. It is important
to unify our voices and if we do this, it makes it easier for
the press to espouse our position and for the legislature to accomplish
- Our long-term work with legislators should
combine "friend raising" with fund raising.
Rollin C. Richmond, Provost, Iowa
James Coffman, Provost, Kansas State
- Academics in the developed world have contented
themselves for many centuries with the same approaches to education
as they themselves experienced. Few scholars have read the literature
on learning styles and best practices for teaching.
- We are likely to experience increasing difficulties
in attracting public and private support for our institutions
unless we change the way that we reward academic scholarship.
Scholarship can be viewed from a broad perspective. It is integral
to all three components of higher education: learning, discovery
and engagement. Scholarship can be communicated through: teaching
materials and methods, classes and curricula as well as publications,
presentations, exhibits, performances, and also patents, copyrights
and the web. We need to place scholarship in the context of the
institution its serves, not just the discipline it supports.
- Applied science is what John Maddox says
has dramatically changed and improved the lives of people in the
twentieth century. Iowa State University has invested in applied
science via the Plant Sciences Institute.
- Funding patterns for higher education in
Kansas reflect more than 100 years of decisions to foster a high
participation rate via community colleges and technical schools
as well as the regional and research universities. In an economic
development context, we see a difference in funding priorities
in Nebraska, Iowa, and Missouri where more investment is being
made in research universities
- While every research university works to
the limits of its ability to expand research and development,
this happens in a context in which education retains primacy.
The federal agenda for academia is focused on research, and the
state agenda is founded and evaluated primarily on the basis of
undergraduate education. These two forces frequently are in conflict.
- The standard model of the complete scholar
is too constraining to be affordable. Not everyone is able to
maintain a research output that is nationally competitive, and
even fewer can establish and maintain a national reputation.
While it is in everyone's best interest to celebrate and capitalize
upon those who can produce optimally in teaching, research and
service, we should recognize that not everyone can do this over
the entire course of a career. It is most effective to create
flexibility in roles and rewards so that work can be allocated
according to an individual's strengths, especially during the