Evaluating Research Productivity
no. 105 - June 2001

ACHIEVING THE "GOLD STANDARD": A REDUCTIONISM DIRECTIVE

K. Michael Welch
Vice Chancellor for Research
University of Kansas Medical School

Introduction

This paper takes the position that, from an administrative point of view, research productivity must be measured in a simple, easily understood, well rationalized and goal directed manner. I will put the case that, for biomedical institutions, total NIH award (from the National Institutes of Health) is the measure that meets these key characteristics. This simple index is a compound of all other quality measures, each of which correlate strongly with the overall index. The paper uses the example of biomedical research conducted in institutions that comprise a medical center.

Productivity Considerations in Medical Institutions

Administrators must first set the goal define the audience , enunciate the values and consider the practicalities that may determine the process of achieving the goals. The most commonly articulated goal of most biomedical institutions is to become a leader among peers in biomedical research. Their audience is not only the community of medical science, but also an expanded interest sphere that includes health care delivery systems whose viability may depend on recognition of research excellence being inextricably entwined with the highest standard of health care. More and more the public has joined the audience as it uses science in everyday life, thirsts for biomedical information and demands more of its health care systems. All biomedical institutions should strive for the values of delivering the highest standard of health care, educating the best providers and creating new knowledge. Any institution designated as a research university must achieve its goals while dealing with contemporary challenges in providing healthcare and educating health professionals. Practical issues of size, funding environment and budget must be taken into account.

Biomedical Research Productivity: A Top-down Approach

When taking a top down approach, always involve faculty in decision-making and communication of goals and process. Where can you get a better, already paid for, group of consultants?

First set the goal. If the University of Kansas wishes to move into the top twenty-five public universities for example, it must understand what it takes to get there and define the productivity measures that best fit the goal. Arguably, the top biomedical research universities are defined by NIH award ranking. Therefore, the best measure of productivity is the amount NIH awards to an institution through its granting programs. Accordingly, all productivity should be directed to the goal of achieving NIH funding.

The rationale for using NIH awards as the sole productivity measure includes the simplicity of having one clearly understood outcome. NIH funding is generally accepted as the "gold standard" of quality and the measure used by most major ranking authorities. Further, no matter what the measures used for research productivity, they all flow towards the NIH award. For example, there are strong associations with other markers of research productivity such as alternative funding, publications and markers of national recognition. Thus, total NIH award is a compound or global index that reduces complexity of measurements. As a marker of new knowledge, it also correlates with clinical excellence by enabling the acquisition of cutting-edge scientific information to enhance clinical practice. In and of itself, it is also a marker of research infrastructure growth through the indirect cost mechanism. Since the ranking of medical establishments by NSF (the National Science Foundation) using NIH funding is very similar to their ranking by the popular magazine U.S. News and World Report, the public audience that we must engage for approval of research funding may have a closer understanding of this measure than more complex academic measures of productivity.  

Implementing the Process

Each school in the health center should be given a separate mandate to increase NIH funding as its foremost goal. Each should complete a strategic research plan for a five to seven year period. Realistic dollar award targets must be set, knowing also that movement up the ranks by institutions competing for the same fund pool demands planning, not to just keep pace but to move ahead. Individual departments or centers must buy in to the mandate that NIH awards are the ultimate measure of productivity. The departments/centers should be structured to provide an appropriate balance of activities that should support the overall values of the school, emphasizing integration and collaboration. For example, a clinical department might be designated to focus on clinical care, which will assist the school in its research agenda through the residual income it generates. Promotions and Tenure Committee policies and procedures must take into account the essential value that an individual faculty member brings to the research mission of the school when not engaged directly in research, but rather in clinical, educational and service activities.

Department/Center Strategy

The research strategy is accomplished at the level of department/center. The Chair/Director must be given autonomy in setting up and maintaining the process. The value of every faculty member must be recognized as contributing to research. Roles and responsibilities must be clearly defined for each individual, with the prime understanding of the contribution to overall research productivity. Accordingly, all incentive programs should be directed to the department faculty as a team.

Administrator Productivity

Administrators who set the goals, generate strategic plans and oversee the process must be held accountable, using NIH awards as the productivity measure of programs in their area of responsibility. 

Caveat

The use of a productivity index must not be confused with the goals and values of the institution. Such an index is simply a chosen measure of how the mission of the institution is best monitored. A mission of scholarship and balance in clinical care, education and service for example, is not directly articulated by such an index. A productivity index is part of process, although it should reflect how well philosophy, mission and policy are accomplished.

Conclusion

Vice Chancellor Barnhill, in an address to the conference on productivity measures, expressed his preference to replace the attitude of "only the paranoid survive" with "only the flexible thrive," to which in turn I would add "with a like-minded inflexible goal." If one goal of the institution becomes achieving top university status, and this is equated with NIH award ranking, the level of NIH funding received would seem a logical comprehensive measure of research productivity.