Don't put a cart in front of the horse: latest trends and a practitioner's view on research evaluation metrics and standards

Abstract:

Citation counts and other publication-based metrics are widely adopted as evaluation indicators despite the long-standing debate and criticisms. Generally, those indicators are considered to complement expert's judgments in the peer-review process. However, they can lead to unintended consequences if set as primary requirements for academic promotion, and the impacts on researcher behaviors are foremost because they are compelled to comply with such policies for their job security and academic advancement. There is also a wider impact on the scholarly communication ecosystem, where certain journals and commercial services are automatically given priorities, and it discourages healthy market competition as a result.

In early 2020, Chinese science and education ministries announced new guidelines for research institutions to stop evaluating researchers based on the number of published papers and citations. The guidelines include regulating the use of particular standards and indicators, notably the Journal Impact Factor and Science Citation Index, and set new priorities to the societal impact of research as well as local relevance by whitelisting qualified Chinese journals. Not only in China but globally, there has been a shift from using single-point metrics in research evaluation to more context-based, autonomous, and broader measures of research impact, as observed in the San Francisco Declaration on Research Assessment (DORA) and other global initiatives.

In this contribution, I will explore the possible consequences of such changes in policy and regulation by drawing examples from the latest publication practice and trends. The session participants are encouraged to share ideas on how the research community can effectively set their own priorities through active and continuous dialogue.