In defence of metrics


Scholars have long debated the merits and pitfalls of metric-based evaluation – the pages of Scientometrics and JASIS&T contain dozens of analyses and variations of the h-index (to name just one metric). But in recent years, bibliometric data have gained attention beyond the scope of the scholars who study them.

As research funding fails to keep up with the growth of institutions and researchers, competition for that funding becomes increasingly fierce and purse-holders turn to metrics to inform their decisions. And those using or interacting with universities also want to compare and distinguish between institutions. Governments, students, ranking companies, bibliometric providers, industry and funding bodies are all driving the demand for methods to quantify research outputs.

calipers measuring people

Academics should suggest suitable metrics before others are imposed on them © Shutterstock

Metric system

In short, metrics are here to stay. Nonetheless, their ubiquity brings challenges, and many researchers express misgivings about, or even resent, the growing emphasis on metrics. Perhaps chief among these issues is deciding exactly what to measure. For instance, can we compare two universities in the discipline ‘chemistry’ if one department has mostly analytical chemists and the other, largely biochemists? These two subfields have vastly different rates and modes of obtaining funding and disseminating research, yet most quantitative methods operate at the ‘discipline’ level, insensitive to these variations, creating potentially dubious comparisons.

Another challenge is choosing the time frame. For example, a university administrator seeking to measure the impact of his or her policies is best served by assessment data that are current and frequent. With data extending back three, five or 10 years, they can measure (after only one or two years) the impact of their strategies. But a researcher’s 30-year career citation count is unnecessary. Conversely, full career histories are essential if we want to characterise the overall research prowess of an academic unit. Unfortunately, the relative ‘academic age’ of a unit is often overlooked in measurement schemes and metric-based rankings. This disadvantages relatively young departments, and can mask declining performance in formerly productive units.

Perhaps the most often-cited challenge is that some things simply cannot be measured, or cannot be adequately measured. A research university may be said to have three missions: teaching, research and service. However, only teaching and research are ‘measurable’, and in both cases it is not clear that the ultimate, and arguably the most important, impacts are captured (for example, student success after graduation, or local economic or public health benefits). Moreover, among the measurable aspects of research (articles and citations, awards won, funding obtained, conference proceedings presented, years since PhD etc) there are some disciplines in which these metrics are unfit indicators of a scholar’s activity. For example, performing arts and other disciplines where exhibition is a major means of dissemination are challenging to quantify because the breadth of opportunities and available venues for a performance or exhibition is tremendous. And, given the subjective nature of the material, it is unclear whether collecting records of such exhibitions is even a fruitful pursuit.

Data with discipline

The challenges are substantial, and include many others not mentioned here; but this should not stop us from using metrics. Moreover, given the wealth of data now available, it is the academy itself – the body of researchers trained in making and improving observations – that is best-placed to refine metrics and create new ones. The discussion within the ivory tower must change from ‘should we measure?’, to ‘what should we measure?’ Disciplinary experts have a window, possibly open only briefly, to determine the best way to measure themselves before others, less informed, impose metrics upon them.

This will also ensure that metrics are not simply a burden to be endured, but a genuinely useful tool. A refined taxonomy of subfields, taking into account researchers’ academic ages, collaborators, publishing habits and countless other data points can be extracted from existing sources. These data enable predictive models and allow fair comparisons to be made between disciplines, institutions and countries. Such specific metrics and comparisons, honed by the scholars who know the discipline, are a positive force. They can be used to promote the merits of a group’s research when applying for funding, to create an objective framework within which to mentor younger scholars and to guide administrative decisions.

But perhaps the greatest promise of metrics lies in cost saving. Assessment exercises such as the UK’s Research Excellence Framework and the US National Research Council’s Assessment of Research-Doctorate Programs require huge investments of time and money, often from the institutions being evaluated. There may never be a metric-based replacement for the judicious evaluations of expert panels and survey respondents, but supplementing them with well-chosen data could vastly reduce the time and expense.

If predictive models can accurately reproduce the results of human deliberation in these studies, then it is difficult to justify the costs of employing scholars in these extracurricular exercises.

Anthony Olejniczak is co-founder and chief knowledge officer at Academic Analytics


Related Content

Can research quality be predicted by metrics?

15 December 2014 News and Analysis

news image

Prognostications unveiled for how UK universities will fare in latest assessment exercise

Metrics' role in assessing research reviewed

9 April 2014 News and Analysis

news image

UK higher education body launches investigation into the pros and cons of metrics in managing research

Most Read

Antimicrobial resistance will kill 300 million by 2050 without action

16 December 2014 News and Analysis

news image

UK report says resistance will cost global economy $100 trillion

Cutting edge chemistry in 2014

10 December 2014 Research

news image

We take a look back at the year's most interesting chemical science stories

Most Commented

Smart skin for prosthetic limbs senses heat and touch

12 December 2014 Research

news image

Ultra-thin plastic skin can bend and flex without affecting the skin's ability to detect stimuli

Chemistry behind the ‘blue man’ unlocked

1 November 2012 Research

news image

Biochemical model suggests that silver ions, not nanoparticles, cause a rare skin complaint