Metrics' role in assessing research reviewed


abacus

© Shutterstock

A committee set up by Hefce (Higher Education Funding Council for England) aims to grapple with the thorny issue of using metrics to assess and manage research. Metrics include analysis of journal articles and citations, but have expanded to altmetrics, which track what people are saying about a paper online.

Hefce’s review will consider the role that such metrics might play in determining quality, impact and other key characteristics of research in the higher education sector. But the review will also consider the negative effect of metrics on research culture.

‘Individual researchers are experimenting with different tools, but to what extent do we want metrics to carry real weight in overall evaluations of the research system? That is a serious question to think through,’ says James Wilsdon, professor of science and democracy at the University of Sussex, UK, and chair of the independent review. The committee hopes to complete its work by spring 2015, in time to influence the next assessment of UK universities, the Research Excellence Framework.

The review will consider how a metric approach fits with the missions of universities and research institutes and look at what can, and cannot, be measured quantitatively. The inappropriate use of metrics will also be considered. ‘We want to avoid any rush to adopt particular matrices or indicators that may exacerbate particular negative features of the research system,’ says Wilsdon, citing the inappropriate use of journal impact factors in promotion decisions.

A central question is whether measuring metrics for your activities influences the way you do those activities, says Henry Rzepa, a computational chemist at Imperial College London, UK. The h-index, which looks at a scientist’s most cited papers and how many citations they receive, can be misused by ‘consortiums or cartels’ who agree to cite each other’s papers. ‘This activity has no scientific purpose, but it does mean the members of the cartel benefit by having their h-count increase quite dramatically,’ says Rzepa. ‘I know this happens in chemistry.’

Pharmacologist David Colquhoun at University College London, UK, is even more sceptical: ‘The h-index is terrible. It is biased against young people.’ He argues that you assess quality by reading scientific papers, not by looking at phony metric numbers. He predicts that if the steering group recommends more use of metrics it will result in ‘more dishonesty, more corruption, more gaming’. He has outlined on his blog that many papers topping the altmetrics chart are of little consequence and says this measure may be inversely related to quality.

Colquhoun adds that the biggest corrupting influence is probably impact factors and the importance given to ‘glamour journals’ that are well regarded because of their high impact factors and criticised recently by Nobel prizing winning cell biologist, Randy Schekman. ‘I think almost all practising scientists regard metrics as corrupting honest science. It is the hanger’s on, like Hefce, the government bodies and to some extent research councils [that are taken with them].’


Related Content

Can research quality be predicted by metrics?

15 December 2014 News and Analysis

news image

Prognostications unveiled for how UK universities will fare in latest assessment exercise

June 2014

1 August 2014 Letters

news image

More speculation on the source of sulfur in the Mary Rose, a plea to mitigate the influence of metrics and spontaneous combus...

Most Read

First pictures of hydrogen bonds unveiled

26 September 2013 Research

news image

Observation of intermolecular interactions in quinolines could help to settle the nature of this kind of bonding

Flowing rivers of mercury

7 January 2015 Feature

news image

Philip Ball investigates claims that the burial chamber of China’s first emperor contains rivers of shimmering mercury

Most Commented

SiRNA treatment cures Ebola in monkeys

23 April 2015 Research

news image

Researchers perform the first successful trial for a drug against the current strain of Ebola

Relativity behind mercury's liquidity

21 June 2013 Research

news image

First evidence that relativistic effects are indeed responsible for mercury's low melting point