### Metrics failed to predict REF outcomes

2 April 2015 News and Analysis

Predictions based on numerical measures of research quality were ‘wildly innacurate’, say mathematicians

25 October 2012

‘I don’t like it, you (probably) don’t like it, but let’s face it: the *h* index is here to stay.’ Physicist Michael Schreiber, one of the most diligent critics of the *h* index, put it very appositely.^{1} But a recent analysis by cognitive scientist Daniel Acuna of the Rehabilitation Institute of Chicago, US, and colleagues^{2} implies that the *h* index’s imposition might not stop at reducing a life’s work to a bare number. It can be used to forecast what the future significance of that work will be.

Named after Jorge Hirsch, the physicist who devised this measure of achievement, the *h* index quantifies how many highly cited papers an individual has written: *h* of his or her papers have been cited at least *h* times. Hirsch says that tenured researchers tend to have an *h* index of at least 12 (this varies between disciplines), and top scientists typically 50 or more.

'I am not a number!' The plight of Patrick McGoohan's character in The Prisoner may seem familiar to some © Bettmann/Corbis

Now, with equations perhaps more affronting and presumptuous than even the pseudoscientific prescriptions for the ‘worst day’ or ‘sexiest body’, Acuna and colleagues can predict your *h* index next year, in five years or a decade hence. The first of those equations reads thus:

where *h* is your current index, *n* is the number of papers you’ve written, *y* the number of years since publishing your first article, *j* the number of different journals you’ve published in and *q* the number of your papers in *Nature*, *Science* and certain other leading journals for your discipline (Acuna used neuroscience to devise and test the formulae). Don’t you love the specificity – that opaquely pedantic *n*^{–1/2}? But before you scoff, the formula appears to predict extremely well (at least on paper). The predictions are admittedly (and unsurprisingly) worse the further ahead you look, but it does seem a better indicator of future performance than the *h* index alone.

The equations would need to be adjusted for other disciplines but, on close inspection, the bottom line is that your *h* index next year will probably increase by a fixed amount (around 0.76), tweaked by the number of papers you’ve published, and minutely refined by how broadly and visibly you publish. No surprises there. And of course all the caveats and benefits adduced for the *h* index apply equally to the formulae of Acuna et al, a fact they’d surely not hesitate to concede.

All this might leave you feeling a little like Fyodor Dostoyevsky’s misanthropic narrator in *Notes from the Underground*:

‘If ever there shall be discovered a formula which shall exactly express our wills and whims … what laws they are governed by … what tendencies they follow under given circumstances; if ever there shall be discovered a formula which shall be mathematical in its precision … man will have ceased to have a will of his own – he will have ceased even to exist’

The creeping culture of metric-based assessment leaves many scientists uneasy that they are being measured by the number of *Nature* papers they have rather than by the real impact of their work. Yet faced with a flood of applications for a job, isn’t it better to have some objective, unbiased and quantitative criterion of quality rather than making decisions based on cursory, subjective impressions? Might not such methods help to root out gender and race discrimination?

They might, but it’s not clear that the *h* index or its variants are the ideal answer. Perhaps the real issue is that we should even wish to forecast scientists’ careers. Of course, it has always been pertinent to ask how well a candidate for a post is likely to perform. But the subliminal message of these formulae is that a life in science is a predictable, steady accumulation of influence. For many, perhaps most, scientists there may be some truth in that. But will one trajectory really be differentiated from another by a small difference in a metric, or by an unpredictable insight, discovery or stroke of luck? Or for that matter the ability to inspire students or bridge disciplines? The accretion of terms in Acuna’s formulae, which could continue indefinitely, hints at deeper questions, which the *h* index can’t quite grasp: how does scientific creativity occur? And what is it in a scientist’s life that we truly value?

1 M Schreiber, *Ann. Phys.*, 2007, **16**, 640

2 D E Acuna, S Allesina and K P Kording, *Nature*, 2012, **489**, 201

3 J E Hirsch, *Proc. Natl Acad. Sci. USA*, 2007, **104**, 19193

2 April 2015 News and Analysis

Predictions based on numerical measures of research quality were ‘wildly innacurate’, say mathematicians

15 December 2014 News and Analysis

Prognostications unveiled for how UK universities will fare in latest assessment exercise

23 May 2016 Business

UN and WHO panel conclude the herbicide glyphosate is ‘unlikely’ to cause cancer at realistic exposure levels

24 May 2016 News and Analysis

US funding agency will enlist 5400 people for HIV vaccine study in South Africa in November