In June 2006 the Department for Education and Skills (DfES) launched a consultation on proposals to move to a system of assessing university research based on funding income with no element of peer review. The RSC responded, and along with many other professional bodies opposed the suggested changes for the next assessment exercise, planned for 2008.

At present universities receive research funding through a block grant provided by the higher education funding councils. This has a quality related (QR) component which is informed by the results of the Research Assessment Exercise (RAE). It is the mechanism of this process for grading university research which is under review. 

20 years of assessment

The first RAE (then called the Research Selectivity Exercise) was organised in 1986 and the process repeated 3 years later in 1989. In 1992, the process grew to cover not only those institutions that had taken part in the previous two exercises, but also those previously funded through the Polytechnics and Colleges Funding Council, together with a small number of other institutions. Further RAEs were held in 1996 and 2001.

The outcomes of these assessments were used as the main element of a formula to allocate funding for research. (This remains separate to the process for allocating research council project grants to individual researchers).

The process involves the assessment of data submitted by university departments. A series of subject panels assess the submitted information. The fifth exercise in 2001 was the most rigorous exercise to date. It considered the work of almost 50,000 researchers in 2,598 submissions from 173 institutions. Original plans for the 2008 exercise suggested the assessment covered research output (publications), research environment (such as departmental strategy and income) and esteem indicators (such as awards). 

The net effect of successive RAEs has been to increase the quality of research but also to concentrate resources into a smaller number of institutions - in other words, to increase selectivity. This was in fact precisely the intension when the system was set up. In the context of chemistry, there is little doubt that changes in funding after successive RAEs have contributed to some departmental closures. 

Alternative methods

While it's undeniable that the present system has been successful in raising research quality, the RAE approach has attracted increased criticism in recent years. It is alleged that the scale and cost now outweighs its ongoing value and that more efficient alternative methods for assessing research quality can be found. It has also been observed that the distribution of QR funding is well correlated with research income from other sources. 

There is a also a contention that the workings of the system, and in particular the research communities perceptions of RAE panel expectations, may inhibit particular approaches to research.  

The end of the RAE as we know it?

Along with the 2006 Budget statement, the Government published a document titled Science and innovation investment framework: next steps. In it, they stated their intention to abolish the RAE.  

It stated that the Government's 'firm presumption' is that after the 2008 RAE QR allocations will be allocated on the basis of quantitative indicators or 'metrics'. The only indicators that were mentioned were those based upon income from research funders such as the UK Research Councils and UK industry. 

This lead to some confusion as to whether the 2008 RAE would in fact go ahead at all. But following some debate, it was confirmed in December 2006 that the exercise will go ahead as previously planned. 

"the net effect of successive RAEs has been to concentrate resources into a smaller number of institutions"

The DfES July consultation proposed that the 2008 RAE might be changed to include a metrics-based element and that after this, future assessments move to a system on 'metrics' alone with no peer review. Specifically the consultation proposed that the metric to be used would be research income and went on to propose a number of different research income-based assessment systems.

Metrics are not the answer

Metrics alone will skew research. In its response to the consultation, the RSC urged the DfES to stick with the original published procedures for the 2008 RAE panels. We are confident that the criteria already laid down will result in a sound assessment of research.

None of the systems proposed in the consultation, nor indeed any system solely based on input metrics such as research income, are suitable for assessing research quality. We believe that, whilst metrics can play a part in the assessment of research, the RAE 2008 chemistry panel, or indeed any other future subject panel, should not adopt a greater or wholly metrics-based approach which diminishes the role of peer review in the assessment process.

One problem with any system based essentially on one measure is the likely behavioural consequences. With such a system institutions are likely to put emphasis on their staff winning research contracts and grants rather than on writing up the research carried out. 

It is also possible that research managers will favour expensive research fields as these are likely to generate more QR funds and this will result in a flow of funds into expensive research. The increased emphasis on winning grants will also discourage researchers from pursuing innovative high risk proposals. Such behaviours cannot be good for the standing of UK research. 

Does input match output?

There is also a philosophical problem with the proposed systems in that it relies on an input measure (income) rather than on the quality of output. In extremis some areas of academic work may produce high quality outputs with very little research income and such areas will be unfairly treated in any model based solely on research income.

Using metrics alone misses academic activities associated with research that are unrelated to funding.  Such esteem indicators are captured currently in the RAE process. In fact, for institutions with less research funding, the correlation between research grant funding and research quality, as measured by the last RAE, is weak. 

Peer review is essential

The RSC believes that the any system of research assessment should include an element of peer review. A package of metrics including output and input measures, as well as activity (or volume) measures, could be useful, but only in supplementing peer review.

There may be a place for a light touch metrics approach that might be used between full peer reviewed based or metric informed peer reviewed based assessment exercises, or as a trigger for more detailed reviews.
The use of a system incorporating peer review and a package of metrics will make it more difficult for there to be a significant shift in behaviour as behavioural changes to optimise one metric will not necessarily affect other metrics in a positive way.  

One model for all?

One major issue that the RSC and others have flagged up is that of the differences within the academic community. Can one assessment model actually be designed to fit all disciplines?

Output measures such as citations, that could form part of a metrics package, cannot by themselves be used as a raw measure of quality. Even different sub-disciplines of subjects display different publication and citation behaviour. At a simple level the citation rate depends on the size of a community:  the larger the community the greater the citation rates observed. Any model based on metrics alone would be insensitive to such differences.

Similarly, different sub-disciplines within chemistry require different levels of financial support. Consequently, if research income were the main measure of research quality, a small department which has research mainly in a cheaper chemistry sub-discipline will apparently have lower quality research than one which concentrates on a more expensive sub-discipline. 

"any system of research assessment should include an element of peer review"

It is likely that different subjects, or groups of subjects, would be better suited to different mixes of metrics. If metrics are introduced, the RSC sees the need for panels to have the flexibility to define how they use them in consultation with their respective communities. As long as all panels are working to the same general principles, they should be able to chose the criteria that work best for their particular disciplines.

The exact mix of metrics chosen must be balanced between inputs and outputs and 'activity' measures, rather like the system in place for the 2008 RAE.

Assessment is here to stay

Successive RAEs have brought refinements and improvements to the 1986 format. Nevertheless, despite the passage of 20 years, the massive expansion of the sector, the effects of devolution and several formal reviews (most recently in 2003 by Professor Sir Gareth Roberts), the salient features of the 2008 RAE will still closely resemble those of the 1986 exercise. 

An independent assessment of UK higher education research quality for benchmarking purposes is desirable and necessary. The RAE has driven up research quality and this is a good thing. The RAE was also introduced to allow for greater selectivity in funding. Having introduced this system of selectivity, it is necessary to continue to assess quality so that these indicators can be used to inform the allocation of future QR funding. 

In the December 2006 pre-Budget report, the outcome of the RAE consultation was published, and it was confirmed that after 2008 a new metrics based mechanism will be used. A set of indicators of research income and quality will be devised using advice from experts including researchers themselves. How the DfES and higher education councils will chose to implement the new system remains to be seen.