Who's afraid of Impact Factors?
In 2008 British academia faces another Research Assessment Exercise (RAE). Scientists and institutions fear the panels and suspect them of using flawed measures such as a journal's Impact Factors to evaluate the work of individuals. Clinicians especially claim medicine has been severely damaged by the way government has assessed research quality and allocated funding in the last twenty years. Brynja Aadam-Radmanic.
(Apr 2nd 2007) Since 1986, the results of the Research Assessment Exercises (RAE's) have determined how research funding worth billions of pounds is shared between UK institutions. Until 30 November 2007 each researcher again submits hers or his four best pieces of work for next years round of assessments. From early to mid-2008 panels of experts will then scrutinise enormous amounts of articles, books, patents and reports on behalf of the four funding councils of the UK (HEFCE in England, SFC in Scotland, HEFCW in Wales and DEL in Northern Ireland), the results will be published in December 2008.
Given the importance of the process for institutional funding, researchers are trying hard to anticipate the principles governing their fate. Many UK scientists are deeply convinced that their papers will be judged by the reputation of the journal in which they are published, irrespective of the quality of the individual paper. A Nature-paper is suspected to do more for the rating of the author than the same paper would, when published elsewhere.
The ranking of institutions by previous RAE's is thought to reflect the impact factors of a journal more than it does the papers' content. They feel that publishing a good paper anywhere except in the journal with the best impact factor it can reach will be punished by lower RAE ranking and result in withdrawal of research money.
If this were true, then this measure constitutes abuse. Adding impact factors was designed as a shortcut for librarians to help them choose which journals to subscribe to, not to influence research funding. Impact factors were never meant to be used when comparing authors. They show how often the average paper in a journal is cited by others. Although this is a good measure for the quality of a journal, it has no meaning for an individual article because only a small fraction of all papers make up the bulk of citations a journal achieves. For individual authors there is often no correlation at all between the rate of citations their papers actually receive and the impact factor of the journal in which they are published. Their Nature-papers don't automatically have more impact than their other papers.
RAE officials also deny that journal impact factors form any basis for the weighting of research quality. Explaining the methods and criteria used, RAE documents say repeatedly: “No panel will use journal impact factors as a proxy measure for assessing quality.” Is the fear of the scientists a chimera?
In a report in 2004 the House of Commons stated: “Whether or not RAE panels use journal impact factors as an indication of the quality of the articles that they assess, the perception that this is the case causes a bias amongst UK authors towards journals with higher impact factors.” The House of Commons therefore urged “HEFCE to remind RAE panels that they are obliged to assess the quality of the content of individual articles, not the reputation of the journal in which they are published.”
The government responded to this by once again emphasising that “there will certainly be no mechanistic link between the medium of publication and quality assessment”. However, they state panels are allowed to use the reputation of journals to decide which articles to have a better look at. They do not necessarily need to read papers from high quality journals “but should give greater emphasis to reading and assessing work published in new or unfamiliar media.”
As reasonable as this sounds given the work load of panels, it will continue to fuel the obsession with impact factors and the mistrust in the RAE process. Judged by the discussion in the current issue of the British Medical Journal (BMJ) this reaction of researchers and institutions seems to be the biggest problem. Even if the RAE ranking itself causes no harm, the universities “trying to second guess the research assessment exercise” actually do. In “selectively encouraging research that achieves publication in high impact factor journals” UK's universities have caused severe detrimental effects on academic clinical medicine, as cited in BMJ by the chair of the academic staff committee of the British Medical Association (BMA), Michael Rees.
Universities favour basic research which causes the funding for clinical investigation to decrease because “laboratory based research gets published in journals with generally higher impact factors than their clinical counterparts”. “There has been a haemorrhage of clinical academic staff from universities during the past 10 years-mirroring the existence of the research assessment exercise-and wide ranging cuts in specialist teaching available in medical schools”, BMJ reports.
As every discipline has its own average impact factor without any meaning for quality, the unfair comparison of impact factors across research fields is a race clinicians can't win. Hopefully the plans of the British government to introduce a mainly metrics based system of assessment after 2008 will help stop these damaging practices. A transparent automated system to judge the impact of authors and papers individually may help to overcome the fear of human prejudice - justified or not.
