Impactful – Part 1

ab initio

An interesting thing has been coming up a lot recently. Impact Factors [IF’s] – a thing I had been only dimly aware of previously – have been thrown somewhat into a public debate due to some interesting [not entirely sensible] actions performed at Queen Mary’s Physics/Astronomy department (more information). Essentially, the issue breaks down as follows;

• Impact Factors are a statistic related to scientific journals that gives a very approximate idea of how important they are [mathematically, the IF is the mean number of citations per paper published – more than one is good]
• IF’s originated as a way for libraries to allocate budgets – high IF journals first – but have become somewhat of a widely used tool within the academic world
• IF’s are known to be used as a mental short-cut to judging the quality of academic work, and hence an academic (i.e. a research job or funding applicant)
• Sometimes, IF’s have been the sole (or most important) basis for judgement of academics – as is the case in the Queen Mary’s debacle; their professors are being forced into a career structure based almost solely around IF’s.

Since I am a young scientist only starting out on my career, it’s fairly natural to be interested – indeed, worried – about this sort of thing. Having read only the first two sources I linked, I was somewhat unhappy that a scientist should be judged so simply – by a figure of merit that simply didn’t do what it said on the tin.

The Challenge

I was particularly caught by this line;

“…if you use impact factors you are statistically illiterate.”

Clearly, this is an insult to any scientist. No self-respecting researcher would stand for that kind of statement even if it were well substantiated – yet I hadn’t seen anyone specifically disagreeing with the statement. Upon reading the statement again, I decided that this might well be hyperbole – but how seriously should I take it? Clearly the author fervently believes that IF’s are a bad thing (his argument was nonetheless convincing without it), but such outright and drastic statements are a warning light on the great dashboard that is human behaviour.

While at the time I was both incensed at the idea that people would pander to a contrived measure of worth – I saw this as something directly contradicting the spirit of inquisitiveness which makes the world of science such a wonderful place to inhabit and be a part of – I was reminded quite sharply recently that it’s rarely a good idea to jump to conclusions. Naturally, being a scientist and all, I thought it was time to question the accuracy of this hypothesis, and in doing so exercise the very core idea of science that I felt was being violated so thoroughly. This signalled the time to start my own little case study experiment!

I think it’s important to share the process of thought which I went through and to draw out the conclusions in detail, because it’s very easy to get worked up about things like this. Many people do, in fact, and I wonder if all of them have bothered to think through carefully and without prejudice on the issue – I reckon probably not. Hopefully I will be able to give you some sane evaluation of the issue and sum it up appropriately.

The story of how I investigated Impact Factors and how they influenced the way I think about science isn’t short enough for this post, so this is going to be a multi-part series. Next time, in Impactful – Part 2; what exactly I found out, and how that started to affect myself and the way I thought of IF’s. Some new characters enter our story, and we get lots and lots of lovely juicy opinions to dissect.