Impactful – Part 1

ab initio

An interesting thing has been coming up a lot recently. Impact Factors [IF’s] – a thing I had been only dimly aware of previously – have been thrown somewhat into a public debate due to some interesting [not entirely sensible] actions performed at Queen Mary’s Physics/Astronomy department (more information). Essentially, the issue breaks down as follows;

  • Impact Factors are a statistic related to scientific journals that gives a very approximate idea of how important they are [mathematically, the IF is the mean number of citations per paper published – more than one is good]
  • IF’s originated as a way for libraries to allocate budgets – high IF journals first – but have become somewhat of a widely used tool within the academic world
  • IF’s are known to be used as a mental short-cut to judging the quality of academic work, and hence an academic (i.e. a research job or funding applicant)
  • Sometimes, IF’s have been the sole (or most important) basis for judgement of academics – as is the case in the Queen Mary’s debacle; their professors are being forced into a career structure based almost solely around IF’s.

Since I am a young scientist only starting out on my career, it’s fairly natural to be interested – indeed, worried – about this sort of thing. Having read only the first two sources I linked, I was somewhat unhappy that a scientist should be judged so simply – by a figure of merit that simply didn’t do what it said on the tin.

The Challenge

I was particularly caught by this line;

“…if you use impact factors you are statistically illiterate.”

Clearly, this is an insult to any scientist. No self-respecting researcher would stand for that kind of statement even if it were well substantiated – yet I hadn’t seen anyone specifically disagreeing with the statement. Upon reading the statement again, I decided that this might well be hyperbole – but how seriously should I take it? Clearly the author fervently believes that IF’s are a bad thing (his argument was nonetheless convincing without it), but such outright and drastic statements are a warning light on the great dashboard that is human behaviour.

While at the time I was both incensed at the idea that people would pander to a contrived measure of worth – I saw this as something directly contradicting the spirit of inquisitiveness which makes the world of science such a wonderful place to inhabit and be a part of – I was reminded quite sharply recently that it’s rarely a good idea to jump to conclusions. Naturally, being a scientist and all, I thought it was time to question the accuracy of this hypothesis, and in doing so exercise the very core idea of science that I felt was being violated so thoroughly. This signalled the time to start my own little case study experiment!

I think it’s important to share the process of thought which I went through and to draw out the conclusions in detail, because it’s very easy to get worked up about things like this. Many people do, in fact, and I wonder if all of them have bothered to think through carefully and without prejudice on the issue – I reckon probably not. Hopefully I will be able to give you some sane evaluation of the issue and sum it up appropriately.

The story of how I investigated Impact Factors and how they influenced the way I think about science isn’t short enough for this post, so this is going to be a multi-part series. Next time, in Impactful – Part 2; what exactly I found out, and how that started to affect myself and the way I thought of IF’s. Some new characters enter our story, and we get lots and lots of lovely juicy opinions to dissect.


About stoove

A physicist, researcher, and gamesman. Likes to think about the mathematics and mechanics behind all sorts of different things, and writing up the thoughts for you to read. A competent programmer, enjoys public speaking and mechanical keyboards. Has opinions which might even change from time to time.
This entry was posted in General Science, Opinion, Physics and tagged , , , , , , , , , . Bookmark the permalink.

4 Responses to Impactful – Part 1

  1. I am surprised that a physicist should be unaware of the statistical illiteracy of characterising a very highly skewed distribution by its mean. The distribution of citation numbers is FAR more skewed than an exponential distribution.

    • stoove says:

      Well yes. Since I’m a student, I haven’t had to deal with IF’s until now – once I’d read about them I was rather surprised that anyone would be silly enough to treat them as an indicator of merit, but this appears to be so to some extent. The second part of the series (coming on Wednesday) will explore that more fully; to what extent IF’s are taken at face value.

  2. Pingback: Impactful – Part 2 | UNconstant

  3. Pingback: Impactful – Part 3 | UNconstant

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s