Two years ago I discovered an attempt at fabrication and misrepresentation of data in a manuscript which had been given to me to read ahead of its planned submission. (I want to make clear at the outset that no staff or student of Uppsala University had any role in perpetrating this fraud.)
One of the corresponding authors of the manuscript asked me to read and give my opinion on it, since I had more experience on the specific topic (related to materials for a rechargeable battery). From what I knew about the work previously I suspected the electrochemical measurement (battery test) data as being a little “too good”, even though it looked plausible. I had access to the raw data, so I analysed it myself. My instincts were correct, but I was staggered by just how “too good” the results were. The electrodes being made were not even close to the specification claimed in the experimental section, with capacities actually an order of magnitude lower than claimed. Data points were moved or removed to reduce the appearance of capacity fade. Data points were even added to make the tests look like they had run more cycles than they actually had. And there was no reproducibility in the experiments – the few tests presented were more or less the only ones that even worked.
Essentially, the results could not have been more misleading if they had been fabricated in their entirety. I raised this immediately with the author who had sent me the manuscript in the first place, and it soon became clear this was deliberate misconduct on the part of the first author, a PhD student. After that the matter was out of my hands – the student was at a different institution in a different country. The manuscript, of course, was now not going to be submitted. Had that data been published, it would almost certainly have got through peer review without suspicion.
So why am I writing about this now? Because I discovered recently that the work is now published in a journal of high repute in the battery materials field. It’s changed a bit from the manuscript I read – all the battery test data is new, and there are some other new experiments. But it is, to a large extent, the same work with the same first author – now at a different institution – with a lot of the same data and text. On top of that, the original corresponding authors are not co-authors anymore.
What the original authors do about this is now for them to decide, but this leaves me wondering what I should do with this knowledge. None of the data I could prove was altered and misrepresented is in the published version (it is for this reason that I am not naming names or the paper, though I can still prove the original fraud). The data in the published version could well be acquired and presented honestly and accurately, for all I know.
But then this is the problem: I don’t know, I have no way of knowing, and nor does anyone else who reads it. But what concerns me now is that this is only one bad symptom of a wider disease.
Many battery scientists would probably agree with me when I say that there are a lot of papers published which are missing important experimental details, making them impossible to reproduce properly. Raw data for battery tests is very rarely, if ever, made readily available. The practice of cherry-picking results from the cells which work best – and which are therefore not necessarily representative – is I suspect rather widespread.
Also problematic – although not any kind of misconduct – is a general lack of knowledge and experience with electrochemical methods for batteries and battery materials, and that this is probably part of the reason why important details are missing. For example, many researchers do not fully grasp the interplay of different, seemingly-minor variables – such as electrolyte volume, thickness of the lithium foil in half cells, electrode thickness, quality of electrode preparation, amount or type of binder, and more – on rate capability, capacity fade and so on. This routinely leads to conclusions which aren’t supported by the data. This should get picked up in the course of peer review, but then papers are reviewed by other researchers with a similar lack of experience in the area, and very quickly poor practices propagate through the literature. This can be the case regardless of the reputation of the PI and the impact factor of the journals the work gets published in.
Couple this with the perverse incentives surrounding publishing already – publish or perish, the well-observed tendency for “high impact” journals to publish eye-catching but not necessarily rigorous results, and career or even financial rewards for publishing in such journals, and so on – and clearly we have a problem.
The sum of all this – unavailability of raw data, normalisation of bad practices, bad incentives – is that it is hard not only to spot manipulation of battery test data but to even separate it from mere poor practice. How widespread is data manipulation or other practices done in bad faith, when the incentives are there and the chances of getting away with it are high? What fraction of papers published in the literature are actually reproducible, and make conclusions which are truly warranted by the results?
This situation surely has to change for the field as a whole to retain credibility, and I am increasingly of the opinion that open data is required. This would make fraud such as the type I discovered more difficult, but also allow other researchers to verify for themselves the quality of results, as well as to explore the data more deeply than conventional selected plots can allow.
That is why I, with the support of my PhD student, will now commit to releasing openly all raw and processed battery test data at the time of publication (at least where I am corresponding author, and as far as possible otherwise), and I will work to encourage my colleagues to do the same elsewhere. I do not know yet the best way to do this, but I hope to think of a useful solution in the coming months. In the meantime, I welcome any feedback on how this might best be done.
I also hope that other researchers in the field carefully consider the importance of how battery data is treated and presented, as well as the necessity to be absolutely clear about all experimental parameters and the validity of control experiments - and how important it is as reviewers to ensure this. As a field, we should have better standards.
comments powered by Disqus