Showing posts with label pseudo. Show all posts
Showing posts with label pseudo. Show all posts
Friday 13 October 2023
Saturday 6 June 2020
Scientific or Pseudo Knowledge? How Lancet's reputation was destroyed
The now retracted paper halted hydroxychloroquine trials. Studies like this determine how people live or die tomorrow writes James Heathers in The Guardian
---
It is natural to ask how this is possible. How did a paper of such consequence get discarded like a used tissue by some of its authors only days after publication? If the authors don’t trust it now, how did it get published in the first place?
The answer is quite simple. It happened because peer review, the formal process of reviewing scientific work before it is accepted for publication, is not designed to detect anomalous data. It makes no difference if the anomalies are due to inaccuracies, miscalculations, or outright fraud. This is not what peer review is for. While it is the internationally recognised badge of “settled science”, its value is far more complicated.
At its best, peer review is a slow and careful evaluation of new research by appropriate experts. It involves multiple rounds of revision that removes errors, strengthens analyses, and noticeably improves manuscripts.
At its worst, it is merely window dressing that gives the unwarranted appearance of authority, a cursory process which confers no real value, enforces orthodoxy, and overlooks both obvious analytical problems and outright fraud entirely.
Regardless of how any individual paper is reviewed – and the experience is usually somewhere between the above extremes – the sad truth is peer review in its entirety is struggling, and retractions like this drag its flaws into an incredibly bright spotlight.
The ballistics of this problem are well known. To start with, peer review is entirely unrewarded. The internal currency of science consists entirely of producing new papers, which form the cornerstone of your scientific reputation. There is no emphasis on reviewing the work of others. If you spend several days in a continuous back-and-forth technical exchange with authors, trying to improve their manuscript, adding new analyses, shoring up conclusions, no one will ever know your name. Neither are you paid. Peer review originally fitted under an amorphous idea of academic “service” – the tasks that scientists were supposed to perform as members of their community. This is a nice idea, but is almost invariably maintained by researchers with excellent job security. Some senior scientists are notorious for peer reviewing manuscripts rarely or even never – because it interferes with the task of producing more of their own research.
However, even if reliable volunteers for peer review can be found, it is increasingly clear that it is insufficient. The vast majority of peer-reviewed articles are never checked for any form of analytical consistency, nor can they be – journals do not require manuscripts to have accompanying data or analytical code and often will not help you obtain them from authors if you wish to see them. Authors usually have zero formal, moral, or legal requirements to share the data and analytical methods behind their experiments. Finally, if you locate a problem in a published paper and bring it to either of these parties, often the median response is no response at all – silence.
This is usually not because authors or editors are negligent or uncaring. Usually, it is because they are trying to keep up with the component difficulties of keeping their scientific careers and journals respectively afloat. Unfortunately, those goals are directly in opposition – authors publishing as much as possible means back-breaking amounts of submissions for journals. Increasingly time-poor researchers, busy with their own publications, often decline invitations to review. Subsequently, peer review is then cursory or non-analytical.
And even still, we often muddle through. Until we encounter extraordinary circumstances.
Peer review during a pandemic faces a brutal dilemma – the moral importance of releasing important information with planetary consequences quickly, versus the scientific importance of evaluating the presented work fully – while trying to recruit scientists, already busier than usual due to their disrupted lives, to review work for free. And, after this process is complete, publications face immediate scrutiny by a much larger group of engaged scientific readers than usual, who treat publications which affect the health of every living human being with the scrutiny they deserve.
The consequences are extreme. The consequences for any of us, on discovering a persistent cough and respiratory difficulties, are directly determined by this research. Papers like today’s retraction determine how people live or die tomorrow. They affect what drugs are recommended, what treatments are available, and how we get them sooner.
The immediate solution to this problem of extreme opacity, which allows flawed papers to hide in plain sight, has been advocated for years: require more transparency, mandate more scrutiny. Prioritise publishing papers which present data and analytical code alongside a manuscript. Re-analyse papers for their accuracy before publication, instead of just assessing their potential importance. Engage expert statistical reviewers where necessary, pay them if you must. Be immediately responsive to criticism, and enforce this same standard on authors. The alternative is more retractions, more missteps, more wasted time, more loss of public trust … and more death.
‘At its best, peer review is a slow and careful evaluation of new research by appropriate experts. ... At its worst, it is merely window dressing that gives the unwarranted appearance of authority’. Photograph: George Frey/AFP/Getty Images
The Lancet is one of the oldest and most respected medical journals in the world. Recently, they published an article on Covid patients receiving hydroxychloroquine with a dire conclusion: the drug increases heartbeat irregularities and decreases hospital survival rates. This result was treated as authoritative, and major drug trials were immediately halted – because why treat anyone with an unsafe drug?
Now, that Lancet study has been retracted, withdrawn from the literature entirely, at the request of three of its authors who “can no longer vouch for the veracity of the primary data sources”. Given the seriousness of the topic and the consequences of the paper, this is one of the most consequential retractions in modern history.
The Lancet is one of the oldest and most respected medical journals in the world. Recently, they published an article on Covid patients receiving hydroxychloroquine with a dire conclusion: the drug increases heartbeat irregularities and decreases hospital survival rates. This result was treated as authoritative, and major drug trials were immediately halted – because why treat anyone with an unsafe drug?
Now, that Lancet study has been retracted, withdrawn from the literature entirely, at the request of three of its authors who “can no longer vouch for the veracity of the primary data sources”. Given the seriousness of the topic and the consequences of the paper, this is one of the most consequential retractions in modern history.
---Also watch
---
It is natural to ask how this is possible. How did a paper of such consequence get discarded like a used tissue by some of its authors only days after publication? If the authors don’t trust it now, how did it get published in the first place?
The answer is quite simple. It happened because peer review, the formal process of reviewing scientific work before it is accepted for publication, is not designed to detect anomalous data. It makes no difference if the anomalies are due to inaccuracies, miscalculations, or outright fraud. This is not what peer review is for. While it is the internationally recognised badge of “settled science”, its value is far more complicated.
At its best, peer review is a slow and careful evaluation of new research by appropriate experts. It involves multiple rounds of revision that removes errors, strengthens analyses, and noticeably improves manuscripts.
At its worst, it is merely window dressing that gives the unwarranted appearance of authority, a cursory process which confers no real value, enforces orthodoxy, and overlooks both obvious analytical problems and outright fraud entirely.
Regardless of how any individual paper is reviewed – and the experience is usually somewhere between the above extremes – the sad truth is peer review in its entirety is struggling, and retractions like this drag its flaws into an incredibly bright spotlight.
The ballistics of this problem are well known. To start with, peer review is entirely unrewarded. The internal currency of science consists entirely of producing new papers, which form the cornerstone of your scientific reputation. There is no emphasis on reviewing the work of others. If you spend several days in a continuous back-and-forth technical exchange with authors, trying to improve their manuscript, adding new analyses, shoring up conclusions, no one will ever know your name. Neither are you paid. Peer review originally fitted under an amorphous idea of academic “service” – the tasks that scientists were supposed to perform as members of their community. This is a nice idea, but is almost invariably maintained by researchers with excellent job security. Some senior scientists are notorious for peer reviewing manuscripts rarely or even never – because it interferes with the task of producing more of their own research.
However, even if reliable volunteers for peer review can be found, it is increasingly clear that it is insufficient. The vast majority of peer-reviewed articles are never checked for any form of analytical consistency, nor can they be – journals do not require manuscripts to have accompanying data or analytical code and often will not help you obtain them from authors if you wish to see them. Authors usually have zero formal, moral, or legal requirements to share the data and analytical methods behind their experiments. Finally, if you locate a problem in a published paper and bring it to either of these parties, often the median response is no response at all – silence.
This is usually not because authors or editors are negligent or uncaring. Usually, it is because they are trying to keep up with the component difficulties of keeping their scientific careers and journals respectively afloat. Unfortunately, those goals are directly in opposition – authors publishing as much as possible means back-breaking amounts of submissions for journals. Increasingly time-poor researchers, busy with their own publications, often decline invitations to review. Subsequently, peer review is then cursory or non-analytical.
And even still, we often muddle through. Until we encounter extraordinary circumstances.
Peer review during a pandemic faces a brutal dilemma – the moral importance of releasing important information with planetary consequences quickly, versus the scientific importance of evaluating the presented work fully – while trying to recruit scientists, already busier than usual due to their disrupted lives, to review work for free. And, after this process is complete, publications face immediate scrutiny by a much larger group of engaged scientific readers than usual, who treat publications which affect the health of every living human being with the scrutiny they deserve.
The consequences are extreme. The consequences for any of us, on discovering a persistent cough and respiratory difficulties, are directly determined by this research. Papers like today’s retraction determine how people live or die tomorrow. They affect what drugs are recommended, what treatments are available, and how we get them sooner.
The immediate solution to this problem of extreme opacity, which allows flawed papers to hide in plain sight, has been advocated for years: require more transparency, mandate more scrutiny. Prioritise publishing papers which present data and analytical code alongside a manuscript. Re-analyse papers for their accuracy before publication, instead of just assessing their potential importance. Engage expert statistical reviewers where necessary, pay them if you must. Be immediately responsive to criticism, and enforce this same standard on authors. The alternative is more retractions, more missteps, more wasted time, more loss of public trust … and more death.
Saturday 17 December 2011
Why pretend we know everything? It's time to embrace uncertainty
It is certainty that we need to worry about, as extreme ideologies prosper in these uncertain times
Who knows if David Cameron's refusal to sign the EU treaty is a good thing or not. Photograph: Yves Herman/Reuters
I don't know what I am talking about. And, quite frankly, you should be relieved that I know that I don't know. The world is full of people proclaiming about stuff they don't know much about. My trade depends on it. Pundits, politicians and economists, too, all depend on some kind of bladder-busting meta-analysis to keep us quiet. In fact, they are just winging it.
Too many nights I have watched economists on television being treated with undeserved reverence. "Economics is largely a made-up pseudo-science!" I want to scream. After all, it has been almost entirely useless in predicting the mess we are in. Indeed, by coming up with grotesque calculations whereby rich people's investments were effectively risk-free and financed by the jobs and homes of the poor, many economists were cheerleaders pre-crisis.
This is not another anti-bank rant. It is now self-evident that banks did some bad stuff, but the diplomatic immunity they were granted was not merely political. Anyone who makes out that they know what they are doing and can turn a fast buck and believes, yes really believes, in something – anything, themselves even – is facilitated by society. And, yes, this is usually backed up by a narrative of questionable facts.
What is valued is certainty. What is devalued in such a world is uncertainty. Those who aren't sure are weak. Poor. Faithless. Uncertainty is often worrying and feminised. Real men know real things. So they have been lining up to tell us that David Cameron's refusal to sign the EU treaty is the best thing ever to have happened, or the worst thing ever to have happened, when, actually, no one is quite sure. Reconciling a belief in the democratic process with the recognition that the euro is still in big trouble and Greece may well go anyway, means it is impossible to line up clearly in the Eurosceptic versus Europhile shadow boxing. It is up in the air.
As this year has been a news tsunami, it would be far more interesting to acknowledge what we did not know rather than what we did. Most experts did not predict the riots, the Arab spring, the extent of the economic meltdown. I recall meeting a learned professor in Tel Aviv three years ago who explained that Iran not Egypt was now the centre of the Arab world and everything would start there. Like many others, I thought recession would produce some kind of resistance but had no idea how that would manifest itself. As for the financial crisis, our lack of foresight is mind-boggling.
But in public, and especially in politics, an admission of uncertainty is seen as problematic. At a dinner I attended a few years ago, a young politician was asked a question to which he had no answer. He said: "I don't know about that; I will go away and find out." It was Ed Miliband as it happens, and I was impressed. But the guys I was with crowed: "We got him there!" This relentless reduction of politics to point-scoring, this public-school obsession with certainty, is a turn-off. Look where it leads. Not so long ago, George W Bush said that if America "shows uncertainty and weakness in this decade, the world will drift towards tragedy. This will not happen on my watch." Apart from war, this "certainty" helped to produce the debt crisis.
It is certainty that we need to worry about, as extreme ideologies prosper in these uncertain times. Yet there have always been ways of thinking that properly refute certainty. The school of "weak thought" coming out of Italy via Gianni Vattimo follows a clear line from Nietzsche onwards that pushes against finality, and urges us to understand historical circumstance. "There are no facts, only interpretations and this too is an interpretation," Vattimo has said.
The work of Nassim Taleb also confronts us with the idea that the economic models used by the banks were based on the idea of stability. The ecology of the banking system could not predict risk properly at all (although Taleb did, actually). Then we have a genius such as Zygmunt Bauman, who has long been telling us that we live in "liquid modernity", that individuals can no longer plan careers and progress in linear, certain ways. Yes, it is like the weather: changeable.
Of course those who most understand the value of uncertainty are scientists themselves. As the delightful Jon Butterworth wrote this week, science has nothing to fear from uncertainty. The sexy little Higgs Boson particle, which may have flashed up in the data in Cern (I imagine it as a burlesque sort of particle) has meant we have listened to physicists telling us very excitedly about how much we just don't know.
This has been wonderful. The opposite of political discourse: to hear clever people talking about the limits of their own knowledge. How weighed down is public life with its emphasis on certainty. How dumbed down is belief. The big divides are not between different beliefs, but the differing degree of certitude in which those beliefs are held.
No one knows. No one has the answers. Uncertainty is where we are. It is to be embraced. Christopher Hitchens, when asked which word he had most overused, said he was shocked to find on rereading his work that it was "perhaps".
I love that. Perhaps, right now, is the best word. I'm sure of that. Perhaps.
Subscribe to:
Posts (Atom)