Search This Blog

Showing posts with label peer. Show all posts
Showing posts with label peer. Show all posts

Sunday 16 January 2022

Will blockchain fulfil its democratic promise or will it become a tool of big tech?

Engineers are focused on reducing its carbon footprint, ignoring the governance issues raised by the technology writes John Naughton in The Guardian

Illuminated rigs at the Minto cryptocurrency mining centre in Nadvoitsy, Russia. Photograph: Andrey Rudakov/Getty Images



When the cryptocurrency bitcoin first made its appearance in 2009, an interesting divergence of opinions about it rapidly emerged. Journalists tended to regard it as some kind of incomprehensible money-laundering scam, while computer scientists, who were largely agnostic about bitcoin’s prospects, nevertheless thought that the distributed-ledger technology (the so-called blockchain) that underpinned the currency was a Big Idea that could have far-reaching consequences.

In this conviction they were joined by legions of techno-libertarians who viewed the technology as a way of enabling economic life without the oppressive oversight of central banks and other regulatory institutions. Blockchain technology had the potential to change the way we buy and sell, interact with government and verify the authenticity of everything from property titles to organic vegetables. It combined, burbled that well-known revolutionary body Goldman Sachs, “the openness of the internet with the security of cryptography to give everyone a faster, safer way to verify key information and establish trust”. Verily, cryptography would set us free.

At its core, a blockchain is just a ledger – a record of time-stamped transactions. These transactions can be any movement of money, goods or secure data – a purchase at a store, for example, the title to a piece of property, the assignment of an NHS number or a vaccination status, you name it. In the offline world, transactions are verified by some central third party – a government agency, a bank or Visa, say. But a blockchain is a distributed (ie, decentralised) ledger where verification (and therefore trustworthiness) comes not from a central authority but from a consensus of many users of the blockchain that a particular transaction is valid. Verified transactions are gathered into “blocks”, which are then “chained” together using heavy-duty cryptography so that, in principle, any attempt retrospectively to alter the details of a transaction would be visible. And oppressive, rent-seeking authorities such as Visa and Mastercard (or, for that matter, Stripe) are nowhere in the chain.
 
Given all that, it’s easy to see why the blockchain idea evokes utopian hopes: at last, technology is sticking it to the Man. In that sense, the excitement surrounding it reminds me of the early days of the internet, when we really believed that our contemporaries had invented a technology that was democratising and liberating and beyond the reach of established power structures. And indeed the network had – and still possesses – those desirable affordances. But we’re not using them to achieve their great potential. Instead, we’ve got YouTube and Netflix. What we underestimated, in our naivety, were the power of sovereign states, the ruthlessness and capacity of corporations and the passivity of consumers, a combination of which eventually led to corporate capture of the internet and the centralisation of digital power in the hands of a few giant corporations and national governments. In other words, the same entrapment as happened to the breakthrough communications technologies – telephone, broadcast radio and TV, and movies – in the 20th century, memorably chronicled by Tim Wu in his book The Master Switch.

Will this happen to blockchain technology? Hopefully not, but the enthusiastic endorsement of it by outfits such as Goldman Sachs is not exactly reassuring. The problem with digital technology is that, for engineers, it is both intrinsically fascinating and seductively challenging, which means that they acquire a kind of tunnel vision: they are so focused on finding solutions to the technical problems that they are blinded to the wider context. At the moment, for example, the consensus-establishing processes for verifying blockchain transactions requires intensive computation, with a correspondingly heavy carbon footprint. Reducing that poses intriguing technical challenges, but focusing on them means that the engineering community isn’t thinking about the governance issues raised by the technology. There may not be any central authority in a blockchain but, as Vili Lehdonvirta pointed out years ago, there are rules for what constitutes a consensus and, therefore, a question about who exactly sets those rules. The engineers? The owners of the biggest supercomputers on the chain? Goldman Sachs? These are ultimately political questions, not technical ones.

Blockchain engineers also don’t seem to be much interested in the needs of the humans who might ultimately be users of the technology. That, at any rate, is the conclusion that cryptographer Moxie Marlinspike came to in a fascinating examination of the technology. “When people talk about blockchains,” he writes, “they talk about distributed trust, leaderless consensus and all the mechanics of how that works, but often gloss over the reality that clients ultimately can’t participate in those mechanics. All the network diagrams are of servers, the trust model is between servers, everything is about servers. Blockchains are designed to be a network of peers, but not designed such that it’s really possible for your mobile device or your browser to be one of those peers.”

And we’re nowhere near that point yet.

Saturday 6 June 2020

Scientific or Pseudo Knowledge? How Lancet's reputation was destroyed

The now retracted paper halted hydroxychloroquine trials. Studies like this determine how people live or die tomorrow writes James Heathers in The Guardian

 

‘At its best, peer review is a slow and careful evaluation of new research by appropriate experts. ... At its worst, it is merely window dressing that gives the unwarranted appearance of authority’. Photograph: George Frey/AFP/Getty Images


The Lancet is one of the oldest and most respected medical journals in the world. Recently, they published an article on Covid patients receiving hydroxychloroquine with a dire conclusion: the drug increases heartbeat irregularities and decreases hospital survival rates. This result was treated as authoritative, and major drug trials were immediately halted – because why treat anyone with an unsafe drug?

Now, that Lancet study has been retracted, withdrawn from the literature entirely, at the request of three of its authors who “can no longer vouch for the veracity of the primary data sources”. Given the seriousness of the topic and the consequences of the paper, this is one of the most consequential retractions in modern history.

---Also watch

---

It is natural to ask how this is possible. How did a paper of such consequence get discarded like a used tissue by some of its authors only days after publication? If the authors don’t trust it now, how did it get published in the first place?

The answer is quite simple. It happened because peer review, the formal process of reviewing scientific work before it is accepted for publication, is not designed to detect anomalous data. It makes no difference if the anomalies are due to inaccuracies, miscalculations, or outright fraud. This is not what peer review is for. While it is the internationally recognised badge of “settled science”, its value is far more complicated.

At its best, peer review is a slow and careful evaluation of new research by appropriate experts. It involves multiple rounds of revision that removes errors, strengthens analyses, and noticeably improves manuscripts.

At its worst, it is merely window dressing that gives the unwarranted appearance of authority, a cursory process which confers no real value, enforces orthodoxy, and overlooks both obvious analytical problems and outright fraud entirely.

Regardless of how any individual paper is reviewed – and the experience is usually somewhere between the above extremes – the sad truth is peer review in its entirety is struggling, and retractions like this drag its flaws into an incredibly bright spotlight.

The ballistics of this problem are well known. To start with, peer review is entirely unrewarded. The internal currency of science consists entirely of producing new papers, which form the cornerstone of your scientific reputation. There is no emphasis on reviewing the work of others. If you spend several days in a continuous back-and-forth technical exchange with authors, trying to improve their manuscript, adding new analyses, shoring up conclusions, no one will ever know your name. Neither are you paid. Peer review originally fitted under an amorphous idea of academic “service” – the tasks that scientists were supposed to perform as members of their community. This is a nice idea, but is almost invariably maintained by researchers with excellent job security. Some senior scientists are notorious for peer reviewing manuscripts rarely or even never – because it interferes with the task of producing more of their own research.

However, even if reliable volunteers for peer review can be found, it is increasingly clear that it is insufficient. The vast majority of peer-reviewed articles are never checked for any form of analytical consistency, nor can they be – journals do not require manuscripts to have accompanying data or analytical code and often will not help you obtain them from authors if you wish to see them. Authors usually have zero formal, moral, or legal requirements to share the data and analytical methods behind their experiments. Finally, if you locate a problem in a published paper and bring it to either of these parties, often the median response is no response at all – silence.

This is usually not because authors or editors are negligent or uncaring. Usually, it is because they are trying to keep up with the component difficulties of keeping their scientific careers and journals respectively afloat. Unfortunately, those goals are directly in opposition – authors publishing as much as possible means back-breaking amounts of submissions for journals. Increasingly time-poor researchers, busy with their own publications, often decline invitations to review. Subsequently, peer review is then cursory or non-analytical.

And even still, we often muddle through. Until we encounter extraordinary circumstances.






Peer review during a pandemic faces a brutal dilemma – the moral importance of releasing important information with planetary consequences quickly, versus the scientific importance of evaluating the presented work fully – while trying to recruit scientists, already busier than usual due to their disrupted lives, to review work for free. And, after this process is complete, publications face immediate scrutiny by a much larger group of engaged scientific readers than usual, who treat publications which affect the health of every living human being with the scrutiny they deserve.

The consequences are extreme. The consequences for any of us, on discovering a persistent cough and respiratory difficulties, are directly determined by this research. Papers like today’s retraction determine how people live or die tomorrow. They affect what drugs are recommended, what treatments are available, and how we get them sooner.

The immediate solution to this problem of extreme opacity, which allows flawed papers to hide in plain sight, has been advocated for years: require more transparency, mandate more scrutiny. Prioritise publishing papers which present data and analytical code alongside a manuscript. Re-analyse papers for their accuracy before publication, instead of just assessing their potential importance. Engage expert statistical reviewers where necessary, pay them if you must. Be immediately responsive to criticism, and enforce this same standard on authors. The alternative is more retractions, more missteps, more wasted time, more loss of public trust … and more death.

Saturday 22 October 2016

So much for scientific publications: Nonsense paper written by iOS autocomplete accepted for conference

Elle Hunt in The Guardian


A nonsensical academic paper on nuclear physics written only by iOS autocomplete has been accepted for a scientific conference.


Christoph Bartneck, an associate professor at the Human Interface Technology laboratory at the University of Canterbury in New Zealand, received an email inviting him to submit a paper to the International Conference on Atomic and Nuclear Physics in the US in November.

Since I have practically no knowledge of nuclear physics I resorted to iOS autocomplete function to help me writing the paper,” he wrote in a blog post on Thursday. “I started a sentence with ‘atomic’ or ‘nuclear’ and then randomly hit the autocomplete suggestions.

“The atoms of a better universe will have the right for the same as you are the way we shall have to be a great place for a great time to enjoy the day you are a wonderful person to your great time to take the fun and take a great time and enjoy the great day you will be a wonderful time for your parents and kids,” is a sample sentence from the abstract.

It concludes: “Power is not a great place for a good time.”
Bartneck illustrated the paper – titled, again through autocorrect, “Atomic Energy will have been made available to a single source” – with the first graphic on the Wikipedia entry for nuclear physics.

He submitted it under a fake identity: associate professor Iris Pear of the US, whose experience in atomic and nuclear physics was outlined in a biography using contradictory gender pronouns.

The nonsensical paper was accepted only three hours later, in an email asking Bartneck to confirm his slot for the “oral presentation” at the international conference.

“I know that iOS is a pretty good software, but reaching tenure has never been this close,” Bartneck commented in the blog post.

He did not have to pay money to submit the paper, but the acceptance letter referred him to register for the conference at a cost of US$1099 (also able to be paid in euros or pounds) as an academic speaker.

“I did not complete this step since my university would certainly object to me wasting money this way,” Bartneck told Guardian Australia. “... My impression is that this is not a particularly good conference.”

The International Conference on Atomic and Nuclear Physics will be held on 17-18 November in Atlanta, Georgia, and is organised by ConferenceSeries: “an amalgamation of Open Access Publications and worldwide international science conferences and events”, established in 2007.

An organiser has been contacted by Guardian Australia for comment.

Bartneck said that given the quality of the review process and the steep registration fee, he was “reasonably certain that this is a money-making conference with little to no commitment to science.

“I did not yet reply to their email, but I am tempted to ask them about the reviewers’ comments. That might be a funny one.”

The conference’s call for abstracts makes only a little more sense than Bartneck’s paper.

“Nuclear and sub-atomic material science it the investigation of the properties, flow and collaborations of the essential (however not major) building pieces of matter.”

A bogus research paper reading only “Get me off Your Fucking Mailing List” repeated over and over again was accepted by the International Journal of Advanced Computer Technology, an open-access academic journal, in November 2014.

Saturday 29 November 2014

Misjudging universities

Pervez Hoodbhoy in The Daen

THE headlines earlier this week were celebratory: a Pakistani university has been included in the “500 Best Global Universities” by the US News and World Report. Although Quaid-i-Azam University (QAU) in Islamabad occupied only the 496th place — well below 10 other universities from neighbouring India and Iran — this is welcome news. Have we actually zoomed up and away from the rock-bottom standards of our higher education?
Sadly, the flawed methodology used by the Report means that this happy conclusion cannot be affirmed or denied. If used to assess universities in the United States and Europe the approach, though controversial, has some degree of validity. But, applied to developing countries like Pakistan, Iran, and India, it can lead to absurd conclusions.
Take, for example, the inclusion of QAU but the absence of the Lahore University of Management Sciences (LUMS) from the top 500 list. QAU is a rather ordinary Pakistani public-sector university where I have taught for 41 years, and where I still continue to teach voluntarily. I have much affection for it. LUMS, on the other hand, is a private university for Pakistan’s pampered super-elite with much greater resources, financial and intellectual. I do not have the same positive feelings for LUMS, where my two years of teaching ended mysteriously and unpleasantly. So, whereas I wish it were the other way around, honesty compels me to say that LUMS is superior as a university to QAU. Those familiar with both institutions will surely agree.

The rot can be stemmed if the HEC and PCST agree to reverse policies that incentivise corruption.


Just what have the Report’s editors been smoking? According to its website, the Report judges a university by the quantity of research produced. More precisely, 65pc of the grade comes from counting the number of PhDs produced, papers published, and citations earned. Another 25pc is for an (undefined and undefinable) “research reputation”, while the remaining 10pc is for “international collaborations”. This recipe is not unreasonable. After all, publishing research articles in good journals and counting citations is important in assessing individual and institutional academic achievement. Having PhD students undoubtedly helps generate a culture of research.
But there’s a catch. Social scientists call it Campbell’s Law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” This law is nearly as ironclad as one of physics. A corollary: robust systems may suffer some distortion but weaker ones can be willfully deformed and massively manipulated by prevailing local interests.
Pakistan’s academic system is, as everyone agrees, far from robust. An estimated 40pc of students cheat in matriculation, intermediate, and college examinations. Teachers are no more ethical than shopkeepers, policemen, politicians, judges, and generals. Because of policies that reward authors of research articles and PhD supervisors with cash and promotions, our universities have turned into factories producing junk papers and PhDs. Publishing papers is now a well-developed art form that combines outright plagiarism, faking data, showcasing trivia, repeating old papers, and using fly-by-night journals. With apologies to the few genuine students and their supervisors, the fact is that PhDs are awarded to all and sundry.
From many grotesque examples, I will repeat one that I had argued out fruitlessly for many months with Dr Javed Leghari, who became the Higher Education Commission (HEC) chairman following Dr Atta-ur-Rahman, the principal architect of the numbers game.
This concerned a physics PhD thesis that was guided by an “HEC meritorious professor” at Balochistan University, co-supervised by the then vice-chancellor of Quaid-i-Azam University, Dr Masoom Yasinzai. The thesis title was A quantitative study on chromotherapy. The text contained equations that apparently bestowed respectability. Together with several notable Pakistani physicists, I saw this as nonsense. But months of effort failed to convince Dr Leghari, who refused to reveal the names of the referees.
As a last-ditch effort, I sent a copy to two distinguished physicists who I knew for many years. One was the physics Nobel Prize (1979) winner, Steven Weinberg, and the other was the physics Nobel Prize (1988) winner, Jack Steinberger.
Weinberg wrote a point by point criticism which ended up saying: “I am appalled by what I have seen. The thesis shows a lack of understanding of the fundamentals of physics. This thesis is not only unworthy of a PhD, it is positively dangerous, since it might lead patients with severe illnesses to rely on ‘chromotherapy’ rather than on scientific medicine. I find it difficult to understand how this thesis could have earned its author any academic degree.”
Steinberger was equally negative: “a reasonable physics department should not have accepted anything like this work … Following world news this past decade, I have been very unhappy about the Pakistani political instability and social problems, but I had imagined that its cultural level was better than what I now see.”
This rot can be stemmed if the HEC and Pakistan Council for Science & Technology agree to reverse policies that incentivise corruption. This will not be easy. Resisting pressures from greedy beneficiaries of the present system will require enormous moral strength, especially now that the Report has demonstrated the rewards for wholesale publishing.
My last meeting with the current HEC chairman, Dr Mukhtar Ahmad, was a surprisingly pleasant one. He expressed concern at the decay within and seemed receptive to the following suggestion: “Let the HEC require that an author of a research paper, for which he or she desires official credit, to give a video-recorded presentation before the institution’s faculty. This would be archived for free access on the HEC website.”
All necessary technologies needed to implement the above are already in place. The benefits would be two-fold. First, any piece of genuinely important research would become widely known. Second, fake research and corrupt practices would be readily spotted.
Many months have passed since our meeting. Although my emails requesting signs of progress remain unanswered, I remain hopeful that the honourable chairman’s reply is somewhere out there in cyberspace.
The writer teaches physics and mathematics in Lahore and Islamabad.

Tuesday 30 August 2011

Academic publishers make Murdoch look like a socialist


Academic publishers charge vast fees to access research paid for by us. Down with the knowledge monopoly racketeers
  • College Students Library
    'Though academic libraries have been frantically cutting subscriptions to make ends meet, journals now consume 65% of their budgets.' Photograph: Peter M Fisher/Corbis
     
    Who are the most ruthless capitalists in the western world? Whose monopolistic practices make Walmart look like a corner shop and Rupert Murdoch a socialist? You won't guess the answer in a month of Sundays. While there are plenty of candidates, my vote goes not to the banks, the oil companies or the health insurers, but – wait for it – to academic publishers. Theirs might sound like a fusty and insignificant sector. It is anything but. Of all corporate scams, the racket they run is most urgently in need of referral to the competition authorities. Everyone claims to agree that people should be encouraged to understand science and other academic research. Without current knowledge, we cannot make coherent democratic decisions. But the publishers have slapped a padlock and a "keep out" sign on the gates. You might resent Murdoch's paywall policy, in which he charges £1 for 24 hours of access to the Times and Sunday Times. But at least in that period you can read and download as many articles as you like. Reading a single article published by one of Elsevier's journals will cost you $31.50. Springer charges €34.95, Wiley-Blackwell, $42. Read 10 and you pay 10 times. And the journals retain perpetual copyright. You want to read a letter printed in 1981? That'll be $31.50. Daniel Pudles illo Illustration by Daniel Pudles Of course, you could go into the library (if it still exists). But they too have been hit by cosmic fees. The average cost of an annual subscription to a chemistry journal is $3,792. Some journals cost $10,000 a year or more to stock. The most expensive I've seen, Elsevier's Biochimica et Biophysica Acta, is $20,930. Though academic libraries have been frantically cutting subscriptions to make ends meet, journals now consume 65% of their budgets, which means they have had to reduce the number of books they buy. Journal fees account for a significant component of universities' costs, which are being passed to their students. Murdoch pays his journalists and editors, and his companies generate much of the content they use. But the academic publishers get their articles, their peer reviewing (vetting by other researchers) and even much of their editing for free. The material they publish was commissioned and funded not by them but by us, through government research grants and academic stipends. But to see it, we must pay again, and through the nose. The returns are astronomical: in the past financial year, for example, Elsevier's operating profit margin was 36% (£724m on revenues of £2bn). They result from a stranglehold on the market. Elsevier, Springer and Wiley, who have bought up many of their competitors, now publish 42% of journal articles. More importantly, universities are locked into buying their products. Academic papers are published in only one place, and they have to be read by researchers trying to keep up with their subject. Demand is inelastic and competition non-existent, because different journals can't publish the same material. In many cases the publishers oblige the libraries to buy a large package of journals, whether or not they want them all. Perhaps it's not surprising that one of the biggest crooks ever to have preyed upon the people of this country – Robert Maxwell – made much of his money through academic publishing. The publishers claim that they have to charge these fees as a result of the costs of production and distribution, and that they add value (in Springer's words) because they "develop journal brands and maintain and improve the digital infrastructure which has revolutionised scientific communication in the past 15 years". But an analysis by Deutsche Bank reaches different conclusions. "We believe the publisher adds relatively little value to the publishing process … if the process really were as complex, costly and value-added as the publishers protest that it is, 40% margins wouldn't be available." Far from assisting the dissemination of research, the big publishers impede it, as their long turnaround times can delay the release of findings by a year or more. What we see here is pure rentier capitalism: monopolising a public resource then charging exorbitant fees to use it. Another term for it is economic parasitism. To obtain the knowledge for which we have already paid, we must surrender our feu to the lairds of learning. It's bad enough for academics, it's worse for the laity. I refer readers to peer-reviewed papers, on the principle that claims should be followed to their sources. The readers tell me that they can't afford to judge for themselves whether or not I have represented the research fairly. Independent researchers who try to inform themselves about important scientific issues have to fork out thousands. This is a tax on education, a stifling of the public mind. It appears to contravene the universal declaration of human rights, which says that "everyone has the right freely to … share in scientific advancement and its benefits". Open-access publishing, despite its promise, and some excellent resources such as the Public Library of Science and the physics database arxiv.org, has failed to displace the monopolists. In 1998 the Economist, surveying the opportunities offered by electronic publishing, predicted that "the days of 40% profit margins may soon be as dead as Robert Maxwell". But in 2010 Elsevier's operating profit margins were the same (36%) as they were in 1998. The reason is that the big publishers have rounded up the journals with the highest academic impact factors, in which publication is essential for researchers trying to secure grants and advance their careers. You can start reading open-access journals, but you can't stop reading the closed ones. Government bodies, with a few exceptions, have failed to confront them. The National Institutes of Health in the US oblige anyone taking their grants to put their papers in an open-access archive. But Research Councils UK, whose statement on public access is a masterpiece of meaningless waffle, relies on "the assumption that publishers will maintain the spirit of their current policies". You bet they will. In the short term, governments should refer the academic publishers to their competition watchdogs, and insist that all papers arising from publicly funded research are placed in a free public database. In the longer term, they should work with researchers to cut out the middleman altogether, creating – along the lines proposed by Björn Brembs of Berlin's Freie Universität – a single global archive of academic literature and data. Peer-review would be overseen by an independent body. It could be funded by the library budgets which are currently being diverted into the hands of privateers. The knowledge monopoly is as unwarranted and anachronistic as the corn laws. Let's throw off these parasitic overlords and liberate the research that belongs to us. • A fully referenced version of this article can be found on George Monbiot's website. On Twitter, @georgemonbiot