Psychology experiments are failing the replication test – for good reason

John Ioannidis in The Guardian

‘The replication failure rate of psychology seems to be in the same ballpark as those rates in observational epidemiology, cancer drug targets and preclinical research, and animal experiments.’ Photograph: Sebastian Kaulitzki/Alamy


Science is the best thing that has happened to humankind because its results can be questioned, retested, and demonstrated to be wrong. Science is not about proving at all cost some preconceived dogma. Conversely religious devotees, politicians, soccer fans, and pseudo-science quacks won’t allow their doctrines, promises, football clubs or bizarre claims to be proven illogical, exaggerated, second-rate or just absurd.

Despite this clear superiority of the scientific method, we researchers are still fallible humans. This week, an impressive collaboration of 270 investigators working for five years published in Science the results of their efforts to replicate 100 important results that had been previously published in three top psychology journals. The replicators worked closely with the original authors to make the repeat experiments close replicas of the originals. The results were bleak: 64% of the experiments could not be replicated.
We often feel uneasy about having our results probed for possible debunking. We don’t always exactly celebrate when we are proven wrong. For example, retracting published papers can take many years and many editors, lawyers, and whistleblowers – and most debunked published papers are never retracted. Moreover, with fierce competition for limited research funds and with millions of researchers struggling to make a living (publish, get grants, get promoted), we are under immense pressure to make “significant”, “innovative” discoveries. Many scientific fields are thus being flooded with claimed discoveries that nobody ever retests. Retesting (called replication) is discouraged. In most fields, no funding is given for what is pooh-poohed as me-too efforts. We are forced to hasten from one “significant” paper to the next without ever reassessing our previously claimed successes.

Multiple lines of evidence suggest this is a recipe for disaster, leading to a scientific literature littered with long chains of irreproducible results. Irreproducibility is rarely an issue of fraud. Simply having millions of hardworking scientists searching fervently and creatively in billions of analyses for something statistically significant can lead to very high rates of false-positives (red-herring claims about things that don’t exist) or inflated results.

This is more likely to happen in fields that chase subtle, complex phenomena, in those that have more noise in measurement, and where there is more room for subjective choices to be introduced in designing and running experiments and crunching the data. Ten years ago I tried to model these factors. These models predicted that in most scientific fields and settings the majority of published research, findings may be false. They also anticipated that the false rates could vary greatly (from almost 0% to almost 100%), depending on the features of a scientific discipline and how scientists run their work.

Probably the failure rate in the Science data would have been higher for work published in journals of lesser quality. There are tens of thousands of journals in the scientific-publishing market, and most will publish almost anything submitted to them. The failure rate may also be higher for studies that are so complex that none of the collaborating replicators offered to attempt a replication. This group accounted for one-third of the studies published in the three top journals. So the replication failure rate for psychology at large may be 80% or more overall.

This performance is even worse than I would have predicted. In 2012 my anticipation of a 53% replication failure rate for psychology at large was published. Compared with other empirical studies, the failure rate of psychology seems to be in the same ballpark as replication failure rates in observational epidemiology, cancer drug targets and preclinical research, and animal experiments.

However, I think it is important to focus on the positive side. The Science paper shows that large-scale replication efforts of high quality are doable even in fields like psychology where there was no strong replication culture until recently. Hopefully this successful, highly informative paradigm will help improve research practices in this field. Many other scientific fields without strong replication cultures may also be prompted now to embrace replications and reproducible research practices. Thus these seemingly disappointing results offer a great opportunity to strengthen scientific investigation. I look forward to celebrate one day when my claim that most published research findings are false is thoroughly refuted across most, if not all, scientific fields.

Why happy couples cheat?

Why happy couples cheat? Esther Perel

Why we love, why we cheat? Helen Fisher

The Elite Opt-out - 'Government officials should send their children to government schools'

Manish Sabharwal in The Indian Express

Earlier this month, the Allahabad High Court gave the state chief secretary till the next academic session to require anybody drawing a government salary to send their children to only government schools. The order also specified that the promotions and increments of violators should be deferred, and required that any fees paid to private schools by government servants be deducted from their salary and paid into the state treasury. The judge felt this extreme step was the only way to improve government schools. Is this judgment absurd or wonderful common sense?

This judgment pulls me in two different directions because of who I am (son of government servants who sent me to a private school) and what I do (our company hires only 5 per cent of the children who come to us for a job because their schools let them down). Who I am believes this judgment is absurdity. What I do believes it is wonderful common sense. Why this divergence?

The first reaction is because this judgment violates the fundamental rights of all children of government employees by limiting where they can go to school based on their parents’ profession. I know my professional progress is a child of my private school education; I would not be where I am if I had been forced into a government school in Uttar Pradesh (where my parents are from) or in Jammu and Kashmir (where my parents worked). It’s also unfair to hold every present government servant accountable for the actions and outcomes of a small number of past and present education department bureaucrats. It hardly seems fair that the judgment should not be applied to past politicians who have grossly distorted government school-teacher recruiting, compensation and performance management, or past judges whose judgments have distorted the governance of educational institutions. The problem with holding the current government-employee cohort accountable for school outcomes is the long shadow cast by education policy decisions, where toxic effects show up only after a decade.

But my second reaction is rooted in a recognition of human nature. Wouldn’t Employee’s State Insurance hospitals improve if we forced government servants to use them and abolished their exclusive Central Government Health Scheme? Wouldn’t EPFO reform have happened years ago if every government servant was forced to deal with the organisation for their pensions? Wouldn’t ministers standing in line for security and boarding at airports have forced security forces to reduce lines by junking the meaningless stamping, and checking of stamping, of hand baggage tags and boarding cards? Wouldn’t there have been more urgency for power reforms if distribution companies were prohibited from creating VIP areas where power is uninterrupted? Wouldn’t we have better urban planning, housing and public transport if government servants were not given houses and cars and all their benefits were monetised instead?

Given India’s poor service-delivery outcomes; it’s certainly a tantalising possibility to subject government servants to the consequences of their actions. This judgment is obviously the product of an interesting mind — Justice Sudhir Agarwal — but it is also a child of broader trends in society and merely reflects the rising aspirations and expectations of millions of Indians. India’s poor and youth are no longer willing to be held hostage to poor government provision. They recognise that “elite opt-out” accelerates the decline of the public system because powerful and loud voices don’t care. My parents retired to Kanpur, where the fastest growing industries are private bottled water, private security, private generators, private healthcare and private schools; the poor in UP are buying what should be public goods because their rights as consumers are greater than their rights as citizens. India is reaching the point where government sins of commission (what it does wrong) are not as toxic as government sins of omission (what it does not do). Alexander Hamilton wrote that the courts are the weakest of the three branches of government because they control neither sword nor purse. India’s courts cannot sustainably fix public service-delivery. Government schools can only be fixed by politicians obsessed with execution, not inputs.

The execution problem goes beyond schools. The Indian state has been designed for less complexity, scale and accountability than it faces. Going forward, the state must do fewer things, but do them well. It must retreat where the market works but act muscularly where the market fails. It must separate its role as policymaker, regulator and service-provider in all areas. It must create the hope of rising and fear of failing for the permanent generalist civil service and supplement them with specialist lateral entry. Fixing government schools is crucial to economic democracy; I work for a company that has hired somebody every five minutes for the last five years, but only hired five per cent of job applicants. You can’t teach kids in one year of vocational training at the exit gate of K-12 education what they should have learnt in the 12 years of school. An unskilled or unemployed Indian is not a free Indian. So — with all the hypocrisy of somebody whose turn for a government school education under this judgment is past — I hope the wonderful common sense that this judgment represents is upheld.

Financial markets are not free – they're one of the last bastions of socialism

Larry Elliot in The Guardian

 

You can’t buck the market, Margaret Thatcher once said, but the world’s policymakers have been giving it their best shot. Photograph: PA/PA


You can’t buck the market, so said Margaret Thatcher back in the late 1980s. Maybe you can’t, but the world’s policymakers are giving it their best shot.

Let’s just consider the context. During August there was a big sell-off in shares amid concerns that the Chinese economy was in trouble. The declines, however, followed a long bull market in which prices were supported by the quantitative easing programmes pursued by central banks rather than by underlying economic strength. A true believer in free markets would have seen the recent sell-off as both inevitable and healthy.

But financial markets are by no means free. They are, on the contrary, one of the last bastions of socialism left on earth. Everything possible is done to boost asset prices and when overstimulation leads to bubbles bursting it is all hands to the pump to prevent them from falling too far.

So, in the last couple of days we have seen the chief economist of the European Central Bank musing openly about the possibility of additional QE. We have had the president of the New York Federal Reserve dropping the biggest possible hint that an interest rate rise will be delayed. And we have had the authorities in Beijing buying up shares on the Shanghai stock market.

And guess what? It has worked. Shares have been going up around the world because investors have been given guarantees that they will be protected from the consequences of their own folly. Despite the upward revision to US growth figures, all that Wall Street needs to do to prevent a Fed rate rise is to have another flash crash between now and mid-September. The Fed will then back off. If things get really bad, it will need to consider a fourth dose of QE. Yes, really.

George Saravelos, of Deutsche Bank, says there is a reason western central banks, starting with the ECB and the Bank of Japan, might need to again resort to the electronic printing presses: China’s use of its foreign exchange reserves to defend the external value of the yuan.

Effectively, China has been selling a chunk of its holdings of foreign bonds, such as US Treasuries. QE involves buying bonds, so what the People’s Bank of China has been doing is QE in reverse or, as Saravelos puts it, quantitative tightening or QT.

Ostensibly, QT has always been part of the plan. When central banks embarked on their asset-buying programmes, the intention was that they would be sold back to the markets when things had returned to normal. The problem is that things have never returned to normal and, judging by recent events, never will.

Zero interest rates were supposed to be temporary. They are now the norm. QE was an unconventional measure for use in an emergency only. It is here to stay, albeit subject to the law of diminishing returns.

Clearly then, you can buck the market. Corrections can be delayed and halted, even reversed, by determined policy action. But, as the history of bubbles from Dutch tulips to subprime mortgages has shown, only for a while. In the long run, Mrs T was right.