Volatility: how ‘algos’ changed the rhythm of the market

Critics say high-frequency trading makes markets too fickle amid rising anxiety over the global economy  writes Robin Wigglesworth in The FT

Philippe Jabre was the quintessential swashbuckling trader, slicing his way through markets first at GLG Partners and then an eponymous hedge fund he founded in 2007 — at the time one of the industry’s biggest-ever launches. But in December he fell on his sword, closing Jabre Capital after racking up huge losses. The fault, he said, was machines. 

“The last few years have become particularly difficult for active managers,” he said in his final letter to clients. “Financial markets have significantly evolved over the past decade, driven by new technologies, and the market itself is becoming more difficult to anticipate as traditional participants are imperceptibly replaced by computerised models.” 

Mr Jabre is not alone. There has been recently a flurry of finger-pointing by humbled one-time masters of the universe, who argue that the swelling influence of computer-powered “quantitative”, or quant, investors and high-frequency traders is wreaking havoc on markets and rendering obsolete old-fashioned analysis and common sense. 

Those concerns were exacerbated by the volatility in financial markets in December, when US equities suffered their biggest monthly decline since the financial crisis, despite little fundamental economic news. And with growing anxiety over the strength of the global economy, tightening monetary policy across the world and an escalating trade war between China and the US, these trades are getting more attention. 

Even hedge fund veterans admit the game has changed. “These ‘algos’ have taken all the rhythm out of the market, and have become extremely confusing to me,” Stanley Druckenmiller, a famed investor and hedge fund manager, recently told an industry TV station. 

It is true that markets are evolving. HFTs dominate the market-making once done by humans in trading pits and the bowels of investment banks. Various quant strategies — ranging from simple ones packaged into passive funds to pricey, complex hedge funds — manage at least $1.5tn, according to Morgan Stanley. JPMorgan estimates that only about 10 per cent of US equity trading is now done by traditional investors. Other markets remain more human, yet are slowly but surely being transformed. 

This has made “the algos” a fashionable bugbear whenever markets tremble like they did in December. Torsten Slok, Deutsche Bank’s chief international economist, put them at the top of his list of the 30 biggest risks for markets, and even Steven Mnuchin, the US Treasury secretary who caused market unease with comments on liquidity late last year, has said the government will study whether the evolving market ecosystem fed the recent turmoil. 

But markets have always been tempestuous, and machines make a convenient, faceless bogeyman for fund managers who stumble. Meanwhile, quants point out that they are still only small players compared with the vastness of global markets. 

“It’s insane,” says Clifford Asness, the founder of AQR Capital Management. “People are missing the forest for the trees. That we trade electronically doesn’t change things, we just deliver the same thing more efficiently . . . It’s just used by pundits and fund managers as an excuse.” 

The recent turmoil has unnerved many investors, but two other debacles stand out as having first crystallised the fear that algorithms are making markets more fickle and fragile. 

At 2:32pm on May 6 2010, US equities suddenly and mysteriously careened lower. In just 36 minutes the S&P 500 crashed more than 8 per cent, before rebounding just as powerfully. Dubbed the “flash crash” it put a spotlight on the rise of small ultra-fast, algorithmic trading firms that have elbowed out investment banks as the integral intermediaries of many markets. 

Michael Lewis, author of Flash Boys, fanned the flames with his book by casting HFTs as mysterious, investor-scalping antagonists “rigging” the stock market. What was once an esoteric, little-appreciated evolution in the market’s plumbing suddenly became the topic of a vitriolic mainstream debate. 

“It was a wake-up call,” says Andrei Kirilenko, former chief economist at the Commodity Futures Trading Commission who wrote the US regulator’s report on the 2010 event and now leads Imperial College London’s Centre for Global Finance and Technology. “The flash crash was the first market crash in the era of automated, algorithmic trading.” 

In August 2015, markets were once again abruptly thrown into a tailspin — and this time volatility-sensitive quantitative strategies were identified as the primary culprits. The spark was rising concern over China’s economic slowdown, but on August 24, the S&P 500 crashed on opening, triggering circuit-breakers — implemented in the wake of the flash crash to pause wild trading — nearly 1,300 times. That rippled through a host of exchange traded funds, worsening the dislocations as they briefly became divorced from the value of their underlying holdings. 

Many investors and analysts blamed algorithmic strategies that automatically adjust their market exposure according to volatility for aggravating the 2015 crash. Targeting a specific level of volatility is common among strategies known as “risk parity” — trend-following hedge funds and “managed volatility” products sold by insurance companies. Estimates vary, but there is probably more than $1tn invested in a variety of such funds.

Risk parity, a strategy first pioneered by Ray Dalio’s Bridgewater Associates in the 1990s, often shoulders much of the opprobrium. The theory is that a broad, diversified portfolio of stocks, bonds and other assets balanced by the mathematical risk — in practice, volatility — of each asset class should over time enjoy better returns than traditional portfolios. Bonds are less volatile than equities, so that often means “leveraging” these investments to bring the risk-adjusted allocation up to that of stocks. As volatility goes up, risk parity funds in theory rein in their exposure. 

However, risk parity funds can vary greatly in the details of their approach, and are generally slower moving than the $300bn trend-following hedge fund industry. These funds surf market momentum up and down, and also use volatility metrics to scale their exposure. When markets are calm they buy, and when turbulence spikes they sell. 

This has been a successful strategy over time. But it leaves the funds vulnerable to abrupt reversals — such as the market tumble last February — and means they can accentuate turbulence by selling when markets are already sliding.

Leon Cooperman, the founder of Omega Advisors, has argued that the US Securities and Exchange Commission should investigate and tame the new “wild, wild west environment in the stock market” caused by these volatility-sensitive strategies. 

“I think your next guest ought to be somebody from the SEC to explain why they have sat back calmly, quietly, without saying anything and allowing these algorithmic, trend-following models to wreak havoc with what has, up to now, been the best capital market in the world,” he told CNBC in December. 

Some quants will grudgingly admit that volatility-targeting is inherently pro-cyclical and can at least in theory exacerbate market movements. But they say critics wildly overestimate just how much money is invested in these strategies, how much they trade, and their impact. 

“Risk parity is basically a passive portfolio with some periodic, counter-cyclical rebalancing. Our volatility targets aren’t perfectly static, but they only change over a 10-year window,” says Bob Prince, co-chief investment officer at Bridgewater. Other risk parity strategies may vary, but overall “it's only ever going to be a drop in the ocean”, he adds. 

Markets had been vulnerable to panicky plunges long before trading algorithms emerged, yet fears over machines seem deeply embedded in our psyche. A 2014 University of Pennsylvania paper found evidence of what it dubbed “algorithm aversion”, showing how human test subjects instinctively trusted human forecasters more than algorithmic ones, even after seeing the algo make fewer and less severe forecasting errors. 

And there are plenty of other potential culprits to blame for exacerbating recent turbulence. Many traditional active funds suffered a battering in 2018. That has led to a rise in investor redemption notices and has forced many to sell securities to meet the end-of-year withdrawals. 

Hedge fund flow data come with a lag, but traditional equity funds saw withdrawals rise to nearly $53bn in the seven days up to December 12, according to data provider EPFR — comfortably the biggest one-week outflow on record. That probably both reflected and exacerbated the slide that left the S&P 500 nursing a 6 per cent loss for 2018. 

At the same time, market liquidity— a broad term denoting how easy it is to trade quickly without causing prices to move around too much — tends to weaken in December, when many fund managers become more defensive ahead of the end of the year. Liquidity can be particularly poor in the last weeks of the year, when bank traders ratchet back how much risk they take on to avoid extra regulatory charges. 

“This makes it more expensive for dealers to perform their essential functions: providing liquidity, absorbing shocks and facilitating the transfer and socialisation of risk,” Joshua Younger, a JPMorgan analyst, wrote in a recent note. “These costs are generally passed on to customers in the form of higher rates on short-term loans, thinner markets and the risk — now realised — of spikes in volatility.” 

That markets are undergoing a dramatic, algorithmic evolution is an inescapable fact. Although some humbled hedge fund managers may unfairly castigate “algos” for their own failings, there are real risks in how some of these different factors can interact at times of market stress. 

HFTs are far more efficient market-makers than human pit traders. Yet the entire sector probably has less capital than just one of the major banks, says Charles Himmelberg, head of global markets research at Goldman Sachs. It means that they tend to adjust their bids aggressively when market mayhem breaks out. 

Under those circumstances, even a modest amount of selling could have an outsized impact. This is an issue both for human traders and quants, but quant strategies are programmed, quick and on autopilot, and if they start pounding an increasingly thin market, it can cause dislocations between buy and sell orders that can produce big gains or falls. 

For example, JPMorgan estimates that the depth of the big and normally liquid S&P 500 futures market — as measured by how many contracts trade close to the current price — deteriorated in 2018, and was exceptionally shallow in the last months of the year. In December it was even worse than the levels seen in the financial crisis. 

“While it is incorrect to say that systematic flows are the sole driver of recent market moves, it would be equally incorrect to say that systematic flows don’t have a meaningful impact,” says Marko Kolanovic, head of quantitative strategy at JPMorgan. 

Poor liquidity and market volatility have always been linked, and it is in practice impossible to dissect and diagnose the myriad triggers and drivers of a sell-off. But modern markets do appear more vulnerable to abrupt dislocations. 

The question is whether anything should, or even could, be done to mitigate the risks. Mr Kirilenko cautions that a mix of better understanding and modest tweaks may be the only conclusion. 

“We just have to accept that financial markets are nearly fully automated,” he says, “and try to make sure that things don’t get so technologically complex and inter-connected that it’s dangerous to the financial system.” 

Anxiety inducing: the triggers for market fears 

Although the recent market slide has reawakened the debate about whether modern machine-driven markets can exacerbate the severity of any volatility, the fundamental drivers of the turbulence are more conventional. As 2018 progressed, investors grew concerned at three factors: signs that the global economy is weakening; the impact of tighter monetary policy in the US and the end of quantitative easing in Europe; and the escalating trade war between the US and China. The global economy started last year on a strong footing, but markets are always focused on inflection points. Since the summer the impact of US tax cuts has appeared to fizzle, European growth has slowed, and China’s decelerating economy has been buffeted by the trade dispute. That has led analysts to trim their estimates for corporate profits in 2019. At the same time, the Federal Reserve raised interest rates four times last year, and has kept shrinking its balance sheet of bonds acquired in the wake of the financial crisis. That has lifted short-term ultra-safe Treasury bill yields to a 10-year high, and undermined the long-term argument that “there is no alternative” which has helped sustain market valuations. As a result, Treasury bills beat the returns of almost every major asset class last year. Goldman Sachs says that over the past century there have only been three other periods when Treasury bills have enjoyed such a broad outperformance: when the US ratcheted up interest rates to 20 per cent in the early 1980s to subdue inflation; during the Great Depression; and at the start of the first world war.

Daniel Goleman on Focus: The Secret to High Performance and Fulfilment

Corbyn's opposition to Brexit is vindicated by the EU state aid rules

Jonathan Ford in The FT

Some six years ago, British minicab company, Addison Lee took Transport for London on a journey through the law courts that ended up in 2015 at the European Court of Justice. At issue was a rule that allows only black cabs to use London’s bus lanes. 

Hardly an EU matter you might think, worrying about transport policy in Britain’s capital. But think again. The minicab company argued not only that the rule infringed its freedom to provide services; it was also illegal state aid. Letting black cabs use a bus lane was, in effect, selective support for a specific participant in the wider taxi trade. 

Admittedly the argument did not prevail in Luxembourg and London’s bus lanes remain limited to buses and traditional taxis. But while the ECJ failed to deem the policy state assistance, it did not entirely rebut the principle. 

The justices recognised that TfL’s policy could affect trade between member states because it potentially made it harder for European-owned minicabs to penetrate the market. The definitional ratchet of unfair state aid turned gently another notch. 

“It illustrates how the definition of state aid has expanded over the years,” says James Webber, an antitrust partner at the law firm Shearman & Sterling. “Originally designed to contain industrial subsidy competition, state aid is now much wider than commonly assumed, covering many public spending decisions such as infrastructure investment and tax changes which could be seen to benefit a specific region, sector or even just to incentivise desirable corporate behaviour.” 

Quite right too, you might say. Unfettered subsidy competition is not only wasteful; it is ultimately pointless, for any advantage is lost if it descends into a free-for-all. Which is why it was so uncontentious when Theresa May conceded early in Brexit negotiations that the UK would stick with EU state-aid rules. But in practice what has been signed up to is far from uncontentious. The prime minister’s agreement has contrived to put the UK into a uniquely disadvantageous place, forced to follow the EU’s state-aid rules but not protected by them in return. 

The mischief lies in the so-called Irish backstop, which would leave the whole of the UK still subject to single market regulation in state aid even though the country was formally outside the EU. As a consequence, Britain will continue to apply not only those substantive state-aid rules that existed at exit, but any new ones too. 

The rules are only part of any state-aid decision. What really matters is the discretionary power possessed by the commission to approve or disapprove — sometimes with conditions — any plans member states propose which trigger state-aid concerns. That is inevitably a political process. The UK’s Hinkley Point nuclear project was generously state-aided. But the commission blessed it because two big member states — Britain and France (which was supplying the technology) — had the clout to ram it through. 

While that clout disappears along with Britain’s representation in Brussels at Brexit, UK interests are in theory protected because the discretionary power passes from the EU to the Competition and Markets Authority. But really it does not. In practice the UK regulator will have to get the European Commission’s sign-off, and the final arbiter will be the ECJ. And that is just in Great Britain. In Northern Ireland, the commission will remain in total control of the whole process itself. 

Its discretion may not be used in Britain’s favour. Not only will the UK no longer have insider clout, but it will also be seen as a strategic competitor. Just last week, the foreign secretary Jeremy Hunt was in Singapore, talking admiringly about its economic model, which extends fiscal support to favoured industries. It remains to be seen how keen the commission will be to sign off fiscal measures designed to enhance UK competitiveness and offset any exit bumps. 

Worse, Britain’s anomalous backstop status may mean that while it cannot protect its own industries, it will be defenceless against EU states dangling subsidies to encourage British companies to relocate. State aid in the EU is not problematic if it attracts jobs to the union from non-EEA countries. European goodwill is the only backstop here. 

There is no certainty too that this will be unravelled in any trade deal. Given the permanence of the backstop, the EU has no incentive to offer Britain more favourable terms than the ones in the withdrawal agreement. 

 The deal would unwittingly lock Britain into a regulatory iron maiden of Brussels’ manufacture. As with such contraptions, extraction once inserted may prove harder than clambering in.

How To ABSORB TEXTBOOKS Like A Sponge

Apple and China’s problems show that today’s titans may not rule the world tomorrow

Even superpowers and trillion-dollar tech giants are at risk in a fast-changing society writes Will Hutton in The Guardian

 

An Apple Store in Beijing, where the company’s sales figures have slumped. Photograph: Wu Hong/EPA


Our mental geography is bounded by what has gone before. What has happened in the recently remembered past is most likely to continue. Inflection points, when trends decisively change, are more infrequent than the many instances when things go on as they have done.

Two of today’s trends seem unstoppable. China’s astounding growth will continue, so the story runs, underwriting its arrival as the second economic superpower. To get a share in that China action, underpinning the entire growth of Asia, is one of the prime economic arguments for Brexit. Abandon sclerotic Europe, embrace the prosperity of Asia – even if it is a world of semi-democracy at best, authoritarian government at worst. It can be guaranteed to grow.

Second, the west coast big tech companies, from Facebook to Apple, are the new wonders of the universe. They are the bewilderingly successful face of the informational, data-driven economy whose value continues to soar. Apple, then Amazon, became the first trillion-dollar corporations last year, both exemplars of how first movers in innovation with their transformative technologies have become 21st-century titans, driving stock market growth and changing society alike.

However, last week, both trends were decisively challenged in what looks like a world-changing inflection moment, one where consensus assumptions start to unravel. Apple announced that for the first time in 17 years it would not meet its forecasts for revenue growth – and by a big margin. Its CEO, Tim Cook, explained to investors and staff that apart from the problems facing all makers of mobile phones – it is becoming a mature market – there was unexpected sales weakness in China. This was not just down to intense competition, but because China’s consumers were spending much less. Apple’s share price plunged – cumulatively, its value has fallen by a stunning $400bn in a couple of months.


China’s total debt, even on distrusted official figures, is approaching three times its GDP

But surely China, landing its robotic satellite on the far side of the moon, is suffering no more than a typical cyclical setback, intensified by Donald Trump’s trade war? Perhaps, but look more closely and it is ever clearer that long-standing problems are starting to envelop this continental economy.

It was only eight years ago that China registered 12% growth as the government turbo-boosted its economy with a massive stimulus in the wake of the financial crisis, growth that staved off a world slump. But since then its official growth rate has been consistently sliding, now halved at 6%, and China’s consumers have started to notice what last year’s 25% fall in China’s stock market is also signalling. Alongside Apple’s statement, last week’s surveys showing China’s famed manufacturing sector is set to decline in 2019 and further weakness in retail sales were more evidence that all is not well. China’s consumers are reading the runes.

Part of the issue for both Apple and China is the law of large numbers. Three-fifths of Apple’s sales are iPhones and there just aren’t enough global consumers with pockets deep enough to keep up the growth momentum. China’s issues are even more profound. There comes a limit, even in a state-controlled economy, to manipulating growth through debt and exports when the numbers reach astronomic levels. China’s total debt, even on distrusted official figures, is approaching three times its GDP, a flashpoint ratio for every economy when bank balance sheets, and their borrowers, just become too overstretched.

This was the trigger for Japan’s economic depression 30 years ago. Moreover, if China’s exports carried on growing as fast as they had, they would crowd out every other export from every country in the world by the mid-2020s. This was never likely, economically or politically. If Trump had not launched his trade offensive, another US leader would have done so. Apple and China, bluntly, are in a fix.

The Chinese Communist party is in a gathering crisis of legitimacy. If the growth rate sinks below 6% (unofficial figures now place growth at under 2% in 2018), its job-generating capacity starts to falter and questions will be asked about the competence of the party. Past leaders have responded to crises by intensifying the pace of transition towards a capitalist economy and boosting infrastructure spending as a quick fix.

These are not options for Xi Jinping. So high has infrastructure spending been for so long that the financial returns to justify further debt are nonexistent. Nor can he free the economy further without endangering the party’s control. His options are a combination of hoping hi-tech, driven by extravagant R&D spending, will offer a stronger economy (along with more repression as a safety fallback and finding enemies around which the country can unite).

The speech last week warning China might go to war to end Taiwan’s independence was the most bellicose of any Chinese leader since Mao. Be in no doubt – if economic difficulties worsen, Xi may be compelled to rally his restive population around a limited conflict just to stay in power.

In a mirror image, Apple is spending as aggressively on R&D as China, hoping that will solve its problems too. Apple is indubitably innovative and the scale of its research should throw up new products; it is already developing a great service business. For China, the message is even starker. A one-party state can’t risk any bumps – and there are more than bumps ahead. Nothing, not even the wondrous success of the iPhone, lasts for ever. And that is especially true for a dysfunctional Chinese economy, and the party that runs it.

Protein mania: the rich world’s new diet obsession

Why we can’t get enough when we already eat too much. By Bee Wilson in The Guardian


Are you getting enough protein? The question provides its own answer: if you are worrying about the amount of protein in your diet, then you are almost certainly eating more than enough. This is the paradox of our new protein obsession. For many people, protein has become a kind of secular unction: it instantly anoints any food with an aura of health and goodness. On the menu at the gym where I go, a salad niçoise is now repackaged as “high-protein tuna”. It comes without the usual capers or olives – those are items that merely add flavour, and who needs that?

On Pinterest, the lifestyle-sharing site, you can now choose “protein” as one of your interests in life, along with “cute animals” and “inspirational quotes”. In 2017, there were 64m Google searches for “protein”. Anxiety about protein is one of the things that drives a person to drink a flask of vitamin-padded beige slurry and call it lunch.

You merely need to visit a western supermarket today to see that many people regard protein as some kind of universal elixir – one food companies are profitably adding to anything they can. “When the Box Says ‘Protein’, Shoppers Say ‘I’ll take it’” was the headline of a 2013 article in the Wall Street Journal. In addition to the ubiquitous protein balls, protein bars and protein shakes, you can now buy protein noodles, protein bagels, protein cookies and – wait for it – protein coffee. Even foods that are naturally high in protein such as cheese and yoghurt are sold in protein-boosted versions. Strangest of all might be “protein water” – clear, fruit-flavoured drinks laced with whey protein, as if ordinary water was insufficiently healthy.

Around half of all UK consumers are apparently seeking to add “extra protein” to their diets, according to market research from the cereal brand Weetabix – which has also cashed in on our hunger for protein. The protein version of Weetabix – a 24-pack of which costs 50p more than the same-sized pack of original Weetabix – is worth £7m in sales per year.

In a way, there’s nothing strange in the fact that we see protein as valuable, because it is. Along with fat and carbohydrate, it is one of the three basic macronutrients, and arguably the most important. We could survive without carbohydrates, but fat and protein are essential. Protein is the only macronutrient to contain nitrogen, without which we cannot grow or reproduce. There are nine amino-acid proteins – the building blocks of human tissue – that we can only get from food. Without them, we could grow neither healthy hair and nails nor strong bones and muscle, and our immune system would be impaired. A child who lacks vital protein in the first five years of life will suffer from stunting and sometimes wasting, too, as the dreadful persistence of malnutrition in the developing world reminds us.

So the puzzle is not that we should crave protein, but that our protein anxiety has become so acute at a time when the average person in developed countries has a surfeit of protein in their diet – at least according to official guidelines, which recommend a minimum of 0.8g of protein a day per kilogram of body weight. According to 2015 data from the UN Food and Agriculture Organization, the average person in the US and Canada gets a full 90g a day, nearly twice the recommended amount. The average European is not far behind with 85g of protein a day, and the average Chinese person consumes 75g.

When we seek out extra protein to sprinkle over our diets, most of us in rich countries are fixating on “a problem that doesn’t exist”, said David L Katz, an American doctor and public health scholar who is the director of the Yale-Griffin Prevention Research Center. In his latest book, The Truth About Food, Katz notes that while the “mythology of protein tends to propagate the notion that more is better”, there are serious concerns that a very high protein intake over a lifetime can result in harm to the liver, kidneys and skeleton.

The current protein mania has partly come about because so many people now regard carbohydrates or fats (and sometimes both) with suspicion. In the current nutrition wars, protein has emerged as the last macronutrient left standing. But the whole “macronutrient fixation” is a “boondoggle” that has been calamitous for public health, Katz told me. “First they told us to cut fat. But instead of wholegrains and lentils, we ate low-fat junk food.” Then food marketers heard the message about cutting carbs and sold us protein-enriched junk foods instead. “When we talk about protein,” said Katz, “we are dissociating the nutrient from its food source.”

And yet still we try to get more protein. In this world of abundance, humans seem to be on an eternal quest for the one safe substance that we can consume in limitless quantities without gaining weight. Such is the appeal of Diet Coke.

Our protein anxiety drives us to take diets already high in meats, soya, sugars and ultra-processed foods and dose them with yet more meats, soya, sugary bars and ultra-processed foods because they are marketed to us as “protein” – even though many of these products are not even particularly high in protein.

There is something paradoxical about our collective protein worship. When we pay good money for protein-enhanced food, we hope it will lead us to better health (however that is defined). Yet our single-minded pursuit of protein – as a disembodied nutrient whose presence trumps all other considerations – can lead us to behave as if we have forgotten everything we knew about food.

The intensity of our protein obsession can only be understood as part of a wider series of diet battles that go back half a century. If we now thirst for protein as if it were water, it may be because the other two macronutrients – fats and carbohydrates – have each in turn been made to seem toxic in the public mind.

Official dietary guidelines in the US and UK still insist that a healthy diet is one founded on plenty of carbohydrates with limited quantities of fat, especially saturated fat. The rationale for this low-fat advice goes back to the landmark Seven Countries Study, conducted in the 1950s by the American physiologist Ancel Keys. Based on his observation of healthy, olive oil-eating Mediterranean populations, Keys argued that affluent westerners would suffer fewer cases of heart disease if they could limit consumption of saturated fats such as those found in butter, lard and meat.

But as interpreted in the modern supermarket, the low-fat diet often ended up being a high-sugar and high-refined-carbohydrate diet, which was not quite what nutritionists had originally envisaged. In recent years, the low-fat, high-carb orthodoxy has come under fierce attack. In 2015, a meta-analysis conducted by a team of Canadian researchers concluded that intake of saturated fat was not associated with raised risk of stroke, type 2 diabetes or death from heart disease. Vocal anti-sugar campaigners such as Gary Taubes – author of The Case Against Sugar – have argued that the true cause of our current epidemic of diet-related ill health is not in fact saturated fat, but refined carbohydrate.

While the low-fattists and the low-carbists continue to slug it out, protein comes out the winner as the one safe thing that most of the population feel they can still put their faith in, whether for weight loss or general health. We have to eat something, after all.

The current protein fetish is merely the latest manifestation of a far larger phenomenon that Michael Pollan memorably referred to as “nutritionism” around 10 years ago. For decades now, there has been a tendency to think about what we eat and drink in terms of nutrients, rather than real whole ingredients in all their complexity. A combination of diet fads and clever marketing has got us here. It doesn’t matter whether we fixate on “low fat” or “low carbs” or “high protein” – we are making the same old mistakes about nutrition in a new form.

For a while, on my kitchen counter, next to the jars of rice and flour, there was another canister made of black plastic, much larger than the others. Its label said “SOURCE OF HIGH QUALITY PROTEINS” in huge letters. In much smaller lettering it said “READY TO MIX PROTEIN POWDER WITH SWEETENERS” and listed three kinds of whey protein: whey protein isolate, whey protein concentrate and hydrolysed whey protein. When you opened it, a fake vanilla smell wafted into the air and you saw a whitish powder and a black plastic scoop.

This soulless canister of ultra-processed whey protein was something that I, as a food writer, never thought I would see in my kitchen. The macho aesthetic of the packaging made my heart sink. I am also no fan of artificial sweeteners, which I believe do no favours either to the palate or to gut bacteria. What’s more, I believe most people should be able to get the nutrients we need from a balanced diet, rather than through supplements.

But nothing forces you to bend your own principles like parenthood. I turned to whey protein in a state of mild desperation for my very tall youngest son, who plays competitive sport five or six days a week. Three square meals plus multiple snacks only scratched the surface of his appetite, and he was sometimes almost crying with hunger by dinnertime. My conversations with other sport parents suggest that it’s not uncommon to be at least a little bit obsessed with their child’s protein intake. We grumble that protein bars are a pointless rip-off – and then we buy another pack of them.

 

Vanilla protein powder. Photograph: Arisha Singh/Alamy

Protein means different things to different people. To some, it symbolises “weight loss”, while to others it means “muscles”. To me it appeared as a magical filling substance that just might help my son to be less ravenous.

I had read studies suggesting protein was the most filling – or satiating – of the three macronutrients, and wondered if more protein at breakfast could be the answer. I tried him on homemade waffles enriched with almonds and hidden eggs (at that point he wouldn’t countenance whole eggs) and the improvement in his energy levels was dramatic. It was a short step to making him occasional smoothies from half a scoop of whey protein with milk and bananas or frozen berries. Despite my unease at the powder, I could genuinely see the difference in his levels of fullness. When the canister was empty I didn’t replace it, but I still monitor my son’s protein intake.

Having “enough” protein in your diet to meet your basic needs is not necessarily the same as having the right amount for optimum health. When I asked David Katz how much protein a person should consume, he said certain people may indeed require more than the minimum recommendation of 0.8g per kilo of bodyweight – including athletes like my son. The problem is that once we start thinking more protein is automatically better, it can be hard to know when to stop. The idea that protein is synonymous with healthy eating leads many people to eat in disordered ways that are far from healthy, either for body or mind.

A couple of years ago, Sarah Shephard, a thirtysomething British sports journalist, realised she was obsessed with protein. On a typical day, she was eating three or four protein bars, hard-boiled eggs, meat, fish and non-starchy vegetables and a couple of protein shakes. Virtually the only carbs in her diet came from the protein bars and shakes. It reached the point where she had so little energy in the evenings because of the lack of calories in her body that she stopped going out.

Shephard’s protein obsession started when injury forced her to give up running. After she started doing boxing and circuits with a new trainer, he told her she should be eating more protein to help prevent future injuries. To start with, Shephard’s new low-carb, high-protein regime felt wonderful. She lost weight, gained muscle and became one of the many people at the gym clutching their sleek flask of protein shake like an amulet.

However, she noticed her thoughts about protein were becoming obsessive. Given the choice between an apple and a protein bar, she always chose the protein bar, even though at a rational level she knew that a piece of fresh fruit with its fibre and vitamins had a lot to recommend it over a processed snack. By the time she finally sought help from a sports nutritionist, he said he had never seen anyone with such an intensive fitness regime who ate so little carbohydrate. She was consuming 150g of protein a day, around 2.5g per kilo of bodyweight – far in excess of the upper limit recommended for bodybuilders by the US Academy of Nutrition and Dietetics.

Shephard slowly weaned herself back on to a more balanced diet that included a range of complex carbohydrates such as oats and brown rice. Despite her apprehension, she did not gain weight. When I spoke to her, Shephard had been eating a balanced diet for more than two years without any ill effects, and was a bit mystified as to how she had drifted into her protein fixation.

Encouraged by the marketers of high-protein foods, many people talk about whether we we have reached our daily target for “macros”, but we do not talk so much about how much is too much. Adding extra protein beyond our needs can be harmful for people with underlying kidney or liver problems, as the body can struggle to process the excess.

In 2017 there were sensationalist headlines around the world when Meegan Hefford, a 25-year-old Australian bodybuilder, died after consuming high amounts of protein shakes and supplements. Hefford hadn’t realised she was suffering from a condition called urea cycle disorder, which meant that her body couldn’t metabolise protein normally. Defenders of high-protein diets immediately attacked the coverage of the story, pointing out that Hefford’s condition was rare and that her death was not caused by protein per se. This was true, but it is also true that there is a significant minority of the population for whom a high-protein diet is not advisable. For the 13% or so of people who have chronic kidney disease, a large amount of protein from red meat can damage renal function.

Above and beyond its long-term effects on the body, a fixation with protein can become a form of eating disorder. Three years ago, the American psychologist Richard Achiro decided to study men in Los Angeles engaging in excessive use of protein powders as well as other supplements such as caffeine. Achiro conducted a survey of nearly 200 active men who were using workout supplements and found that, for many of them, protein use had become a “variant of disordered eating” that threatened their health.

These men felt under intense pressure to achieve bodies that were not just thin, but that exhibited a supposedly ideal ratio of fat to muscle. Three per cent of the sample group had been hospitalised as a result of excessive supplement use, yet they still viewed supplements as healthy. Eating disorders have complex causes: Achiro told me the men who were overusing protein supplements also tended to be suffering from body dissatisfaction, low self-esteem and a sense of insecurity about their own masculinity.

But they were not helped by the fact that the culture they lived in told them that when they replaced most of their meals with protein shakes, what they were doing was normal. Achiro found that it was hard for these men to recognise that their relationship with protein might have become a problem, “because those of us living in western society have been groomed to view a protein-rich diet as the apex of healthful eating”.

By 2001, Arla Foods, a vast European dairy cooperative with Danish headquarters, had used up all of the whey in Denmark. The company realised it would have to look further afield to meet the insatiable demand for whey protein. Arla signed a contract with SanCor, an Argentinian dairy company, to build a giant whey protein plant in the town of Porteña, to the north of Buenos Aires. When you order “warm protein pancakes” with blueberries at the gym, the odds are the protein will have come from a plant such as this.

It was David Jenkins, a Scottish track and field athlete and silver medallist at the Munich Olympics, who first had the idea of marketing whey protein as a “recovery optimiser” for athletes called ProOptibol. It was launched in health food stores in Southern California and Hawaii in early 1988. At first it was a niche product that found popularity among cyclists and triathletes. The formula for this original whey protein was called WPC 75. It was a byproduct from the Golden Cheese company, based in Corona, California – a giant factory that produced Monterey Jack and other American cheeses.

In just a few decades, whey protein has gone from waste product to aspirational lifestyle enhancer. Whey is the watery stuff left over during cheese-making after the curds are separated off. On traditional dairy farms, it was put to good use in anything from bread-making to pickles, but in the vast American cheese factories of the postwar years it came to be seen as an unwanted nuisance. In US dairy states such as Wisconsin, cheese factories dumped thousands of litres of whey in nearby rivers. It was only in the 1970s, after local authorities placed limits on the dumping of dairy waste, that cheese manufacturers realised they had to find a way to use up this annoying whey. The quality of whey powders – known as “popcorn whey” – was poor, and it was mostly used to feed pigs. The key technology that made whey protein possible was the development of ultrafiltration techniques to pre-concentrate the whey before it was dried. This was when whey protein started to be made on an industrial scale.

There is nothing on the average tub of whey protein to suggest it ever came from cheese, let alone from a cow. Whey manufacturers work on the assumption that consumers want it to be as close as possible to flavourless, to preserve the illusion that it is some kind of magic potion for humans. In its unadorned form, however, whey varies considerably in flavour. There are two kinds of whey: sweet whey made from rennet cheeses such as cheddar and mozzarella, and acid whey made from the likes of cottage cheese. Cheddar whey has a tendency to taste cardboardy, mozzarella whey is milky and whey from cottage cheese can be sour or reminiscent of cabbage broth. But in the finished product, all these flavours are evened out and disguised by the cloying scent of chocolate, artificial vanillin or salted caramel.

 

A selection of energy bars in a supermarket in New York, US. Photograph: Richard Levine/Alamy

Dairy whey protein has become a commodity that travels the world in 100kg sacks, generating vast profits, coordinated by GVCs (global value chains). Thanks to the shifting patterns of supply and demand, the protein shake a gymgoer in Tokyo drinks after lifting weights may have originated on a farm in Norway. The lowest-quality whey powder is called “permeate” and is mostly shipped to Asia, where it is made into infant formula. The higher-quality whey, called WPC 80 because it has an 80% protein content, travels the world to fuel our protein obsession. The global whey protein market is now a complex and hugely competitive global trade, forecast to reach $14.5bn by the year 2023: more than half as much as the global market in breakfast cereal.

Strolling through London at lunchtime a few weeks ago, I found myself walking down Bread Street, near St Paul’s Cathedral, which in medieval times was the site of the city’s bread market. Turning off Bread Street, I came upon a branch of Protein Haus, which claims to sell the “most amazing protein shakes you will ever taste”. The shakes have names such as Strawberry Warrior and Vegan Coffee Pump and Berry-Yatric, which must be a competitor for the most offensive food name ever. Protein Haus also sells protein foods such as “bliss balls” and indiscriminate steaming portions of various meats.

From Bread Street to Protein Haus – this sums up how our eating habits have changed in modern times. When I was in Protein Haus staring at the heaps of overcooked chicken, slabs of salmon and rows of whey protein shakes and vegan protein shakes, it suddenly occurred to me how crazy it is that we should treat all of these varied “protein” substances as if they were in some way identical. A scoop of ultra-processed whey is not, in fact, the same as a grilled salmon fillet, either in nutrition or in the experience of eating it. Salmon – even the farmed kind – will be high in omega-3 fatty acids and vitamin B12, whereas whey protein is low in vitamins and minerals (except for calcium) and fat-free. The only thing these foods have in common is that they tend to be fuel first and pleasure later (if at all).

At the same time, our reverence for protein as the one perfect nutrient tends to completely ignore how the protein we eat is produced, or what the environmental consequences of that production might be. Of the 90g of protein eaten each day by the average American, two-thirds is composed of animal products.

One irony of Britain’s obsession with protein is that we don’t actually produce very much of it. In fact, only 3% of arable land in Europe is given over to protein crops such as pulses, and Europe imports more than two-thirds of its animal feed. Much of the protein consumed in Europe is meat raised on materials that actually originate in South America or the US as soybean oil or other oilseeds and have to be shipped across the world. So long as we largely consume protein from animal sources, our obsession with protein is also likely to be bad for the planet.

At the end of September, at the Aldeburgh food festival in Suffolk, I had a lunch with Nick Saltmarsh, who runs Hodmedod, a company that works with British farmers to produce locally grown pulses. Saltmarsh told me he feels our mania for protein foods has gone so far that we sometimes can’t recognise real protein when it is right in front of us.

Vegetable proteins such as lentils and peas tend to be regarded as “low-quality” compared with meat, eggs and dairy. But Christopher Gardner, a professor of medicine at Stanford University, has argued that this “quality” argument is misleading. His great discovery was that all plant sources of protein – from peanuts to edamame beans – contain all essential amino acids. Admittedly they contain smaller concentrations of the amino acids than meat or eggs, but in the context of a plentiful and varied diet, this doesn’t matter.

The problem is partly that beans and lentils do not fit our tunnel-visioned conception of what protein is. Pulses such as lentils contain around 25% protein but also 25% carbohydrate – which makes them hard to categorise within the dogmatic categories of modern nutrition. Is the lentil a protein (good) or a carb (bad)?

When Saltmarsh takes his product range to food fairs, he finds that bodybuilders and fitness enthusiasts will sometimes pick up a packet of pea flour and say “ooh! Peas!” – because pea protein has become fashionable as a vegan alternative to whey. “But when they see that it also contains carbohydrate, they put it down again,” he said.

Some now shun any meal or snack that can’t be categorised as a “protein” fix. But they aren’t among the millions of people for whom a lack of protein is indeed a real and pressing problem – and who don’t tend to be the ones browsing pea protein at fitness fairs.

The word protein derives from the ancient Greek protos, meaning first. When a Dutch scientist called GJ Mulder first brought the term into use, in 1838, he proposed – rightly, as it turns out – that protein was a crucial substance in all animal bodies. But new research suggests protein is required not just in an absolute sense, but in a particular ratio to the other nutrients in our diets.

Going by official guidelines, as we’ve seen, the average person has access to more than enough protein for general health. It turns out our perplexing protein obsession may actually be a symptom of a larger problem in our sugar-heavy modern diets: if it feels like we are not eating enough protein, it’s because we’re eating too much of everything else.

In 2005, two biologists called David Raubenheimer and Stephen Simpson put forward the “protein leverage hypothesis”, in which they argued that protein could be the missing link in the obesity crisis. Since the 1960s, the absolute level of protein consumed by the average westerner has not changed. What has changed is the ratio of protein in our diets.

Because our overall calorie consumption has risen by 14%, the ratio of protein to carbohydrate and fat has significantly dropped. In 1961, the average American got around 14-15% of calories in the form of protein, whereas today it is 12.5%. This doesn’t sound like a big drop, but Raubenheimer and Simpson’s research suggests that even a small drop in the percentage of protein can have a big impact on eating behaviour – driving us to overeat.

 

Salmon steaks at a Carrefour supermarket in Calais, France. Photograph: Gary Calton for the Observer

Like many other animals, humans have what biologists call a “dominant appetite” for protein. The biological drive for protein is so strong that a cricket who feels short of protein will resort to cannibalism. When a locust is short of protein, it will explore different food sources to redress the balance. Humans are neither as ruthless as crickets nor as prudent as locusts. When given access to a diet that is low on protein and heavy in carbs and fats, humans will binge, in an attempt to extract the protein they need.

If many of us overeat, it could be partly because our bodies are desperately seeking out protein in a food environment flooded with refined wheat and oils and many kinds of sugar. As Raubenheimer and Simpson wrote in their startlingly original 2012 book The Nature of Nutrition: “Dilution of protein in the diet by fat and carbohydrate drives excess consumption, perhaps more so in some individuals, life stages and populations than others.” In other words, obesity may often really be hunger hiding in plain view.

The urgent question raised by this research is how we can get our protein ratios back to a healthy level. Our current protein mania – encouraged by the food business and the whey protein industry – suggests that the answer is to supercharge our diets with a flood of added protein. But eating protein to excess comes with its own costs, the main one being that it tends to shorten the lifespan.

A more effective way to concentrate protein in our diets, Raubenheimer and Simpson say, would be to keep our protein levels constant (assuming we have enough) but cut down on “fat, sugar and other readily digested carbohydrates”, which would allow us to reach the protein our bodies need at a lower level of calories. But given that sugar is poured into everything from bread to stir-fry sauces, this solution would require a radical restructuring of our food environment.

Our protein needs do not remain constant over the human lifespan: 0.8g per kilogram of bodyweight may be enough for a twentysomething, but not for an octogenarian. If anyone needs extra protein, it is not fit young gymgoers, but old people – particularly those on low incomes who may struggle to buy or cook meals for themselves. Instead of protein bars for the young and rich, we need omelettes and chickpea soup for the old and poor. From the age of 50 onwards we progressively lose muscle and our protein requirements become higher, just as appetite tends to decline. Rates of protein malnutrition are alarmingly high among elderly people admitted to hospital.

Most of those who can afford to buy a “high-protein” tuna plate are already well nourished in amino acids. By contrast, in these austere times, many hard-pressed eaters are forced into a kind of protein hunger by the economic circumstances of their lives. Think of the families who go to the chip shop and buy chips with no fish, or the person on universal credit living on plain pasta until the next cheque comes in. This is one reason there is a huge market for savoury snacks that taste like a ghostly echo of the hearty, protein-based meals of the past while consisting of little but refined carbs and oil: roast chicken flavour crisps, barbecue tortilla chips. Raubenheimer and Simpson’s research suggests that the colossal market for these cheap snacks could be another symptom of a world in which – despite all those bars and shakes on the shelves – many are still hungry for protein.

Behind the current protein hype, there is a kernel of truth. A deficit of protein is indeed part of the hugely complex puzzle of what’s wrong with modern diets. The problem is that the right question – am I getting enough protein? – is being asked by the wrong people.

PM Modi's First Interview on New Year 2019