How algorithms secretly shape the way we behave

Algorithms, the key ingredients of all significant computer programs, have probably influenced your Christmas shopping and may one day determine how you vote

• 
• • John Naughton
  • The Observer, Sunday 16 December 2012

Program or be programmed? Schoolchildren learn to code. Photograph: Alamy

 

Keynes's observation (in his General Theory) that "practical men who believe themselves to be quite exempt from any intellectual influences, are usually the slaves of some defunct economist" needs updating. Replace "economist" with "algorithm". And delete "defunct", because the algorithms that now shape much of our behaviour are anything but defunct. They have probably already influenced your Christmas shopping, for example. They have certainly determined how your pension fund is doing, and whether your application for a mortgage has been successful. And one day they may effectively determine how you vote.

On the face of it, algorithms – "step-by-step procedures for calculations" – seem unlikely candidates for the role of tyrant. Their power comes from the fact that they are the key ingredients of all significant computer programs and the logic embedded in them determines what those programs do. In that sense algorithms are the secret sauce of a computerised world.

And they are secret. Every so often, the veil is lifted when there's a scandal. Last August, for example, a "rogue algorithm" in the computers of a New York stockbroking firm, Knight Capital, embarked on 45 minutes of automated trading that eventually lost its owners $440m before it was stopped.

But, mostly, algorithms do their work quietly in the background. I've just logged on to Amazon to check out a new book on the subject – Automate This: How Algorithms Came to Rule Our World by Christopher Steiner. At the foot of the page Amazon tells me that two other books are "frequently bought together" with Steiner's volume: Nate Silver's The Signal and the Noise and Nassim Nicholas Taleb's Antifragile. This conjunction of interests is the product of an algorithm: no human effort was involved in deciding that someone who is interested in Steiner's book might also be interested in the writings of Silver and Taleb.

But book recommendations are relatively small beer – though I suspect they will have influenced a lot of online shopping at this time of year, as people desperately seek ideas for presents. The most powerful algorithm in the world is PageRank – the one that Google uses to determine the rankings of results from web searches – for the simple reason that, if your site doesn't appear in the first page of results, then effectively it doesn't exist. Not surprisingly, there is a perpetual arms race (euphemistically called search engine optimisation) between Google and people attempting to game PageRank. Periodically, Google tweaks the algorithm and unleashes a wave of nasty surprises across the web as people find that their hitherto modestly successful online niche businesses have suddenly – and unaccountably – disappeared.

PageRank thus gives Google awesome power. And, ever since Lord Acton's time, we know what power does to people – and institutions. So the power of PageRank poses serious regulatory issues for governments. On the one hand, the algorithm is a closely guarded commercial secret – for obvious reasons: if it weren't, then the search engine optimisers would have a field day and all search results would be suspect. On the other hand, because it's secret, we can't be sure that Google isn't skewing results to favour its own commercial interests, as some people allege.

Besides, there's more to power than commercial clout. Many years ago, the sociologist Steven Lukes pointed out that power comes in three varieties: the ability to stop people doing what they want to do; the ability to compel them to do things that they don't want to do: and the ability to shape the way they think. This last is the power that mass media have, which is why the Leveson inquiry was so important.

But, in a way, algorithms also have that power. Take, for example, the one that drives Google News. This was recently subjected to an illuminating analysis by Nick Diakopoulos from the Nieman Journalism Lab. Google claims that its selection of noteworthy news stories is "generated entirely by computer algorithms without human editors. No humans were harmed or even used in the creation of this page."

The implication is that the selection process is somehow more "objective" than a human-mediated one. Diakopoulos takes this cosy assumption apart by examining the way the algorithm works. There's nothing sinister about it, but it highlights the importance of understanding how software works. The choice that faces citizens in a networked world is thus: program or be programmed.

So you thought Britain wasn't corrupt?

Mary Dejevsky:

Two of the most deep-rooted maladies of British society are freebies among friends and jobs for the boys

Friday, 22 July 2011 The Independent

Anyone who had expected to drowse through the Home Secretary's Commons statement on the Metropolitan Police might have awoken with a start when she began with "allegations about police corruption". It was the flat, almost casual, way in which Theresa May appeared to accept at least the possibility, that surprised and the use of the actual words "police corruption". She went on to announce a review of "instances of undue influence, inappropriate contractual arrangements and other abuses of power in police relationships..."

The reason this bald catalogue shocks is that Britain has long projected an image of itself as a paragon of good governance and the rule of law, to the point where experts on such matters earn a good living advising other countries how to emulate our standards. It also happens to be an image that the vast majority of its citizens share. We regard ourselves as mercifully free of the sort of corruption that blights the lives of, say, Nigerians, Egyptians or Russians, and a cut above most southern Europeans.

That may be how we see ourselves, but it is not quite how others see us. Transparency International, an independent organisation which monitors this sort of thing, places the UK 20th in its latest (2010) corruption perception index. Overall, this may not look so bad – 178 countries are listed. Look closer, though, and you will see that the UK comes well below all the Nordic countries, below Luxembourg, Ireland and Germany, and just below the small Gulf state of Qatar. It is only marginally ahead of the United States, France and Spain. Is this where Britain should be – in 20th place, and falling?

Corruption, of course, takes many forms. In some countries, bribery is so prevalent as to be tantamount to a tax. Indeed, a theory has recently been advanced that this is how it should be regarded and that it is perhaps not so reprehensible after all. In others, an unofficial tariff – ranging from a box of chocolates to a luxury holiday – dictates access to the best educational establishments, the best hospitals, the best flats. In yet other countries which would not generally be regarded as particularly corrupt, contributions to political parties constitute a perennially murky area in which even otherwise distinguished politicians have come to grief, such as the former German chancellor Helmut Kohl. You might argue that the US system of lobbying is a form of legalised corruption.

Generally, these are not ills that afflict the UK. If you live here, you can probably be fairly confident that you will not have to offer teachers a backhander for admitting your child or ensuring a decent grade. (Although I have heard tell of quite lavish gifts offered.) You will not have to pay a doctor for decent NHS treatment or a fast track up the transplant waiting list. (Although, again, there is apocrypha that hints at exceptions, and it was once intimated to me that a consideration might keep my husband classified as a UK resident when we were living abroad, so that he would still qualify for expensive drugs on the NHS.)

And you probably won't find a speed cop or parking warden suggesting that a small transaction "between us" would "fix it" before he writes out the ticket, or a frontline immigration officer nodding through someone with some crisp banknotes, but no visa, in his passport. Or election officers stuffing ballot-boxes after the polls have closed.

But you will find ways in which Britain falls very far short of Scandinavian-level probity; areas where complacency has meant a blind eye is turned to abuses, and grey zones where transactions take place that are not actually illegal, but which would – and should – embarrass one or both parties if they became public.
Several such instances emerged earlier this week when the Commons Home Affairs Committee questioned the Commissioner of the Metropolitan Police, Sir Paul Stephenson, and the Assistant Commissioner, John Yates – both of whom had just resigned – as well as the head of public affairs, Dick Fedorcio. The most blatant was Sir Paul's acceptance of hospitality from Champney's health farm.

The commissioner may have been recovering from very serious illness (as he was), he may have declared the gift in the register as required (which he said he did on his return to work), and the owner of Champney's may have been a family friend (though it is unclear how close). But the value of this gift – around half the average Briton's annual pre-tax salary – and Sir Paul's apparent inability to understand that accepting it sat uneasily with his position as the country's most senior police officer on a salary of more than £250,000, suggests a blind spot. It left the impression that there was one law, and one set of subsidised living standards, for the well connected, and another for everyone else.

Something similar applied when it came to the hiring of Neil Wallis, former deputy editor of the News of the World, as a media consultant. Despite some close questioning – notably from two sparky new female Tory MPs, Nicola Blackwood and Louise Mensch – there was precious little clarity about how Wallis actually got the job. Between the lines, however, it could be deduced that there was no open advertisement, no standard recruitment procedure, no formal interview and no public disclosure of the appointment. This was a public-sector, tax-payer funded position, yet contacts and networks appear to have been all.

What we have here are two of the most deep-rooted maladies of British society: freebies among friends and jobs for the boys. And there will be many who shrug and say that this is just how the country works. Yet these ingrained ways of doing things are part of the reason why the UK comes below Finland, Australia and Canada in TI's corruption perception index. They are also a reason, along with our segregated schools, why social mobility in Britain is so relatively poor. Advantage compounds advantage.

At root, much of the disparities come down to information and the way so much is still kept from "prying" eyes. The UK-based American journalist Heather Brooke, who has made opening up what she calls Britain's "information cartel" something of a personal crusade and whose work led to the publication of MPs' expenses, notes that records available to US journalists as a matter of course are "off-limits" here, where access to information "depends on one's wealth, power or privilege". She is right – yet the responses, when she argues this, are not all approving. Some accept that it was ever thus; others accuse her of poking her nose into places it does not belong.

Nor has the Freedom of Information Act so far brought the transformation it should have done. Quite basic information still has to be applied for. This government's efforts to open up details of department and local council spending are laudable, but there has hardly been a rush to comply. Until our patronage system is tackled, British boasts of incorruptibility will remain boasts – discredited by our 20th place on the global corruption index and our continuing fall, as those below us move to clean up their act.
m.dejevsky@independent.co.uk