The truth about lying: who does it, and why Are human beings by nature duplicitous?

Psychologist Robert Feldman reports on what his research reveals about fibbing. Plus Pete Docherty, Katie Price and more come clean about their biggest porkies

Hillary Clinton, while campaigning for the US Democratic presidential nomination last spring, described a memorable trip she made to war-torn Bosnia more than a decade earlier. "I remember landing under sniper fire," she said. "There was supposed to be some kind of a greeting ceremony at the airport, but instead we just ran with our heads down to get into the vehicles to get to our base." News footage, however, showed Clinton strolling calmly with her daughter, Chelsea, upon arrival. During that same election cycle, Mitt Romney, a candidate for the Republican nomination, claimed his father, George Romney, a former Michigan governor, had "marched with Martin Luther King". It was soon revealed that the elder Romney had never actually done so.

It would be easy to conclude, particularly in the wake of the MPs' expenses scandal, that politicians are simply by nature duplicitous, and people who seek power are the kind of people who lie. But what if their dishonest behaviour actually makes them resemble us more than it sets them apart?

To attempt to find out just how common lying is, I conducted a study of more than 100 people in ordinary social situations. Two at a time, I had participants meet and spend 10 minutes getting to know each other. I didn't tell them I was conducting a study of lying. Instead, I said I was interested in investigating how people interact when they meet someone new. Most studies of lying involve a fairly artificial setup, but I wanted to reproduce a typical, everyday experience.

I did, though, introduce one other twist. I wanted to know if the frequency of lying might change with the specifics of the conversation. Perhaps some social interactions were more prone to deception than others. To try to find out, I told some of the participants to attempt to come across as very likable, as they might at a party, for example. In other cases, I told one of the participants to convince the other that he or she was very competent, as they might in a job interview. Everyone else I instructed simply to get to know the other person. I reasoned that while assigning goals to certain people did introduce a slightly greater degree of artificiality into the experiment, social situations in which one person is trying to demonstrate his or her charm to another are very common. As far as the pairs of strangers were concerned, deception had no relevance to the study. Something I didn't mention was that each pair's entire conversation would be secretly videotaped.

After the conversation finished, I revealed the secret surveillance to the participants and asked each one to watch the video, identifying, moment by moment, any instances in which he or she said something that was "inaccurate". (I didn't ask participants to report "lies", wanting to avoid their becoming defensive or embarrassed and, consequently, not admitting to deception.)

One conversation, between a man and a woman I'll call Tim and Allison, was fairly typical. Tim was a laid-back college student. Once they were a little bit acquainted, he told her about his band.

Tim: We just signed to a record company, actually.

Allison: Really?

Tim: Yeah, Epitaph.

Allison: Do you sing or...

Tim: Yeah, I'm the lead singer.

Allison: Wow!

Another participant, Natasha, also discussed her musical background, telling her partner how she entered competitions as a pianist and toured the country with a chamber group. It was all more or less what you would expect from two strangers making small talk – one person, directly or unconsciously, trying to impress another with his or her achievements, or just passing the time by discussing his or her life.

What makes this remarkable is that they are all lies. Natasha never toured the country. Tim's band didn't sign with Epitaph. In fact, there's no band at all. And these are only a few examples of what I found to be an extraordinary pattern. Participants in my study confessed to lies that were big and small, rooted in truth and fantastic, relatively defensible and simply baffling. Further, the lying was not limited to those to whom I had given a directive to appear likable or competent. These people lied with greater frequency, but even those with no specific agenda lied regularly.

All told, I found that most people lied three times in the course of a 10-minute conversation. Some lied as many as 12 times, and those are just the lies participants admitted to. It's possible the frequency was even higher. Of course, it would be easy to conclude that the randomly selected participants in my study just happened to be unusually duplicitous, or that some factor in my study induced people to lie far more than they normally would. But my subsequent research on conversations between unacquainted strangers has shown, fairly consistently, that they lie to each other about three times every 10 minutes, both inside and outside the lab.

Indeed, diary research studies, in which participants are asked to record their daily social interactions and indicate which contained lies, show that lying occurs regularly even in the most intimate relationships. Although deception occurs at lower levels, and is often meant to put another person at ease ("Of course you're not putting on weight"), lying is still a routine part of the rapport between spouses, lovers, close friends and family members.

But why, when they meet strangers, do people feel compelled to make up bands they don't belong to or competitions they never entered? We probably don't spend much time wondering why a tobacco executive would lie about the dangers of cigarettes. Nor does it baffle us when a mechanic tells us a replacement part costs three times more than it actually does. Profit, the avoidance of punishment: these are the sorts of motivations we often associate with deception. We might also include mental imbalance as another motive for lies, yet conditions such as mythomania, or pathological lying, are very rare.

Tellingly, those instructed to impress their partner with their likability tended to tell lies about their feelings. They distorted their true opinions and emotions, often in order to mirror those expressed by their partners. Meanwhile, those instructed to come off as competent tended to invent achievements and plans that would enhance the way they were perceived. But when Tim told Allison about his nonexistent band and nonexistent record contract, he did so without any larger agenda of fooling her. For all he knew, he'd never see Allison again. Tim's lies seemed to involve her only secondarily, and his primary goal to be fostering his own persona or addressing his own insecurities when meeting a new person. To put it simply, Tim's lies were about Tim.

Sometimes, too, lying can be used to benefit the conversation itself. There are times in any interaction when a strict adherence to the truth would only interrupt its natural flow. When a friend wants to tell you about the great time he had at Sam's house over the weekend, and asks, "You know where Sam's house is, right?" the conversation goes much more smoothly if you nod and say, "Oh, yes, Sam's house." I call such deception "lies of social convenience". Indeed, psychologists have found an association between socially successful people and skill at deception. Popular people tend to be good liars.

But why lie to appear competent or likable? Why not just be yourself? In fact, "just being yourself", if we examine it closely, takes creative effort. Our expression of who we are involves choices that reflect social and interpersonal context, our mood, our personality, our need to maintain our self-image and so on. If we consider self-presentation as a creative process, we can see how it can easily slide into deception. Every interaction involves decisions about which attributes to emphasise and which to minimise, which impulses to follow and which to ignore. At some point, we may not be choosing among our actual traits and our sincere reactions. We may simply fabricate the traits and reactions the social situation calls for, or that we think it calls for. In other words, we might lie.

Alexi Santana's application to Princeton University must have stood out right away. Rather than include an essay about a backpacking trip or his private school education, Santana discussed his life as a parentless ranch hand: sleeping outdoors, teaching himself the great works of world literature, and running for miles in the Nevada wilderness. An admission officer's dream, Santana was accepted into the class of 1993. He became a sporting standout and earned mostly As.

Then, at a sporting event, a Yale student recognised Santana as James Hogue, an ex-convict in his 30s who had posed as a high school student years earlier. Hogue was eventually jailed for defrauding the Princeton student-aid office of financial assistance.

You could write volumes about the pathology of people like Hogue – what compels them to deceive, how they live with their fabrications. The causes underlying impostorism remain a topic both fascinating and elusive. Many psychologists would argue that even the perpetrators don't really know why they've carried out their deceit. Yet perhaps the most incredible part of Hogue's unlikely story is that for so long, he got away with it. He was far from the polished, unflappable charlatan we expect from movie and television portrayals of con men, yet he fooled some of the brightest, most educated people in America. Hogue's success illustrates an essential truth about deception, one we rarely recognise: lying is easy.

Study after study has shown that most people have a great deal of faith in their ability to catch a lie. Few of us think of ourselves as pushovers, easily susceptible to cons and dishonesty. But in 2006, psychologists Charles Bond and Bella DePaulo analysed tens of thousands of individual performances and found that people can differentiate truth from lies only 47% of the time. In other words, we are actually a little worse at figuring out when someone is deceiving us than we would be if we just guessed randomly.

Why is that? First, let's assume lie detection is a skill like any other. The way you acquire and improve upon a skill is through practice. How, though, does one practise lie detection in ordinary life? Identifying a lie is difficult to master. We think that when someone tells a lie, there are red flags: a person lying will avert his gaze. He'll shuffle his feet or drum his fingers. If it's a particularly big lie, he might even start to sweat. When we don't see these red flags, we often decide the person is telling the truth.

Yet experts on deception have concluded that there are no physical tics that universally signal that a person is lying. Individual differences in how people lie are strong. One person might blink rapidly when she lies; another might stare at you, taking elongated pauses between blinks. Further, practised liars learn their own giveaways (or the conventionally assumed giveaways, such as gaze aversion) and teach themselves to avoid them.

Even polygraph machines are unreliable. The polygraph is predicated on the idea that when people lie, they experience anxiety, but the fact is that some people don't get anxious when they lie. Some even experience pleasure. Paul Ekman of the University of California is one of the leading researchers in the area of nonverbal behaviour and deception. He has identified a feeling among liars he calls "duping delight". Just as people will jump out of aeroplanes and climb mountains precisely because of the physical and mental challenges, so, too, do people find an almost recreational thrill in deception.

Once we recognise that it is possible to enjoy a lie with intent, this form of deceit becomes more understandable and more complex. It is not just the function of the lie that matters. It is the form, too. The act of telling the lie brings a kind of profit: an adrenaline rush, a feeling of superiority or accomplishment. Just like a lie that defends self-esteem, one with intent can make a liar feel good.

Meanwhile most of us, in fact, don't spend a lot of time in our daily lives wondering, "Am I being lied to?" This psychological phenomenon, in which we assume we aren't being deceived, is known as the truth bias: our default belief is that other people are telling the truth. Someone needs to give us a compelling reason to think they're lying; otherwise the idea never occurs to us. Recent thinking in the psychological community suggests the truth bias operates as a judgment heuristic, or cognitive rule of thumb. Rather than assess every situation based on all the available information, we use subconscious mental rules to make quick determinations about things. To scrutinise a statement for the truth takes up mental energy – and we like to save that when we can. But this allows liars to float beneath our cognitive radar.

Other times, we simply don't want to uncover a lie. It's not only less strenuous cognitively, it's also more flattering and comforting to accept certain statements at face value. But, like the truth bias, what I call the "willing accomplice principle" may operate more powerfully than we might expect.

Imagine that an estate agent is showing you a house. Not surprisingly, she is falling over herself to praise it. You know enough to be sceptical of someone with an obvious financial motive but you love the house, too. If you are like most people, your initially sceptical assessment of what the estate agent says probably begins to change: instead of being on guard against deceit, you will start to want to believe her. In this way, we enter into a kind of unstated conspiracy with liars.

In many cases, a liar and the target of the lie both benefit from the lie's success. If a liar can hit upon deception that we'd like to believe, too, we're both hoping on some level that the lie won't be revealed for what it is.

Consider the collapse of the American sub-prime market in 2008. Lenders extended loans to those with terrible track records of repayment. There was mutually convenient deception on both sides: borrowers with miserable credit ratings assured lenders that this time they would repay their loans; lenders assured borrowers that astronomical interest rates wouldn't lead them to financial ruin and eventual default. Both sides had a financial stake in allowing the deception to continue. Economic forces, though, rarely tolerate such arrangements. The (now seemingly inevitable) collapse of the market triggered global repercussions that are still being felt.

Although it seems clear in retrospect that vigilance in seeking the truth would have been appropriate, the problem is that the cognitive rules we play by – the truth bias, our trust in flatterers, our need for cognitive efficiency – are not, as the Princeton professors discovered, ones we can easily alter.

• This is an edited extract from The Liar In Your Life: How Lies Work And What They Tell Us About Ourselves, by Robert Feldman, published by Virgin Books on 13 August at £18.99. To order a copy for £16.99, with free UK p&p, go to guardian.co.uk/bookshop or call 0330 333 6846.

Famous fibbers
"The last lie I told was 'not guilty'."
Pete Doherty, musician

"Earlier I said, 'I'm ready, I'll be down in a minute.' But I was only getting in the shower. Half an hour later…"
Katie Price, businesswoman

"I've never lied in all my life."
Arthur Smith, comedian

"My last lie was, 'I am absolutely fine.' If you ask anyone over 70 how they are and they say, I am absolutely fine' they are lying."
Jilly Cooper, author

"I often lie about my name. If you were called Lewycka, wouldn't you? I say Mary Lewis, or occasionally I branch out into something more exotic like Lucinda Firestorm."
Marina Lewycka, author

"The most recent outright lie I told was to my publisher, about how fantastically well I was getting on with writing my current book."
John Simpson, journalist

"I'm like everybody else: if somebody says, 'Do you like my new dress?' then I'm going to say yes."
Ann Widdecombe, MP

"I declined work by claiming that I was busy doing something else. And I didn't have to do it, my agent did it very politely on my behalf, so the lie was once removed. But it was me lying."
Reece Shearsmith, comedian

"The only lie I ever tell is in answer to the question: 'Can you come to my wedding/birthday party/baby's christening?' 'Oh, when is it?... Oh, damn, I can't.' This has gone wrong only once (and we put it in The Office) when someone where I used to work invited me to their party and I said: 'Oh, I can't… When is it?' The biggest ever white lie I had ready was when my mum was dying. If she asked me if I thought there was a God, I planned to say, 'Yes. Definitely.' She never asked. I wish she had."
Ricky Gervais, comedian.