Are economists selfish? Does studying economics make you selfish

Tim Harford in The FT

Against my better judgment, I was recently prevailed upon to play a game of Monopoly with the family. It soon developed in a fashion that has become familiar: everyone tried to rip everyone else’s face off, except me. I proposed a mutually beneficial deal to each player, offering extra concessions myself to make sure those deals got done. This dealmaking tactic didn’t go down well. Every time I reached an agreement with one player, the other players fumed. 

Before long, I was being roundly denounced as a ruthless exploiter of innocents. This made me sad. It was partly the disheartening realisation that I am clearly a punchable opponent at the board-game table. But there was something more. My plight in being cast as a pantomime villain seemed to stand in for the fate of economists as a whole. 

I should explain. In looking for advantageous trades, I was doing what comes naturally to economists. The basic building block of economic activity — so basic that we take it for granted — is two people making each other better off by finding gains from trade. You can easily spot the economists at the Monopoly table, they’re the ones trying to find the deals that make both sides happy. 

But are we lauded for our fascination with voluntary agreements for mutual benefit? We are not. Instead, economists are often accused both of celebrating selfishness and of being selfish. As Yoram Bauman, an economist and comedian, once joked: “The only reason we don’t sell our children is that we think they’ll be worth more later.” 

What have we done to earn this reputation for ruthlessness? Perhaps it’s that altruism and charity are not front and centre in economic analysis. It may be the character of Gordon Gekko in Wall Street (1987), assuring us that “greed, for lack of a better word, is good”, somehow being associated with economists. 

But our reputation for being calculating and unfeeling may also be thanks to the experimental evidence. Over the years, a series of studies have emerged which seem to show that studying economics causes students to behave more selfishly. 

The basic idea sounds plausible. If you sit in enough classes being told that people are fundamentally self-interested, you yourself might become more self-interested. A 1993 paper by Robert Frank, Tom Gilovich and Dennis Regan summarised some of this evidence. It found economics students tended to behave less cooperatively in experimental games. They also expected less honesty from others. For example, if asked whether they would expect a stranger who found some lost cash to try to return it. More recent research by Bauman and his colleague Elaina Rose found that economics students were less likely to contribute to two named charities in a classroom exercise. 

Yet there is a pair of big question marks hanging over this collection of studies. The first is whether economics teaches people to be selfish, or whether instead selfish people gravitate towards economics. Bauman and Rose note that economics majors are equally mean whether they are near the beginning or the end of their studies — in other words, perhaps economics has no effect on people’s generosity, but big-hearted people avoid economics classes. 

Perhaps more important, do these questions really measure honesty, selfishness or any other moral virtue? That’s not clear. In the Bauman and Rose study, for example, the two charities in question were both left-leaning activist groups. So did economics students refuse to contribute because they hate giving to charity? Or did they feel that these particular charities were not very worthy causes? 

As for classroom exercises, there is a sense in which the selfish move is the “correct” answer in certain experimental settings, such as the prisoner’s dilemma game. If a student is taught that, and then plays the selfish move, have they become more selfish in everyday life? It seems just as plausible to suggest that they have been taught how to reproduce the textbook answer in an academic setting and want to pass the economics test. 

There are certain tendencies in mainstream economics that might nudge people towards a cynical view of human nature, but there is also a long tradition in economics arguing that free markets promote co-operation, honesty, respect for others, freedom and reciprocal benefit. 

So does studying economics make you selfish? A new study with that title, by Girardi, Mamunuru, Halliday and Bowles, finds “no discernible effect” of studying economics either on self-interest or on the belief that other people are self-interested. 

I suggest that before besmirching the good character of economics students we should look for more convincing real-world evidence. So far I have found nothing. But my search did turn up the fascinating discovery — courtesy of the philosopher Eric Schwitzgebel — that books about moral philosophy were more likely to be missing from libraries than other philosophy books. A deep academic interest in ethics appears to be correlated with larcenous behaviour. It makes you think. 

Ironically, the game that inspired Monopoly, The Landlord’s Game, was designed by the activist and writer Elizabeth Magie to teach lessons about a fairer taxation system, and then refined by a socialist professor of economics, Scott Nearing, and his students. Yes, the economics nerds were proposing a co-operative, pedagogical version of Monopoly. Alas their vision was eclipsed by the ruthless battle of attrition we all know today. 

Our own session of Monopoly might have been more fun if only my fellow players had embraced the constructive, co-operative spirit of economics. Alas, they did not, so our game finished in the traditional fashion. It petered out with no clear winner and several sore losers.  

Young people are wising up to the Great British student rip-off – and they’re voting with their feet

As universities wind down teaching for yet another round of exams, more and more prospective graduates are asking: why bother? writes Simon Jenkins in The Guardian

This week begins one of the worst deals offered by any British professional institution. Almost all universities are about to stop teaching students and subject them to pointless exams, mocks and quantification, before passing or failing them, then packing up and reassembling some months later in September. For an average price of tens of thousands of pounds a head (except in Scotland), most students will get virtually no teaching for a good proportion of their course. From any other service – medicine, law, accountancy – this would be regarded as a scam.

The tradition of scholars teaching academic subjects part-time while doubling as researchers is a relic of medieval monasticism. Oxbridge operates for just 24 weeks a year while many other universities operate two semesters. Staff and buildings may be otherwise employed, but students will sit idle, doing odd jobs or studying on their own. No one dares challenge this system. Whitehall inspectors never declare universities “failing” or “inadequate” as they do schools.

But I sense the worm is turning. Last year the percentage of British school leavers going to university fell for the first time – other than briefly in 2012, when the £9,000 fees came in in England. Even before lockdown and the years of online-only teaching, an Ipsos Mori poll showed a falling demand for university among school-leavers, with just 32% being “very likely” to go in 2018. The same trend is evident in the US where college enrolments have been falling for over a decade.

Meanwhile industrial and professional apprenticeships are rising fast. At Lloyds Bank last year, 17,000 school-leavers applied for 215 vacancies. The exam bluff was called by EY’s Maggie Stilwell, who said there was “no evidence” to conclude that exam success correlated with career success. Personal qualities and professional training were what mattered. Her firm, along with accountants PwC and Grant Thornton, have dropped any requirement of degree classes or even A-level results from their application forms. The new “degree apprenticeships” offered by firms such as Dyson and Rolls-Royce are popular, with some 30,000 offered last year. The Institute of Student Employers records that a declining half of firms now ask for a class of degree, and a quarter explicitly state “no minimum requirements”. In Silicon Valley it is even known that an acceptance letter from Stanford University can be sufficient to secure a job. Why waste years swotting for meaningless exams?

The age-old debate over whether a university is really an investment, personal or national, as opposed to a middle-class finishing school has never been resolved. British graduates on average earn £10,000 more than their non-graduate contemporaries, but surely some students might have done equally well with the same number of years’ work under their belts, perhaps studying a favourite subject part- or full-time later in life.

During his brief career as universities minister, Jo Johnson at least hinted at radicalism. He questioned the one-size-fits-all residential university. He floated shorter courses, shorter holidays, broader subjects, more intensive teaching and lifelong learning. He might have added that artificial intelligence is posing a whole new challenge. Johnson may now have gone, but the marketplace is talking. This most reactionary of British institutions may yet be forced to waken from the sleep of ages.

The Great UPSC Coaching Fraud

 

Nandini Rao - Teacher of the Bulbuls

 

How to plan your exam preparation

 

How to study for exams?

How to plan for your exams

Essay Memorisation Techniques

How to learn new content

Essay Writing Workshop

Revision tips

 

Spaced Repetition

How to learn new content

Cheating on online exams

Pervez Hoodbhoy in The Dawn

COVID-19 has made in-person exam proctoring impossible and so normal safeguards have disappeared. My inbox is full of anguished emails from university students across Pakistan bewailing the use of unfair and unethical means by their class fellows. Upon combining these complaints with those of my colleagues in various universities, and adding in my own online teaching experience, a frighteningly dismal picture emerges.

Almost every university student in this country cheats. Perhaps the actual figure is lower (80-90 per cent?) but it’s hard to tell. Many students say they are reluctant and would opt for honesty if there was a level playing field. But exercising virtue brings bad grades or even failure. Rare is the student with strong moral conviction — or perhaps lack of opportunity — who is not complicit.

A system full of holes is easy to beat. Not regarded as a significant moral crime, cheating was plentiful even in the days of in-person classes. But with online exams, the bottom has dropped out. Knowing their paychecks will be unaffected, many teachers don’t care what their students do. If one is somehow caught, cheating can always be deemed to be that student’s fault. After all, the pathways to cheating are so many. 

Consider: while taking an exam the home-bound student supposedly sits facing his/her laptop camera without access to books, notes, or smartphone. Correspondingly, the teacher is supposed to be eagle-eyed, watching many students simultaneously on Zoom or MsTeams. Neither supposition is true. For example moving slightly out of the camera’s field of view allows the student to copy the question and insert it into the Google search bar of that laptop or a hidden smartphone. The answer pops up even before he/she fully finishes typing.

What of a question which Google cannot answer? Such slightly clever questions can indeed be devised by a conscientious professor. One shared with me how that worked out with her class of 30. In an exam none of her students got any question right. But, upon inspection, it turned out that every wrong answer belonged to one of six near-identical sets. Conclusion: the students were either sitting in the same room or had created WhatsApp groups with members messaging each other during the exam.

From a frustrated student who emailed me from an engineering university in Karachi, I learned something brand new after which I explored the matter further. Fact: there exists a plethora of commercial companies that will get you the required answer for almost every exam question. Among them are study aids Chegg, Quizlet, Course Hero and Brainly.

The ones I tried out with physics and math problems give instant answers. All you need to do is cut and paste the exam question into the indicated box. These answer services use artificial intelligence and operate without human intervention. While not cheap, they are affordable. According to my informant, students pool in to buy a subscription and then share answers over WhatsApp. More expensive are answer services staffed by human expert essay writers. The student need provide only basic information such as the topic and some course materials.

Special automated proctoring services, hired by overseas educational institutions, can catch cheaters who are taking their exam at home. These services block browsers from accessing forbidden websites, check to see if the student has contacted a friend or answer service, verify identity and geographical location, and see if the student is looking at flash cards or boards, etc. Some can even detect Bluetooth devices and suspicious movements of the test-takers’ head, keystrokes, and eyes.

Although such proctoring services probably have some value overseas, their utility in Pakistan is doubtful and they are not used. Apart from the cost, they also assume that a student has a quiet room, wide-angle webcam, and stable internet connection. This excludes rural areas but even in cities the last condition is not easily fulfilled.

Can any online exam work in these circumstances? The answer is: yes. A one-on-one oral exam over Skype or Zoom is the only totally safe method. But this is tedious for large classes and checks only a small aspect of his or her learning. To my knowledge, only a few university teachers use it.

Despite difficulties in evaluating students, online university education has worked reasonably well in some countries. Indeed, there are distinct advantages in going digital: an instructor’s recorded lectures can be rewound and reviewed at will for self-paced learning, students can ask questions online without feeling intimidated, and learning is available 24 hours a day. Additionally, a wealth of information and knowledge is just a click away and helps a student understand difficult points.

Why then is online learning failing so miserably in Pakistan? Why has fancy 21st-century education gadgetry not excited our students’ imagination? Why don’t our academic environments sparkle with energy? Two obvious reasons stare at us. First, the generally uninspiring online lectures delivered by teachers. Second, most students and many teachers have insufficient mastery over English to usefully engage with internet learning materials.

But a more serious, much deeper reason underlies this failure. Pakistan’s education system gives importance only to getting high grades, not to actually learning a subject. Even a good teacher — and these are few and far between — cannot make a student study, read books, meet schedules, and take responsibility. Real learning is purely voluntary. Largely a result of childhood training, it cannot be forced upon students. There is an age-old adage: education is all about learning to learn. The internet and Google have made this clear as never before. Every student today has good grades but only a few actually learn while in college or university.

Although our student body is hyper religious and regular in prayer, almost all are perfectly comfortable with cheating. But online testing cannot work unless cheating is viewed for what it is — a white-collar crime. Students willing to experiment, question, model, and wrestle with a problem alone can benefit from 21st-century online education. The bottom line: Pakistan’s education system must change direction. It must seek to create a proactive mindset and an ethical community.

How To ABSORB TEXTBOOKS Like A Sponge

India's 'cheating mafia' gets to work as school exam season hits

Vast network profits from the desperation of students and parents to get ahead in a country where university places and jobs are limited


Michael Safi in The Guardian
 

Men at stalls selling schoolbooks on College Road in Kolkata at the height of India’s annual exam season Photograph: Michael Safi


Afew minutes into the final year maths exam at his Delhi high school, Raghav asked to use the bathroom. Inside, he texted pictures of the test paper he had secretly photographed to a phone number he was sent days before. Minutes later, answers materialised on the screen.

“It isn’t cheating,” insists his mother Sunita, who paid 16,000 rupees (£175) for her son to obtain the phone number. “It’s a way out.”

India’s annual exam season has gripped the country in the last month, with tens of millions of students undertaking gruelling tests to qualify for the limited slots available at Indian universities – the best of them with admission rates about one-tenth those of Oxford and Cambridge.

Also hard at work is the country’s so-called “cheating mafia”, the vast network aimed at profiting from the desperation of students and parents to get ahead in a country where, each year, an estimated 17 million people join a workforce adding only 5.5m jobs.

Last week, in the latest high-profile breach, the papers for two secondary exams were found to have been leaked on WhatsApp about 90 minutes before the tests. More than 2.8 million students in Delhi and the surrounding areas have been ordered to resit the exams later in April.

“It is mental torture,” said Kirath Kaul, 15, an east Delhi student who will be forced to sit a new maths exam this month. “I was spending all day study [for the last one] and even getting up at night to prepare.”

‘A broken education system’

Cheating on exams in India is endemic, organised and elaborate. In Bihar, one of the poorest states in the country, more than 1,000 students were expelled for cheating in February.

Last year, the student who topped the state in one subject, arts, turned out to be a 42-year-old man. The student with the highest arts score in 2016 was stripped of her certificate after arousing suspicions, including by telling a TV interviewer she believed political science was the study of cooking.

In 2015, Bihar made global headlines when videos emerged showing parents literally scaling a five-storey building to pass answers to their children taking exams inside. This year, to ensure probity, the state installed CCTV cameras in exam halls and made every student leave their shoes and socks at the door.

“This is very much the sign of a broken education system,” says Yamini Aiyar, the chief executive of the Centre for Policy Research.

She blames the widespread fraud on a cocktail of intense pressure to earn a university qualification and a system that has been focused on building new schools, but unconcerned about what happens inside them.

“Studies tell us that on average about half of students that get to Standard 5 can read a Standard 2 text,” she says.

Incentives for teachers and administrators are also askew, she adds, measuring success and doling out promotions based on what percentage of children in a school or district are passing their exams. Officials are incentivised either to assist in the cheating or overlook it.

Sunita, who asked for her name to be changed for legal reasons, came into contact with India’s “cheating mafia” through the exam coaching centre Raghav have been attending before his final exams last year.

“The teacher said, your son is very weak ma’am,” she recalls. “My son was not interested in studying and I didn’t want him to repeat the year.”

The private tutor offered to connect with someone who could send Raghav the answers for his economics and mathematics papers. Nobody on either end of the phone would know each others’ identity. She signed up with four or five other families.

“It is totally common,” she says. And lucrative. “All of us probably paid this man about 60,000 rupees.”

India’s cheating industry has proliferated alongside others kinds of organised fraud such as call-centre scams, and is driven by a similar mentality, says Snigdha Poonam, author of a new book about the ambition and guile of Indian youth.

Social mobility in India is improving, according to surveys. But not fast enough to match the expectations of a generation reared on social media, western pop culture, and promises that India’s time has finally come. “The same forces drive young Indians into entering these economies built on fraud,” she says.

“[It is] the lack of legitimate options in the formal job market, a blurring of lines between honest and dishonest work, an ability to identify market gaps for services, and the resourcefulness to turn them into avenues for easy money.”

With the aid of his cheating hotline, Raghav passed all his subjects. He is now taking private photography classes and aims to make a career with his camera. Kaul, meanwhile, is studying hard, preparing to take her maths exam again on 25 April.

“But I’m worried the cheaters will perform better [than me],” she says. “I work very hard, but people only see the result – not that someone has cheated.”

The Intellectual Yet Idiot

by Nassim Nicholas Taleb

What we have been seeing worldwide, from India to the UK to the US, is the rebellion against the inner circle of no-skin-in-the-game policymaking “clerks” and journalists-insiders, that class of paternalistic semi-intellectual experts with some Ivy league, Oxford-Cambridge, or similar label-driven education who are telling the rest of us 1) what to do, 2) what to eat, 3) how to speak, 4) how to think… and 5) who to vote for.

But the problem is the one-eyed following the blind: these self-described members of the “intelligenzia” can’t find a coconut in Coconut Island, meaning they aren’t intelligent enough to define intelligence hence fall into circularities — but their main skill is capacity to pass exams written by people like them. With psychology papers replicating less than 40%, dietary advice reversing after 30 years of fatphobia, macroeconomic analysis working worse than astrology, the appointment of Bernanke who was less than clueless of the risks, and pharmaceutical trials replicating at best only 1/3 of the time, people are perfectly entitled to rely on their own ancestral instinct and listen to their grandmothers (or Montaigne and such filtered classical knowledge) with a better track record than these policymaking goons.

Indeed one can see that these academico-bureaucrats who feel entitled to run our lives aren’t even rigorous, whether in medical statistics or policymaking. They cant tell science from scientism — in fact in their eyes scientism looks more scientific than real science. (For instance it is trivial to show the following: much of what the Cass-Sunstein-Richard Thaler types — those who want to “nudge” us into some behavior — much of what they call “rational” or “irrational” comes from their misunderstanding of probability theory and cosmetic use of first-order models.) They are also prone to mistake the ensemble for the linear aggregation of its components as we saw in the chapter extending the minority rule.

The Intellectual Yet Idiot (IYI) is a production of modernity hence has been accelerating since the mid twentieth century, to reach its local supremum today, along with the broad category of people without skin-in-the-game who have been invading many walks of life. Why? Simply, in most countries, the government’s role is between five and ten times what it was a century ago (expressed in percentage of GDP). The IYI seems ubiquitous in our lives but is still a small minority and is rarely seen outside specialized outlets, think tanks, the media, and universities — most people have proper jobs and there are not many openings for the IYI.

Beware the semi-erudite who thinks he is an erudite. He fails to naturally detect sophistry.

The IYI pathologizes others for doing things he doesn’t understand without ever realizing it is his understanding that may be limited. He thinks people should act according to their best interests and he knows their interests, particularly if they are “red necks” or English non-crisp-vowel class who voted for Brexit. When Plebeians do something that makes sense to them, but not to him, the IYI uses the term “uneducated”. What we generally call participation in the political process, he calls by two distinct designations: “democracy” when it fits the IYI, and “populism” when the plebeians dare voting in a way that contradicts his preferences. While rich people believe in one tax dollar one vote, more humanistic ones in one man one vote, Monsanto in one lobbyist one vote, the IYI believes in one Ivy League degree one-vote, with some equivalence for foreign elite schools, and PhDs as these are needed in the club.

More socially, the IYI subscribes to The New Yorker. He never curses on twitter. He speaks of “equality of races” and “economic equality” but never went out drinking with a minority cab driver. Those in the U.K. have been taken for a ride by Tony Blair. The modern IYI has attended more than one TEDx talks in person or watched more than two TED talks on Youtube. Not only will he vote for Hillary Monsanto-Malmaison because she seems electable and some other such circular reasoning, but holds that anyone who doesn’t do so is mentally ill.

The IYI has a copy of the first hardback edition of The Black Swan on his shelves, but mistakes absence of evidence for evidence of absence. He believes that GMOs are “science”, that the “technology” is not different from conventional breeding as a result of his readiness to confuse science with scientism.

Typically, the IYI get the first order logic right, but not second-order (or higher) effects making him totally incompetent in complex domains. In the comfort of his suburban home with 2-car garage, he advocated the “removal” of Gadhafi because he was “a dictator”, not realizing that removals have consequences (recall that he has no skin in the game and doesn’t pay for results).

The IYI is member of a club to get traveling privileges; if social scientist he uses statistics without knowing how they are derived (like Steven Pinker and psycholophasters in general); when in the UK, he goes to literary festivals; he drinks red wine with steak (never white); he used to believe that fat was harmful and has now completely reversed; he takes statins because his doctor told him to do so; he fails to understand ergodicity and when explained to him, he forgets about it soon later; he doesn’t use Yiddish words even when talking business; he studies grammar before speaking a language; he has a cousin who worked with someone who knows the Queen; he has never read Frederic Dard, Libanius Antiochus, Michael Oakeshot, John Gray, Amianus Marcellinus, Ibn Battuta, Saadiah Gaon, or Joseph De Maistre; he has never gotten drunk with Russians; he never drank to the point when one starts breaking glasses (or, preferably, chairs); he doesn’t know the difference between Hecate and Hecuba; he doesn’t know that there is no difference between “pseudointellectual” and “intellectual” in the absence of skin in the game; has mentioned quantum mechanics at least twice in the past five years in conversations that had nothing to do with physics.

He knows at any point in time what his words or actions are doing to his reputation.

But a much easier marker: he doesn’t deadlift.

Your students’ A-level results could easily be wrong

 

Students may not be getting the exam grades they deserve due to the inconsistent marking. Photograph: Alamy


Anonymous in The Guardian

Thursday 13 August 2015 07.00 BST 

Congratulations. Your students have got their grades, university beckons and you can bask in the warm glow of a job well done. Parents, colleagues and students salute you. But are the results accurate? As a senior examiner with more than 20 years’ experience, let me share my doubts.

Perhaps you picture genteel examiners sitting in Oxbridge common rooms, languidly resting on armchairs as they earnestly discuss whether Chloe’s essay merits an A* or merely an A. Maybe you imagine seasoned professionals kindly donating their holidays to mark in the garden over Earl Grey tea and lemon drizzle cake? Wise up. Examining is a ruthless multi-million pound business. There are two types of examiners: the quick and the dead. The faster we mark, the more we get paid. If we’re slow, we fall foul of exam cheat No 1: the exam board.

Take it from an examiner, your students’ A-level results could easily be wrong



It doesn’t matter whether you teach economics with ABC board or further maths with XYZ, they are as rotten as each other. My board ask for two qualifications from their examiners: they are alive and they need the cash.

Mr Simpson turns up year after year marking different papers in my subject because the exam board doesn’t cross reference sackings with recruitment. Think of him as a zombie – we declare them dead, but they reappear. Simpson is “aberrant”, in examiner parlance. This means that when we look at his marking, some scores are too generous, some are too mean and there is no pattern. Fancy having him mark your students’ papers? Ms Griffin, however, is merely a “lingering doubt”. These markers make big mistakes, but there is a consistent generosity or meanness that we can correct.

Speaking of consistency, here’s exam cheat No 2: Ofqual. The quango is charged with ensuring compatibility between the exam boards but its heavy-handed, ruthlessly statistical approach makes everything much worse. Unless exam boards give 80% of their marks to Ofqual by early July, they face severe sanctions, including public naming and shaming. Senior examiners therefore have to apply the thumbscrews to their juniors, with predictable consequences for accuracy.

In a recent report exam boards confessed to “guesstimating” grades. The only shock for me was that they admitted it. I’ve seen a chief examiner (top of the tree in exam terms) take a set of papers from an aberrant marker and come back minutes later with new grades. Usain Bolt couldn’t have moved at that speed. The examiner had clearly just looked up the school’s predicted grades and scribbled them on top of the papers. The moral of the story is to check the grades your centre sends to the board. They are used more than you think.

If in doubt about a result, always go for a re-mark – the numbers of requests are booming. It’s hardly a surprise; some examiners are not even standardised. In standardisation, they are given a sample of pre-marked papers and tested on how well they can match the agreed marks. If they cannot, they are not allowed to continue marking. But there are thousands of orphan scripts left unmarked every summer and my board was so desperate that it summoned the zombies, the lingering doubters and other barrel scrapings to a special centre to mark against the clock. Several of these worthy souls had failed standardisation but were allowed to carry on (paid at several times the normal rate).

It gets worse; there are gangsters out there. I’ve seen papers given the green light despite major reservations about how it’s been graded because an examiner needed to move on to their next marking gig. Exam cheat No 3: examiners.

Secret Teacher: marking exam papers exposes the flaws in teaching



I get paid £4 per script. There’s an adage about peanuts and primates. We genuinely don’t do it for the money, but no one likes to be exploited. Meanwhile, remarks cost £40, so someone is making a lot of money. Markers are poorly motivated and often poorly qualified. My examiners once needed several years of teaching experience, now I’ll take a PGCE student. What unites us is a genuine love of the subject we are marking and respect for the students who are producing the answers. For me, reading a good script is an emotional and resonant experience. Students deserve nothing less than my best, and I try to give it. I cannot say the same for the examining process. Your students’ marks may be right, wrong or anywhere in between.

Unexpected exam hero No 1: former education secretary Michael Gove. He was unpopular, but he had some good ideas, one of which was to reduce the exam boards to one. A single board with consistent standards, fair rules and fair results. One exam envelope. And no more zombies. Full marks, I say.

Marking exam papers exposes the flaws in teaching

It is staggering how many teaching staff I know that do not read examiners’ reports or even the exam specification and so their class often misses out on marks. Photograph: Alamy


The Secret Teacher in The Guardian

A detailed but incorrect answer appears beside every question on an exam sheet. The answers are peppered with technical language but their ideas make little sense. This is one of the most frustrating errors I see as an exam marker.

I took the position up a few years ago after some persuasion from a colleague and the lure of some extra holiday money. I was told that it would be excellent training and help me to become a more effective classroom teacher as I would understand the demands of the exam boards more closely.

Yes it is true that I understand the application of mark schemes better than before, and it definitely looks good on my CV. But I don’t think I can do it much longer.

Each year, I clear my diary for June and plan my time carefully to ensure that I can mark to a strict timetable, giving the papers my highest level of focus. And every year I become more and more depressed by the standard of the responses and the restrictive nature of the mark schemes.

The most saddening answers are simply left blank, or there could be a crossed out sentence. This may be understandable at GCSE for a short answer, but I have seen full essay questions left blank in A-level exams. I cannot help but picture the student sat in the exam hall, pen in hand and nothing to write. I wonder how they feel; it makes me sad and angry that maybe they’ve not had all the help they deserve from their teacher.

On another occasion a GCSE student covered a whole page in calculations trying to work out a simple percentage change question. They drew a box and arrows pointing to their eventual (wrong) answer, but they must have spent at least 20 minutes on a question that should take no more than two.

Some students miss out on recognition because they lack the simple skill of clear handwriting; we cannot award the marks if we can’t read the answer. If it is illegible, there is no choice but to only credit the parts I can read.

Then there’s the other side of the scale: some essays are magnificent and show understanding of a topic that goes far beyond the requirements of the course. These are beautifully written and include complex analysis worthy of an undergraduate. But many of these responses go uncredited if they do not fit the exacting standard of the mark scheme.

I have seen some students get marks “capped” because they haven’t included a certain phrase or diagram, even though their overall work was of a high standard. This is reflected in the classroom and I have students asking how many sentences of analysis they need, and how many evaluation points. Whatever happened to writing a good essay and answering the question to the best of your ability?

I understand that exams are necessary to be able to award qualifications to students, and that mark schemes can ensure that grades are fair and consistent – perhaps this is something that cannot be changed. But I just cannot stop picturing the students sat in the exam halls, some with nothing to write at all, some writing illegibly, and others writing brilliantly but not being rewarded.

It’s clear something isn’t working if a student is enrolled on a course, but ends up without anything to show for it.

Some students do not engage – perhaps because they are not supported emotionally at home and in school – and cannot cope with the demands of study. Others are simply not on the right course. Even more worryingly, too many students fail to achieve because of poor teaching.

It is staggering how many teaching staff I know that do not read examiners’ reports or even the exam specification and so their class often misses out on marks.

It all boils down to time. We don’t need another initiative or want the system to change again but teachers need support to deliver well-designed courses and give detailed feedback to students. This would happen if class sizes were more manageable, reducing the level of marking we have to get through. It really is that simple.

Education and exams should not be the final stage for young people, but the start of their life. Yet, too many are beginning this journey far behind their peers. Let’s not let students down. We are measured by their results for that one year; but they may be measured by these grades for life.

In teachers they trust

BY

NG JING YNG

Asked how he assesses his teachers, Mr Matti Koivusalo shrugs matter-of-factly that he has “no means” to do so. “There is no evaluation whatsoever for teachers. Everything is based on trust,” says the Principal of Haaga Comprehensive School in Helsinki.

Indeed, the “open” school culture means any feedback quickly reaches his ears, says Mr Koivusalo, who looks after 50 teachers and 600 pupils in grades one through nine (the equivalent of Primary 1 to Secondary 3 in Singapore).

It is easy to see how: Along the school’s hallway, pupils look up from their mobile phones and greet him as he walks past; some engage him in friendly banter. At the school cafeteria where free lunches are served daily — an established practice at all Finnish schools — teachers join him for lunch and chat about how their day has gone.

Said Mr Koivusalo: “If something bad happens, I’ll hear about it in five minutes … The atmosphere is such that (students and teachers) can come and talk about it freely without being afraid.”

Even so, sackings are rare in Finnish schools, say educators. Mr Vesa Valkila, one of the principals at Turku University teacher training school, tried to explain: “Finnish teachers have a lot of freedom and are trusted … that really motivates a lot of them to do their best.”

LEFT TO TEACH

In Finland, a small country of 5.4 million people, its education system operates on this singular principle of trust.

The country’s model shot to global attention after Finnish pupils repeatedly excelled in international tests such as the Programme for International Student Assessment — despite having practically no mandated standardised exams, rankings or competition.

Schools take in students of all varying abilities, including those with learning disabilities, under one roof. The curious result is that, the differences between its weakest and strongest students are the smallest in the world, according to a Organisation for Economic Co-operation and Development (OECD) survey.

School leaders across Finland tell TODAY the same thing: “We trust our teachers”.

There are no national examinations in the first nine years of Finnish formal schooling, and schools and teachers are pretty much left on their own to educate their charges.

As Ms Armi Mikkola, counsellor of education at the Ministry of Education and Culture put it: “The administration is for support and not for inspection … Trust is part of Finnish society, it is a culture.”

Nevertheless, “with trust, there are some risks”, admitted Professor Jouni Valijarvi, Director of the Finnish Institute for Educational Research.

To mitigate risks of having underperforming teachers in schools, a stringent teacher selection process and rigorous teacher training is integral to the system, he said. “It is very important that we can say all schools are good schools,” added Prof Valijarvi. “Because in every school, we’ve highly-trained and qualified teachers”.

SELECTING THE VERY BEST

Yearly, 7,000 teaching aspirants apply to be class teachers (teaching the equivalent of Primary 1 to 6). Typically, there are just 800 spots available.

To teach secondary and upper secondary students (Secondary 1 to Junior College equivalent), 6,000 vie for 1,500 subject teacher positions yearly. Universities cherry-pick from this large pool of applicants, with two different selection processes for each category.

For class teachers, to prepare applicants for an entrance test, authorities will release study materials online on education-related topics such as pedagogical research studies. During the four-hour test, applicants answer about 100 multiple-choice questions. Even so, acing the entrance test does not guarantee a spot in one of the 11 universities offering teacher education.

In phase two, depending on the applicant’s university of choice (they are given up to three picks), there could be a psychometric test along with an interview, or an observed group activity. Some universities also select based on an applicant’s matriculation exam results — the only national examinations taken by Finnish pupils, at the age of 18.

Ms Anna Vaatainen, a student teacher at the University of Turku, is one who succeeded on her second try.

In her first attempt, she was invited by the University of Jyvaskyla for an interview but did not make it through. She went on to obtain a social work degree, and worked in an orphanage for four months, before deciding to give teacher education another go.

This time, after “studying very hard” for the entrance test again, she and three other applicants were tasked by the University of Turku to plan an imaginary school’s sports day. “I am better around people so this group activity might have worked for me,” she said.

Those hoping to be a subject teacher undergo a similar selection process, having to first pass an entrance test set by their subject faculty of choice. They will then apply to the faculty of education, which may require an aptitude test and interview.

The result is that you ensure true commitment to the job. Mr Jari Kouvalainen, a student teacher at the University of Eastern Finland, said: “Because we have to get through this really hard test, you have to be really motivated. With another five years of study, you’re really committed to this career”.

RESEARCH-BASED TRAINING

In the ’70s, Finnish officials moved teacher training under the universities, subsequently implementing a five-year master’s degree programme for all who want to become teachers. A combination of theory, practice and research was key to teacher education, they decided.

Class teachers major in the educational sciences and teach most subjects including Mathematics and Science at the primary levels. Teacher educators say that teaching younger children requires strong pedagogical skills to motivate and excite learners, and not just the transfer of academic knowledge at this stage.

By contrast, subject teachers major in their teaching subjects, while also having to complete pedagogical modules and teaching practicums. In-depth knowledge in their teaching areas is crucial, to give them the confidence to explain complex theories and tackle difficult questions.

Ms Anneli Rautiainen, head of professional development of teachers at the Finnish National Board of Education, thinks that research-based teacher education accounts for the high quality of teaching in Finnish schools today.

“The fact that we have a Master’s degree for teacher initial education is very important. As research-based teachers, they can analyse learning situations and know how to support their students better,” she said.

Student teacher Ms Tuula Hurtig agrees that conducting research has honed her critical thinking abilities and improved her teaching methods. Graduating as a history and civics teacher this year, her thesis involved research into how historical pictures impacted her students’ learning.

GETTING FIELD EXPERIENCE

Head of teacher education at University of Helsinki, Professor Jari Lavonen, calls research-based teacher education vital — it combines with field practice to keep student-teachers in touch with classroom realities and “thinking about their teaching methods”, he said.

All student teachers undergo multiple teaching practicums as part of their five-year programme. Each one lasts between two weeks and a year.

Guided by teacher mentors, student teachers are attached to teacher-training schools set up by the universities, where they plan, teach and observe lessons. These 12 teacher training schools across Finland function as normal schools, with pupils coming from nearby homes. These schools also partner regularly with universities to produce the latest research in education.

Final-year student teacher Mikko Honkamaki, from the University of Jyvaskyla, worked with different mentors during each of his four practicuums — which broadened his perspective on various teaching styles — and got advice before and after each lesson. He also got to observe and critique fellow student-teachers, and vice-versa.

“Watching my peers forced me to focus on my own way of giving instructions ... Receiving and giving feedback has also been crucial to my growth as a professional,” he said.

LEEWAY TO DECIDE

It was a cold winter’s morning when TODAY visited Maininki School in Espoo city, half an hour outside Helsinki, and Ms Rose-Marie Mod-Sandberg was conducting an English Language lesson with her eight-graders (Secondary 2 equivalent).

The classroom was quiet as some students had fallen ill; it was a smaller than usual group. Ms Mod-Sandberg, 55, decided to get her pupils to share about their favourite American cities and imagine what they would do if they got there. As the mood lightened, she gave out worksheets which each student completed on their own.

She has the leeway to tailor her lessons according to her students’ abilities or interests on that very day itself, she told us. For instance, if the children were keen on a topic that was meant only for next year, she could dive into it. And if they seemed more tired than usual — such as after a strenuous Physical Education lesson — she could choose to do something less demanding, and pick things up later.

“If I want to teach a topic, I can teach it anyway and anytime I like,” she said. “Finnish teachers undergo a long training, so (school leaders) can trust us to be professional and to act in the pupils’ interest”.

MORE THAN MONEY’S WORTH

In Finnish schools, teachers typically teach from 8am to 2pm before heading off to plan lessons or attend to parents’ queries. They are not required to take charge of after-school activities such as arts or sports clubs — usually run by private community organisations — and those who do so, are remunerated accordingly.

Schools leaders also said that a layer of stress is removed for teachers as there is no evaluation process linked to their salaries. In fact, the pay structure is relatively flat where pay increases with years of experience and teaching hours.

According to the latest OECD data, Finland’s average annual wage is S$59,852 or approximately S$5,000 a month. For those teaching at the primary level, annual salaries start at S$35,883 (about S$ 3,000 a month). After 20 years, their pay reaches a maximum level of S$64, 530 (S$5,400 a month).

Nevertheless, pay is not a main issue for Finnish teachers, said those TODAY spoke to. People are attracted to the career due to the high status that education is accorded in Finland and the autonomy given to teachers.

The government provides free education in the first nine years of a child’s school life, while schools receive funds to invest in slower learners. Teachers also hold a place in Finnish history, often cited as important figures alongside priests and doctors.

“Young people still see working as a teacher as very creative and independent, where teachers can make a difference in their pupils’ lives,” said Mr Olli Maatta, a teacher trainer at Helsinki Normal Lyceum, a regular Finnish school owned by the University of Helsinki for trainee teachers to serve their attachments.

At Haaga Comprehensive School, the school bell rings and children burst out of their classrooms into the snow-filled courtyard, throwing snowballs at one another and sledding down mini snow hills.

Starring out of his window as one of his teachers leads pupils back from a skiing lesson, principal Mr Koivusalo observes: “The role of an educator is very important. If a teacher loves his job, the children know it and they will want to come to school.”

Ng Jing Yng is a senior reporter with TODAY covering the education beat. She spent one and a half weeks visiting schools in three Finnish cities — Helsinki, Jyvaskyla and Turku — ranging from primary through to upper secondary (JC equivalent) levels. She spoke to students, educators, university faculty who train teachers and officials.

Degree classifications are extremely crude - and pretty useless

When they graduate, students should simply be given a transcript of their marks as a record their study, says Jonathan Wolff

• 
• • Jonathan Wolff
  • guardian.co.uk, Monday 16 July 2012 18.45 BST

Universities themselves do not find the classifications useful. Any student applying for further study will be asked for a transcript of all their marks, in addition to their degree result. Photograph: tomas del amo /Alamy

In Geneva a few weeks ago, as the American and European participants were discussing which insect repellent to use on their post-conference hikes, I had to leave in order to attend yet another meeting of an exam board. This year, I've been chair of several of our boards, "faculty observer" on others, and external examiner elsewhere, and so my desk has been littered with exam scripts and spreadsheets. My head is full of rules for dealing with classification and borderline cases. Degree schemes are like snowflakes: no two are alike.

North Americans rarely understand the expression "exam board" unless they have worked or studied in the UK. Of course, they grade their papers, often with substantial help from their teaching assistants. But once the marks are settled, that is it as far as the department is concerned. Marks go off to the university administration, and in due course find their way on to student transcripts.

Here, by contrast, at least two academics assess or moderate each paper. The mark then exists in a form of limbo until ratified by the exam board, the external examiner and the university examinations section. In some cases, a single essay will be read by three different people, and the mark adjusted twice, although this is rare. Marking in the UK is a process of handicraft, not mass production.

And what do we do with these finally tuned judgments? We put them into a computer that weights them for year of study, ignores some of the bad ones, and produces a number through some form of averaging process. That number will assign the candidate either to a clear degree class, or to a twilight borderline zone. If borderline, we then use another set of rules, apparently too complex for any computer, taking account of such things as "exit velocity", "spread of marks" and any extenuating conditions, in turn graded A, B, C, and X. In such discussions a score of academics can spend a couple of happy hours for each degree programme trying to detect whiffs of high-class performance. Inevitably, and tragically, some students will be consigned to a lower classification by a hair's breadth.

And after all of this, what do we end up with? Given that many students now regard a 2:2 as hugely disappointing, the great majority find a way to do what they need to achieve at least an upper second. Some, with talent and hard work, will do even better and will be awarded a first. Those who in the old days would have performed weakly are likely to have failed at an earlier stage, and so just won't be there in the graduating class. I haven't seen a third in years. Averaging between a 2:2 and a fail is a real challenge. Hence after all this work, we assign perhaps 20% of students to the first-class category and most of the rest to the upper second-class group, with a sprinkling of lower seconds.

In other words, the job of an exam board is to spend a huge amount of effort taking a rich profile of information – how students have done over a wide range of assessments – and turn it into extremely crude classification. And it is classification that we find useless for our own purposes. Any student who applies for further study will be asked for a transcript of all their marks, in addition to their degree result. Universities apparently don't think the degree classification conveys very much useful information, and so why should anyone else?

I'm coming to the conclusion that we should simply issue students with transcripts to record their study, and leave it at that. There are proposals to replace degree classifications with grade point averages, as in the US. That's a move in the right direction, but why have a summary measure at all? School achievement isn't summarised into a single number, and why should it be any different at university? If a student on a German and geography degree did brilliantly in German and miserably in geography what purpose is served by reducing it all to a single score? And so my plea: No more classifications. No more algorithms. No more borderlines. And, most heartfelt of all, no more exam boards.

• Jonathan Wolff is professor of philosophy at University College London

Teachers will work the system as long as they are under pressure for results

There's no room for error now schools are businesses. We need to hire more teachers and give them space to try new ideas

• 
• • Phillip Easton
  • guardian.co.uk, Tuesday 3 April 2012 18.29 BST 
  • 
    

Teachers have been 'gaming' the system to get their pupils through exams. Photograph: Rui Vieira/PA

According to a poll by the Association of Teachers and Lecturers, 35% of teachers say they could be "tempted to cheat". I've been a teacher for 10 years, and those figures hardly surprise me. I've seen everything from teachers openly rubbishing other subjects to ensure revision session attendance, teachers advising pupils to retake a whole year to improve performance in one module (thus ensuring their results don't drop), all the way to teachers encouraging pupils to annotate texts in the form of a shorthand lesson plan. I've seen teachers using exam spec answers and teachers becoming examiners for a year to steal good practice: in other words, we all work the system. What really worries me, however, is that there is no grey area left untapped and staff will feel pressured to go that extra inch into full-blown plagiarism.

Of course, gaming the system to relieve pressure from senior management is as old as the hills. But to get your head around around the whole truth of the ugly situation means unpicking a much larger problem with education in the UK.

Let me burst some bubbles, to start with: schools are businesses. All employees at the schools I have worked at are held accountable for their results. Residuals (how well your students have improved on predicted grades) A*-C pass rates, Alps scores, t-Scores, value-added – all are unpicked by a good management team, in a bid to improve the business. Better pass rates equals a more attractive school, and therefore more students. More students means more money. We are businesses.

The switch to academy status is to most parents confusing and pointless – to staff it means that now, you are officially working for a corporation. These are not your fuzzy, friendly government-run schools, with endless patience for slack teachers. The potential here is to swiftly get rid of staff who don't make the grade. "Good!" you might think. "Lazy teachers! They have it easy anyway, let's cut away the dead wood!" And, perhaps, there is an argument for that.

However, when a teacher sits down and analyses their results, they are set targets. The targets are "aspirational", but still meant to be achievable. Even when your pass rates are 90% and over, or your Alps results scores are a 2 (1 being the best possible, 9 being the worst) targets are put in place. And here is where the problems can begin. It is very easy (and I have known this to be the case) that a teacher's worth is questioned in line with results. Lazy pupils? That'll be your fault for allowing that culture in your classroom. Lack of homework or revision? Why didn't you call parents in to make them understand the importance of the revision sessions after class?

The result is a Mobius strip of a career, where you can feel constantly that you're running to stand still. I've grown pretty resilient to it, but I can empathise with the teachers who haven't. By and large, we all do our best. If you put in the hours, your teaching is focused, you have a keen bunch of kids and you lay on the revision sessions, the outcomes should be good. But when they aren't, there is no room for error. The school up the road had a better year. Raise your game – Bogwood primary sent twice as many kids there this year, and we need bums on seats. And if your results are good, well open a paper and listen to everyone tell you it's because the exams got easier, and it was harder in their day. It might well have been, but it doesn't help the hard work we are doing right now.

Our current school model is not fit for purpose. Schools are hamstrung by a lack of funds to develop teaching practice, the space to develop new ideas, and the confidence to try them out. We need to be attracting innovators with visions for the future, starting with training staff and ending with a flexible, skills-based curriculum that evolves every couple of years. Teachers need time. Look at the current dropout rates of new teachers: over a third leave the job in under three years. Why? Pressure. Hiring more teachers would create jobs and allow us to teach smaller classes, and could create more non-contact time in which to develop the craft. Yet this idea is often ridiculed. Since when did we all get so blasé about the future of our youth?

AQA - Exam board to penalise private school pupils

By Richard Garner, Education Editor in The Independent
Tuesday, 27 September 2011

A controversial plan to rank all A-level students according to the schools they attend – which would allow universities to discriminate against pupils from private schools – is unveiled today by Britain's biggest exam board.

The radical proposal would allow universities to offer places to students from disadvantaged homes who showed potential but had performed less well in exams than their peers at better schools.

The plan by the exam board AQA (Assessment and Qualifications Alliance) provoked a storm of argument among academics and independent schools. There were immediate fears that candidates will be penalised simply because they achieve good A-level results at a good school. Independent schools are also alarmed that the approach could discriminate against disadvantaged pupils to whom they have offered scholarships.
Dr Tim Hands, headmaster of Magdalen College, Oxford, and co-chairman of the Independent Schools' Universities Committee, said: "It is extraordinary. It takes no account of home background or the amount of tutoring a pupil could have."

Professor Alan Smithers, head of the Centre for Education and Employment Studies at the University of Buckingham, added: "There must be concerns about the ranking the candidates are awarded. The possibility for errors is enormous." The plan is contained in a paper prepared for discussion by Dr Neil Stringer, senior research associate at the AQA Centre for Education Research and Policy, and being circulated at the party conferences for debate this month.

It advocates the drawing up of a national system for ranking both candidates’ achievements and the educational context in which they were taught.

Pupils at weak schools would get bonus points; those at elite schools could be penalised in comparison.
Dr Stringer cites the example of St George’s Medical School in London in support of his argument. It offers places to students with lower A-level grades (BBC rather than AAB) providing that their performance is 60 per cent better than the average for their school.

“St George’s reports that students from poorly performing schools who are accepted into medical school with lower grades do just as well as their peers with higher grades,” he adds.

“This strongly suggests that students admitted through the adjusted criteria scheme learned enough at A-level and are able enough learners to compete successfully with students who achieved higher A-level grades under more favourable.”

Under the blueprint he has devised, students would be awarded an exam score based on their best three A-level grades and then placed into different performance bands. They would then be given the ranking for their school.

Dr Stringer says the system could either be offered to universities individually – or drawn up centrally by an existing agency like Ucas, which is currently reviewing its A-level system.

The AQA believes it can be an an alternative to allowing students to apply to university after they have got their results – rather than be awarded places on predicted grades. This plan, under active consideration from ministers and said by some to be fairer towards disadvantaged students, has failed so far to get off the ground largely because of opposition from universities.

Professor Smithers added: “I would hope that any university worth its salt would look at the candidates’ achievement and inform their own view as to their potential.”

Dr Hands added: “Cambridge University, which features at the top of many a global league table, has recently published research that shows prior schooling is of insignificant effect with regard to degree outcome.

“The proposer of this scheme might like to bear this in mind.”

Brian Lightman, general secretary of the Association of School and College Leaders, which represents the majority of state secondary school heads, described it as “a step too far”. He said it should not detract from the need to provide all pupils with a good education in a good school.

Lee Elliott Major, of the education charity the Sutton Trust, which campaigns to get more disadvantaged young people into leading universities, said: “We support the use of so called contextual information when judging students’ potential and achievement.”

However, he added that the “bigger challenges” were in getting “more children with the grades at school to make university a realistic prospect and encouraging pupils to actually apply when they have the grades”.
Dr Stringer stresses in his paper: “The proposed system would not encourage or require universities to relinquish control of their admissions systems. It is not an issue of allocating students to universities on the basis of their respective rankings: admissions tutors would be free to make decisions.”

The AQA said the scheme could be considered as an alternative to Post Qualification Application – allowing students to apply to university after getting their results.

That, argued Andrew Hall, chief executive of AQA, would lead to a shorter teaching year if exams were brought forward.

“We have real concerns about the effect this could have on the performance of some students,” he added.
“So our Centre for Education Research and Policy have devised a different way to tackle the issue that doesn’t disadvantage any student and allows all applicants – from whatever their school type or background – to compete fairly for university places.”

* Meanwhile, plans to mark GCSE students on their spelling and punctuation and scrap most resits were published by Ofqual, the exam standards watchdog, yesterday.

It has launched a consultation on the proposals which would see teenagers – from 2012 – having to sit all their exams in the summer at the end of two-year courses rather than sit modules throughout the course.

They would also only be allowed to resit English and maths.

The reforms were first announced by Education secretary Michael Gove earlier this summer.

How the new system would work

Under the new system, a pupil at a weak school who got a lower grade than a rival pupil at a good school could still be given more university entrance points, writes Richard Garner.

The blueprint would work like this. James goes to a low-performing comprehensive in a disadvantaged area. He manages to get an exam score of 36 out of 40. However, he is entitled to bonus points as a result of his school's low ranking (it scores minus three in the rankings).

Adam, on the other hand, goes to a top performing independent school with no pupils on free school meals and got 38 for his exams. But he faces being penalised on his school's ranking (the school is given a "plus three" ranking).

It would, of course, be up to the individual university to decide what to do with this information but one way of using it will be to add three points to James's exam score because of the background he comes from and deduct three points from Adam. On that basis, the place would go to James.

The argument in the paper is that there are still vastly more points awarded for exam performance than education context and it is unlikely that any university would be as crude as to deduct the maximum ranking points from Adam and give the maximum three extra to James.

However, what is likely is that both Adam and James would be longlisted - something that would not have happened to James without the ranking system. Then James's potential would outweigh Adam's performance.