⭐ Toby's Rating: 6/10 - Recommended For: Decision Makers
3 Big Ideas 💡
Noise is an ignored problem in organisations that leads to bad judgements. Bias gets all the limelight but Noise also creates many problems. To understand error in judgment, we must understand both bias and noise.
Noise is can be wanted and unwanted. The amount of unwanted noise is greater than most organisations realise. This noise has a great cost financially and socially. An example of noise is in Criminal Punishments. We expect the same punishment for the same crime. History shows that is not always the case.
Noise can be reduced through performing Noise Audits and applying the six principles of decision hygiene.
2 Best Quotes 💬
The best quotes from Noise by Daniel Kahneman, Olivier Sibony, Cass Sunstein
Some judgments are biased; they are systematically off target. Other judgments are noisy, as people who are expected to agree, end up at very different points around the target.
Wherever there is judgment, there is noise—and more of it than you think.
1 Top Takeaway 🍔
Prior to reading the Noise book, I hadn't even considered what Noise is and its impact. I've now become much more aware of system noise and how it can lead to negative impacts. The stories shared throughout are striking. For example, when judges are passing down sentences on days following a loss by the local city’s football team, they tend to be tougher than on days following a win. If employers rely on only one job interview to pick a candidate from among a similarly qualified group, the chances that this candidate will indeed perform better than the others are about 56% to 61%. You would be as successful judging the best candidate by picking a name from a hat.
The authors share several tips on how to manage and reduce noise. A primary way to do that is using a Noise Audit. People are presented with a problem that is realistic, the kind of problem that they could encounter on their job. Employees are all presented with the same question and are asked a very precise question—to put a dollar number or in some other way indicate what they expect to happen in that case. Then you just look at the variability of the case. You don’t have to know the correct answer, because what interests you are the variability of judgments. If the judgements are variable, then the errors are variable.
The big takeaway for leaders and organisations:
Be aware of Noise. Start with a Noise Audit and then take steps to reduce the amount of unwanted Noise.
Big Ideas Expanded 💡
Noise Book Summary:
What is Noise?
A team of target shooters whose shots always fall to the right of the bull’s-eye is exhibiting a bias, as is a judge who always sentences Black people more harshly. As these errors are consistent they can more easily be identified and corrected. Another team whose shots are scattered in different directions away from the target is shooting noisily, and that’s harder to correct. A third team whose shots all go to the left of the bull’s-eye but are scattered high and low is both biased and noisy. Noise is often more challenging to address than Bias.
Noise is often like a leak in the basement. It's tolerated not because it was thought acceptable but because it remains unnoticed. Without monitoring Noise, many organisations can be fooled into an illusion of agreement while in fact disagreeing in their judgements. We can live comfortably with colleagues without ever noticing that they actually do not see the world as we do.
In professional judgments of all kinds, whenever accuracy is the goal, bias and noise play the same role in the calculation of overall error. In some cases, the larger contributor will be bias; in other cases, it will be noise (and these cases are more common than one might expect).
System noise creates inconsistency, and inconsistency damages the credibility of the system. For example, inconsistency in candidates promoted in the workplace creates a lack of trust in the system.
A defining feature of system noise is that it is unwanted, and we should stress right here that variability in judgments is not always unwanted.
A general property of noise is that you can recognize and measure it while knowing nothing about the target or bias.
Why is Noise a problem?
In certain situations, we expect consistency in judgement. For example, in criminal punishment, we expect similar sentences for similar crimes. If a group of judges gives vastly different sentences to defendants who committed the same crime—some judges give a one-month sentence, others one-year, others seven years, and others somewhere in between—then one could call the system noisy. We’d expect similar punishments for the same crime. In a biased system, judges might consistently give sentences that are too high for certain types of crimes. Systems can be both biased and noisy.
Personnel decisions are noisy. Interviewers of job candidates make widely different assessments of the same people. Performance ratings of the same employees are also highly variable and depend more on the person doing the assessment than on the performance being assessed.
Most organizations prefer consensus and harmony over dissent and conflict. The procedures in place often seem expressly designed to minimize the frequency of exposure to actual disagreements and, when such disagreements happen, to explain them away.
Whether you make a decision only once or a hundred times, your goal should be to make it in a way that reduces both bias and noise. Practices that reduce error should be just as effective in your one-of-a-kind decisions as in your repeated ones.
What is judgement?
The very concept of judgment involves a reluctant acknowledgement that you can never be certain that a judgment is right.
A matter of judgment is one with some uncertainty about the answer and where we allow for the possibility that reasonable and competent people might disagree.
Judgment can be described as a measurement in which the instrument is a human mind. A judgment is a conclusion that can be summarized in a word or phrase.
Indeed, the word judgment is used mainly where people believe they should agree. Matters of judgment differ from matters of opinion or taste, in which unresolved differences are entirely acceptable.
Matters of judgment, including professional judgments, occupy a space between questions of fact or computation on the one hand and matters of taste or opinion on the other. They are defined by the expectation of bounded disagreement.
Judgement Scenario:
This scenario is shared in Noise by Daniel Kahneman, Olivier Sibony, Cass Sunstein. It highlights the challenge of Noise in judgements.
Imagine that you are a member of a team charged with evaluating candidates for the position of chief executive in a moderately successful regional financial firm that faces increasing competition. You are asked to assess the probability that the following candidate will be successful after two years on the job. Successful is defined simply as the candidate’s having kept the CEO job at the end of the two years. Express the probability on a scale from 0 (impossible) to 100 (certain).
Michael Gambardi is thirty-seven years old. He has held several positions since he graduated from Harvard Business School twelve years ago. Early on, he was a founder and an investor in two start-ups that failed without attracting much financial support. He then joined a large insurance company and quickly rose to the position of regional chief operating officer for Europe. In that post, he initiated and managed an important improvement in the timely resolution of claims. He was described by colleagues and subordinates as effective but also as domineering and abrasive, and there was a significant turnover of executives during his tenure. Colleagues and subordinates also attest to his integrity and willingness to take responsibility for failures. For the last two years, he has served as CEO of a medium-sized financial company that was initially at risk of failing. He stabilized the company, where he is considered successful though difficult to work with. He has indicated an interest in moving on. Human resources specialists who interviewed him a few years ago gave him superior grades for creativity and energy but also described him as arrogant and sometimes tyrannical.
Recall that Michael is a candidate for a CEO position in a regional financial firm that is moderately successful and that faces increasing competition.
What is the probability that Michael, if hired, will still be in his job after two years? Please decide on a specific number in the range of 0 to 100 before reading on.
If you engaged in the task seriously, you probably found it difficult. There is a mass of information, much of it seemingly inconsistent. You had to struggle to form the coherent impression that you needed to produce a judgment. In constructing that impression, you focused on some details that appeared important and you very likely ignored others. If asked to explain your choice of a number, you would mention a few salient facts but not enough of them for a full accounting of your judgment.
The thought process you went through illustrates several features of the mental operation we call judgment: Of all the cues provided by the description (which are only a subset of what you might need to know), you attended to some more than others without being fully aware of the choices you made. Did you notice that Gambardi is an Italian name? Do you remember the school he attended?
This scenario demonstrates that humans have an internal drive for judgment completion. We aim for the achievement of a coherent solution.
How Do Groups Amplify Noise?
Noise in individual judgment is bad enough. But group decision making adds another layer to the problem. Groups can go in all sorts of directions, depending in part on factors that should be irrelevant. Who speaks first, who speaks last, who speaks with confidence, who is wearing black, who is seated next to whom, who smiles or frowns or gestures at the right moment—all these factors, and many more, affect outcomes.
Case Study: Noise in the Music
For evidence, we begin in what might seem to be an unlikely place: a large-scale study of music downloads by Matthew Salganik and his coauthors. As the study was designed, the experimenters created a control group of thousands of people (visitors to a moderately popular website). Members of the control group could hear and download one or more of seventy-two songs by new bands.
The songs were vividly named: “Trapped in an Orange Peel,” “Gnaw,” “Eye Patch,” “Baseball Warlock v1,” and “Pink Aggression.” (Some of the titles sound directly related to our concerns here: “Best Mistakes,” “I Am Error,” “The Belief Above the Answer,” “Life’s Mystery,” “Wish Me Luck,” and “Out of the Woods.”)
In the control group, the participants were told nothing about what anyone else had said or done. They were left to make their own independent judgments about which songs they liked and wished to download. But Salganik and his colleagues also created eight other groups, to which thousands of other website visitors were randomly assigned. For members of those groups, everything was the same, with just one exception: people could see how many people in their particular group had previously downloaded every individual song. For example, if “Best Mistakes” was immensely popular in one group, its members would see that, and so too if no one was downloading it. Because the various groups did not differ along any important dimension, the study was essentially running history eight times.
You might well predict that in the end, the good songs would always rise to the top and the bad ones would always sink to the bottom. If so, the various groups would end up with identical or at least similar rankings. Across groups, there would be no noise. And indeed, that was the precise question that Salganik and his coauthors meant to explore. They were testing for a particular driver of noise: social influence.
The key finding was that group rankings were wildly disparate: across different groups, there was a great deal of noise. In one group, “Best Mistakes” could be a spectacular success, while “I Am Error” could flop. In another group, “I Am Error” could do exceedingly well, and “Best Mistakes” could be a disaster. If a song benefited from early popularity, it could do really well. If it did not get that benefit, the outcome could be very different. To be sure, the very worst songs (as established by the control group) never ended up at the very top, and the very best songs never ended up at the very bottom. But otherwise, almost anything could happen. As the authors emphasize, “The level of success in the social influence condition was more unpredictable than in the independent condition.” In short, social influences create significant noise across groups. And if you think about it, you can see that individual groups were noisy, too, in the sense that their judgment in favor of one song, or against it, could easily have been different.
How to overcome Noise?
There are six principles for organisations or individuals to take on if they want to minimise noise.
Accept that decisions are about accuracy, not individual expression.
Think statistically, and take an outside view of the problem.
Structure judgement into independent tasks - this prevents the problem of excessive coherence, where people distort information that doesn’t fit into an emerging story.
Decision-makers should resist premature intuitions
Take independent judgements from multiple judges and factor them in.
Favour relative judgements, which tend to be less noisy.