I had another one of those connecting insights moments while on my daily walk, listening to a podcast. I had been in a conversation earlier in the week with a long-time client/colleague who was engaged in conversations within his district around a more traditional grading/reporting system than the proficiency-based model that the system has been building. On the walk, I was listening to a Hidden Brain podcast titled “Our Noisy Minds.”
The podcast was an interview with Daniel Kahneman, one of the authors of Noise: A Flaw in Human Judgment. Ten years earlier I had written a blog connecting to Kahneman’s earlier book Thinking Fast and Slow.
Stunning for me was a study in that book involving a group of parole judges in Israel. These judges spend the entire day reviewing cases quickly (about one every 6 minutes) and deciding to deny or approve a person for parole. Tracking these decisions made with the times of breaks and meals, the researchers uncovered that the largest approvals are granted following food breaks and meals and the approval percentage drops to almost zero just before the next break.
“The best possible account of the data provides bad news; tired and hungry judges tend to fall back on the easier default position of denying requests for parole. Both fatigue and hunger probably play a role.” (Kahneman, 44) While reading the above I wrote, “Wow!”, in the column of the text. Thoughts about school schedules and policies ran through my mind.
In Noise, the authors write that noise arises in decision-making because of factors such as cognitive biases, mood, group dynamics, and emotional reactions. On the podcast, Kahneman illustrates the impact of noise with an example from insurance underwriting. Managers predicted that underwriters provided the same criteria to create a quote for coverage, might differ in their proposals by 10%. They found that the difference was 55%. Another example from the podcast shared a study that found that college admissions officers pay more attention to the academic attributes of candidates on cloudy days, and the non-academic attributes when the weather is sunny. An example from the justice system: if you’re a defendant you hope for good weather because, on very hot days, judges assign more severe sentences. Although judges have air conditioning, it’s the outside temperature that nevertheless seems to have an effect. It’s also been established in at least one study that for judges who are keen on football, their team’s result on Sunday or Saturday will affect the judgments they make on a Monday. They will be more severe if their team lost.
I’m guessing that you can see why as I walked along, listing to the interview with Kahneman, the thoughts about what do grades, and report cards, actually communicate to anyone. What are all the noise factors impacting teachers in these decisions? How do we deal with people generally putting way too much value on what a grade communicates?
Interestingly, it was also ten years earlier that I posted a blog around information that I gained from an article and presentation by Tom Guskey, who suggested that grading was an area where our knowledge and practice were perhaps at the greatest gap in all of education. He shared a study where 200 secondary teachers were sent a sample of student writing and asked to grade it using their criteria. The results were 15% A, 15% F, and everything in between. Some suggested that the problem occurred because writing was a subjective product, so the study was repeated using a math paper. The results were an even greater disparity in grades. I was surprised to then learn that the study he shared was conducted in 1912. We have known about the problem for a long time.
The podcast with Kahneman provided some insights for those looking to decrease discrepancies. “If you have a bunch of employees who are performing an interchangeable function, such as different physicians in the E.R. or different federal judges or different underwriters in an insurance company. you can do a noise audit. In a noise audit, people are presented with a problem that is realistic, the kind of problem that they could encounter on their job. A set of those interchangeable employees are all presented with the same question and are asked a very precise question—to put a dollar number or in some other way indicate what they expect to happen in that case. Then you just look at the variability of the case. You don’t have to know the correct answer, because what interests you are the variability of judgments. If the judgments are variable, then the errors are variable.”
This sounds to me like a definite PLC activity. Just the discovery of the discrepancies would raise a consciousness as to what “noise” might be impacting individual teacher decisions.
In another interview with Kahneman, he cautioned that a common move that organizations implement to decrease the discrepancies are fixed criteria such as a pricing guide for premiums given to underwriters. Problem: underwriters are very likely to completely ignore those and to follow their judgment. Suggestion: “Do not impose procedures on them but work with them to make them more uniform because actually, they will recognize that they would like to be in agreement with each other. But letting them feel that what they’re doing is what they want to do rather than what they’re being forced to do, is clearly a very important step if people really want to have organizations that improve their judgments.”
PLCs having discovered discrepancies among the “grading” of student work, can then work as a team to develop plans to align the meaning and purpose of grades and how to best communicate that meaning to all stakeholders.
Administrators working as a team addressing teacher evaluations can consider a very similar application. How might the noise from a parent or student disciplinary meeting this morning influence a teacher evaluation this afternoon?
