Decision hygiene is to product and service selection as testing is to software development. Skip them at your peril.
Decisions about technology can be noisy affairs.
(Please take a moment to relive one you were part of.)
As Daniel Kahneman, Olivier Sibony, and Cass Sunstein masterfully point out in Noise (2021), noise is different than bias. It’s an independent contributor to infelicitous results in professional (and other) judgments. Noise tends to be an invisible enemy. Bias, more obvious, moves decisions in particular directions; noise just adds errors through unwanted variability. Responsible decision-makers seek to minimize both.
The Noise authors provide vivid examples of judgments in which noise plays a role, including some in law, like judicial sentencing decisions, which have been shown to turn on such things as the outside temperature or whether the local city’s football team won its most recent game. Contexts like insurance underwriting can operate like lotteries. As the authors say, “wherever there is judgment, there is noise” (p 12). We are all noisy.
While perhaps inevitable, not all variability is unwanted. Noise can be your friend. It can reflect, or spur, creativity. Evolution depends on it. Disagreements make markets. Yet there are downsides.
Noise and its discontents
There are two main kinds of judgments in which noise can be problematic: (1) when we’re predicting things (what will happen) and (2) when we’re evaluating things (what’s good or not so good.) We’re faced with objective ignorance and intractable uncertainty about the future when predicting; we’re faced with the subjective reality of differing tastes and opinions when evaluating. For the first kind, the passage of time will eventually reveal who was right; for the second kind, there’s no ‘right’ answer, ever. Humility is called for in both scenarios.
Many things contribute to noise. Compilations of cognitive pitfalls illustrate how imperfectly we think. (The Visual Capitalist compilation is particularly good.) We overvalue our intuition. We succumb to confirmation bias and many other cognitive and affective errors. Good moods, ironically, can increase our bullshit receptivity. Stress, fatigue, weather, and a host of other factors impede our ability to make low-noise decisions. We get caught up in informational cascades, buying into views people express enthusiastically because we assume they have evidence for them and don’t want to appear uncongenial. Searches for truth often stop when a good narrative is found. We jump to conclusions.
Things get even more complicated in group deliberations, which tend to increase noise, spurring polarization and extreme views. We can be led astray by the Illusion of Agreement. Apparent closure can be a misleading signal. That internal sense of coherence can vex us even in solo decisions.
My recent Legal Evolution piece on the social life of tech decisions, see Post 309, considered how the clash of agendas, experiences, and preferences haunts group decisions. Such efforts can also drift in random ways, like the Brownian motion that microscopes reveal in drops of water. Satisfaction with an organization’s decision-making processes tends to be in direct proportion to how much individuals agree with the decisions being made.
In tech decisions and elsewhere noiselessness is an impossible dream. Nonetheless, there are things we can do to reduce unwanted noise.
Noise reviews several techniques. Nudges can help push people in desired directions, as can other forms of “choice architecture.” Noise audits can be used to measure variability statistically. “Dialectical bootstrapping” involves making two judgments, while answering critical questions in between. The Mediating Assessments Protocol (MAP) calls upon us “to put off gut-based decision-making until a choice can be informed by a number of separate factors.”
The authors again observe that “many decision makers will reject decision-making approaches that deprive them of their ability to exercise their intuition” (p 146). But “a procedure that compels explicitly comparative judgments is likely to reduce noise” (p 186)
Looking at legal tech
How does the above apply to legal technology decisions?
Legal tech is rife with choices. Which product to ‘hire’ for Job X. Whether to migrate to a current vendor’s next version or to take the opportunity to switch. Whose interests to prioritize? How to configure and deploy a chosen solution. (Configuration can present a maze of choices, with complex dependencies and cross-implications.)
Like most judgments, such choices involve both predictive and evaluative aspects. Lawyers and techies constantly face both. Stuff happens, but you don’t know quite what until it does. Your favorite vendor could go belly up; the market for one of your firm’s most highly profitable services could evaporate; a key staff member may win the lottery or get hit by a meteorite. The scrappy startup whose product you love could get acquired by a duopolist and ‘integrated’ into a solution you hate.
Principles and practices of noise minimization transcend product and service categories. They are worth observing whether you’re a law firm choosing an e-fax solution or a prosecutor deciding which charges to lodge against a former president. Decision hygiene is to product and service selection as testing is to software development. As beneficial as both can be, they feel like chores. Ones that practitioners skip at their peril.
Frequent decisions can usefully be analyzed statistically. But tech decisions are typically singular, or at best episodic, rather than repeating. That’s the case for most legal tech decisions in most law offices. So ‘noise audits’ are not practicable.
But in many ways, legal tech choices are similar to other kinds. Inevitably there are tradeoffs—some quickly and easily settled; others not so easily. Product power and longevity vs. UX (user experience) quality or database integration. Vendor depth of bench vs. individual team member brilliance. And occasionally there are true ‘requirements’ that can trump other considerations and dispose of a choice. The absence of just one key feature can be a deal breaker; conversely, some attributes of a solution can clinch the deal. (Like its provider having been founded by the niece of a senior partner.)
Up and down
I crafted a dozen ‘mantras’ for legal tech selection processes in this Technolawyer feature. Here’s a supplemental up/down list:
- Look up. Step back. Be mindful of the big picture. Know your circumstances, where you are, where you’re going, where you want to be.
- Break it down. Decompose your decision into subjudgments. Compare your options consideration by consideration.
- Slow it down. Delay your holistic judgment. Avoid premature conclusions.
- Check up. Have you identified all of the relevant options and considerations?
- Listen up. Seek to understand before you seek to be understood. Now may not be the greatest time to express your individuality.
- Act up. Promote independent thinking. Welcome eccentric voices. There can often be wisdom in crowds, when e.g. the average of many guesses turns out to be eerily close to the true number of jellybeans in a large jar. But only if folks guess independently and avoid informational cascades.
- Staff up. Enlist professional help. Just being responsible for, or impacted by, a decision does not qualify you to manage it! Veteran project managers generally do this job well. If you can find someone who both knows the ‘category’ and is an experienced decision adviser, so much the better.
- Tool up. Consider whether you’re making good use of tools. Think beyond Word, Excel, and email. Collaborative environments like Teams and Slack, even Google Drive, can serve well. And as some readers know, I’m especially fond of a system/method called choiceboxing, which coincidentally facilitates decomposition and collaborative deliberation.
Of course, you can go overboard. A decision may be low stakes, or obvious. No one is served by decision hygiene theater. Sometimes the choice is ‘wired,’ or claimed as the prerogative of one individual, and the outcome is a foregone conclusion. In any event, execution is usually more critical than selection. As venture capitalist John Doerr said, “Ideas are easy. Execution is everything.” Choices are somewhere in between.
Noise is hardly a quick read. Locate a quiet place to enjoy it. And when you next find yourself in a tech selection trench, as my dad used to say when I cranked the stereo too loud — “Keep it down!”
On the other hand, if you’re already achieving a decent signal-to-noise ratio in your tech decisions – Keep it up!