Someday we’ll wonder why it took so long to use data to solve our diversity problem.
As a social scientist focused on empirical data, I believe the following three statements are true. I’d be curious how many readers agree or disagree with my assessment:
- There is a sincere desire among corporate legal departments to use their purchasing power to increase diversity in the legal profession.
- Diverse teams tend to outperform homogeneous teams.
- Controlling for a wide range of factors, law firms with a higher percentage of diverse attorneys are significantly more profitable (per Figure 1 above, +$180K per partner).
There is much to like about items 1 to 3, which raises the question: Why are efforts to use data to increase legal diversity so often missing in action?
After working on these issues for several years, it is clear to me that the root of the problem is resource-based. What does that mean? In the year 2018, most legal professionals lack the time and training to clean, organize, analyze and simplify data. As a result, lawyers are very rarely presented with valid and reliable data that can be understood in a time-efficient way. In the case of diversity, however, the return of investment – both socially and economically – is staggering.
In this post, I’ll dive deeper into the three statements above, sharing work I’ve done on diversity in my capacity as Managing Director of LawyerMetrix, some of it pro bono for the Leadership Council on Legal Diversity (LCLD), and some of it for paying clients. I’ll also provide some suggestions on how law firms and clients can more reliably achieve greater diversity and inclusion.
1. Clients want diversity: overcoming the “file drawer” problem
Legal departments are using their purchasing power to demand greater diversity from their outside counsel. Although many view data as a central part of this effort, an influential annual survey of legal operations professionals recently reported that, between 2008 and 2018, the percentage of legal departments tracking outside counsel diversity was unchanged (20%). See Blickstein, 10th Annual Law Department Operations Survey (2018). The percentage who said they do not track outside counsel diversity, moreover, increased by 24% (from 40% to 64%).
Yet, as someone who has gone into the trenches to make sense of legal departments’ diversity data, the low reliance on data is less a function of interest than a lack of resources. Specifically, inside a legal department, the responsibility for collecting and analyzing diversity data falls to busy professionals and lawyers, not trained analysts. Invariably, this sets off a familiar sequence: (1) Collect data on outside counsel diversity, (2) review it in a haphazard way, (3) conclude it has limited value, and (4) relegate it to the file drawer. The following year, repeat the sequence.
I was first made aware of this “file drawer” problem four years ago when Scott Westfahl, the Director of Executive Education at Harvard Law School, put me in touch with Robert Grey, the President of LCLD. LCLD was working on a solution to the file drawer problem, and Scott thought that the organization might benefit from an analyst’s perspective. (As the Director of Professional Development at Goodwin, Scott had experienced firsthand that a client could use a diversity “scorecard” to communicate with and motivate firms to increase the diversity of their legal teams.)
Since the year 2014, I have worked with Robert, his team at LCLD, and LCLD’s corporate and law firm members to develop and pilot data-driven solutions. Robert is an expert at understanding both the needs of LCLD’s members and the likelihood that a potential offering or action will resonate with them. According to Robert, this means output that is visually appealing and easy to use. Cf. Post 008 (discussing five key factors in Rogers rate of adoption model, including cultural compatibility and simplicity).
In short, this is less a problem of data than a problem of design. Further, when it comes to lawyers and data, it is not obvious what is going to be perceived as culturally compatible. Lawyers like evidence, but like a lot of people, many find numbers, especially statistical abstractions, to be a turn-off. Through trial and error, we refined a data-driven approach that synthesizes and reports data objectively. In addition to leveraging LCLD’s brainpower and connections to get there, cf. Post 063 (Jae Um emphasizing the importance of innovators having access to users), we were fortunate to have the backing of LawyerMetrix’s affiliate organization, AccessLex Institute, which underwrote our pro bono work because of its congruent mission – to increase diversity in the legal profession.
Today I manage a team that deploys data-driven approaches suited to lawyer culture. We use data visualizations whose meaning is easy to see, and whose importance hits readers between the eyes. For example, Figure 2 gives a sample visual for a simple diversity scorecard. It shows how outside counsel law firms vary on a specific diversity metric: the share of fees billed by attorneys of diverse race/ethnicity. Legal departments already have the data to produce these charts. What they lack is the 100+ hours it takes to get from raw materials to refined data visualizations.
At an LCLD event for diversity professionals in the spring of 2017, I met Sneha Desai, who is now Deputy General Counsel, Litigation at BASF. This chance meeting brought about important developments in how I think about communicating diversity data to stakeholders. Along with BASF’s Global Head of Legal, Matt Lepore, Sneha was candid about BASF’s resource problem. After collecting diversity data using the ABA’s Model Diversity Survey, BASF spent several months struggling to organize it. In their words, “We did not have the internal capacity to create the algorithms needed to analyze a large volume of data quickly and efficiently or to produce the findings in a user-friendly format.”
[click on image to enlarge]
On its cultural compatibility – a key driver of innovation adoption – the Snapshot’s most important feature is that it’s concise. Each firm receives an individualized, two-page report that lists its “Legal Diversity Factor Score” and its relative positions on 12 diversity metrics. In this way, the Snapshot gives firms an opportunity to excel across a range of dimensions, thus harnessing lawyers’ competitive instincts. While at lunch with a firm Chairman who counts BASF as a client, I asked for his honest reaction to the Snapshot. He said it first showed him the areas his firm needed to do better, then gave him comfort in seeing the areas where it excelled. He liked the objective quality and interpretability, and summed up by saying, “The Snapshot? I liked it.”
In summary, to drive adoption of data-driven solutions, aligning purpose and data with requisite resources is key. For both clients and law firms, however, the business case for diversity is equally important.
2. Diverse teams deliver superior results
There is growing appreciation that diverse teams make different, indeed better, business decisions than non-diverse teams. See, e.g., Rock & Grant, “Why Diverse Teams are Smarter,” Harv. Bus. Rev., Nov. 4, 2016; Scott Page, The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies (2007). Diverse teams draw on more and varied perspectives when solving organizational and business challenges. In an example from the legal profession, researchers at Acritas found that mixed gender teams outperformed single gender teams on 12 key performance indicators. See “Gender Diverse Legal Teams Outperform Single Gender Teams,” Acritas, Dec. 15, 2017. This result occurred even though there were no significant differences between teams led by male versus female partners.
However, the most rigorous treatment of this topic was completed several years ago by the researcher Randy Kiser. See Randall Kiser, Beyond Right and Wrong: The Power of Effective Decision Making for Attorneys and Clients (2010). Kiser’s massive and ambitious study involved a systematic comparison of final settlement offers versus verdict and damage awards at trial over a 40+ year period in both New York and California. As part of this study, Kiser asked whether gender composition could predict win rates, decision error rates, and settlement positions. The evidence was striking. For defendants, and among other things, Kiser found that female-male trial teams won 7% more of their cases than all-male teams, had error rates that were 9% lower, and under-priced their settlement offers far less frequently – amounting to an average savings of roughly $2.6 million. For plaintiffs, the benefits of mixed male-female trial teams were no less striking. See Kiser at 81-83 & Figs. 3.14-15.
Returning to the example scorecard in Figure 2, the client-firm relationships identified in the green zone take on added importance. Clients working with these firms see immediately that lawyers of diverse race/ethnicity are making significant contributions. They might reasonably expect, therefore, that these teams are also arriving at more comprehensive decisions and better outcomes. Of course, one virtue of having this information is that, with time and sufficient data, this “green zone” expectation becomes a testable proposition.
3. Diverse law firms make higher profits
At the same time clients benefit from diverse legal teams, so too do law firms. I write this with confidence, having conducted a multi-year study of law firm profitability. This research is anchored by a statistical model that explains close to 90% of the variation in the AmLaw 200 firms’ profitability (and accounts for a litany of audience suggestions, many of which prompted improvements to the analysis).
Focusing on average partner compensation in the AmLaw 200, and pooling three years’ worth of data (FY2015, FY2016, and FY2017), the model explores the roles of a long list of potential “profit differentiators” – i.e., attributes that make some firms more profitable than others. The list includes total attorney headcount, attorney leverage, firm prestige scores (from Vault), partner headcounts by broad practice areas, distinctiveness in the financial services practices, a firm-specific “random effect” to accommodate unobserved firm characteristics (e.g., HQ market), and three “concentration” metrics: (1) client-industry, (2) practice area, and (3) geography.[1]
Figure 3 provides the resulting statistical relationships between average partner compensation and the significant profit differentiators.[2] The dots reflect an attribute’s relationship to profitability (log scale) – specifically, the difference in average partner compensation between a firm with a high versus an average value on the attribute. Lines through the dots represent uncertainty; none of these intersect the baseline, which means all reported predictors are “statistically significant.”[3]
For present purposes, the key predictor in this model involves the percentage share for attorneys of diverse race/ethnicity. The results indicate that, all else being equal, firms with high diversity are more profitable than firms with low diversity. The “all else equal” feature of the approach is important. It means that we account for a range of potentially consequential “profit differentiators” – including many that are not reported because they are not statistically meaningful – and in turn, isolate the connection between racial/ethnic diversity and partner compensation.
Partners take home significantly more money at firms with high diversity. The predictive visualizations in Figure 1 summarize the difference. The orange curve reflects the model-derived profitability differences for firms with low versus average racial/ethnic diversity, and all else equal. The green curve presents the differences for firms with high versus average racial/ethnic diversity. The diversity “dividend” is reflected in the difference between the distributions. Summarizing this difference using the median values (arrow endpoints), the gap between low- and high-diversity firms approaches $180,000 per partner each year.
How can we get more diversity?
In the legal profession, diversity numbers have been flat for over a decade. Despite the best of intentions, change has not materialized. In this post, I’ve covered some of my experiences working on this problem. The key takeaway is that we are getting our data house in order. Having played a part in these data-driven approaches, I am confident that positive change is coming.
What can clients do to realize greater diversity in their outside counsel? They can go a long way by combining existing data with proper resources, getting past the file drawer problem. For a select number of legal departments, LawyerMetrix is equipped to provide early-stage work. Our experience gives us the ability to quickly and cost-effectively convert a company’s “file drawers” into actionable diversity scorecards.
The challenges inherent in increasing law firm diversity are different, yet they are surmountable. A promising way forward is to deploy objective, structured approaches to selecting and developing junior lawyers. In backtesting our services for existing clients, we have found convincing evidence that data plus purpose is powerful. Although part of a longer conversation, I can say that law firm clients of LawyerMetrix have experienced significant diversity gains in both headcount and retention. Our competitive advantage is our ability to use data in a way that lawyers understand, something I’ve had to learn in the trenches rather than in graduate school.
[Editor’s note: Lawyers who dislike the results of a statistical model often attack it by saying it’s wrong. Very few know much about statistics; rather, their objections are rooted in perceptions and personal experience. To deal with this issue head on, I’ve asked Evan to include the following technical notes. wdh]
Technical notes:
[1] The differentiators include each firm’s headquarters market, firm size (total headcount), practice area representations, distinctiveness in the financial services practices, attorney leverage, law firm prestige, and three “concentration” metrics: (1) client-industry, (2) practice area, and (3) geographic. The concentration metrics (0 to 1 scale) are calculated using a statistic termed a “Herfindahl Index.” A score of zero (0) reflects no concentration (e.g., regarding geography, a firm’s attorney headcounts are equal across six offices), a score approaching one (1) reflects high concentration (e.g., 80% of a firm’s attorneys are in a single office, and the remaining 20% are distributed across five offices), and a score equal to one (1) reflects total concentration (i.e., all attorneys are in a single office).
[2] The statistical model includes an adjustment for each firm’s unique, unmeasured characteristics. This is important because it ensures that certain findings are not merely a function of other unmeasured characteristics – say, that the geographic concentration effect does not exist simply because large labor markets have more geographically-concentrated law firms (e.g., New York City).
[3] In Figure 3, the thick horizontal lines capture the values that are most likely to reflect the true relationship between partner compensation and the firm attributes. The thin line captures the range of all but the most extreme values.
What’s next? See An Rx for Data Dizziness: Getting Started with Data Requires Smarter Strategy (075)