The data exist to make legal education measurably better.
Figure 1 above uses data from the 2020 Law School Survey of Student of Engagement (LSSSE) to visualize 10 average score “distributions” based on responses from 12,969 law students at 68 participating U.S. law schools. The distributions are at the law school level. Thus, for all ten measures, each participating law school’s average score exists somewhere within the orange-yellow-grey-blue-green distribution. The grey is the fat part of the underlying bell curve (25-75th percentiles). In addition, each Figure 1 measure maps to one or more of the ABA’s Accreditation Standards (see references in brackets).
Figure 1 is an example of what data-driven education assessment looks like—a theory of quality is embedded inside the Accreditation Standards. Performance on those standards is measured using data. As part of an ongoing and iterative process, data clearly guides law schools through how to improve. Likewise, as overall averages trend upward, legal educators and regulators can credibly say, “The quality of US legal education is improving.”
Since 2014, I’ve collaborated with LSSSE researchers to design and deliver easy-to-grasp reporting on students’ law school experiences. The reports and visualizations, similar to what you see above, help school administrators and faculty identify their school’s areas of strength and areas for potential improvement. Among other things, we benchmark each school’s average scores on every LSSSE question tied to an ABA Accreditation Standard. (Credit to Chad Christensen and LSSSE for mapping the Survey questions to the ABA Standards.) LSSSE is one of the few data-driven ways that a law school can document its compliance with ABA Standards.
In this post, I use LSSSE data to evaluate U.S. legal education quality, first on ABA Accreditation factors (Figures 1-2) and then on a broader set of LSSSE factors crucial to overall student satisfaction (Figure 3).
Before diving into the data, however, let me make two interpretative points that apply to Figures 1-2 and are recurring themes in this post:
- Narrow distributions: when the distributions are compact, law schools are much more alike than they are different. Thus, we can generalize across all law schools with confidence.
- Wide distributions: when the distributions are wider, there is significant variability across law schools. This means that on specific dimensions, some law schools are excelling while others are floundering.
We learn a lot about the state of legal education by comparing the factors with narrow versus wide distributions. And we also see that, in the eyes of students, Accreditation Standards offer an incomplete view of what matters most to an excellent legal education.
I. ABA Accreditation Standards that drive student satisfaction
Of the 82 items on the annual LSSSE survey, 59 are connected to one or more ABA Accreditation Standards. Of those 59 factors, which ones have the greatest impact on a student evaluating their entire law school experience as “Excellent”? The top ten are reported in Figure 1.
One key takeaway of Figure 1 is that law students value the educational transformation that occurs during law school—learning to think critically and analytically, acquiring a broad legal education, and writing effectively. Because these top three factors are narrow distributions (particularly critical thinking) on the high end of the scale, legal education generally does well on these factors.
Yet students also value “fostering collaborative work environments” and “acquiring job or work-related knowledge.” As narrow distributions with relatively low scores, these are areas where all of legal education must do better.
For wide distribution factors, there is wide performance variability across law schools for factors related to technology, academic advising, financial aid, and support in the job search. Besides wide distributions, what do these factors have in common? They are all things that require sufficient resources and skilled professional staff to do well—and also fully within the scope of what a law school dean is supposed to manage. Yet, through the results in Figure 1, we now know these factors are also a big part of how law students are evaluating their overall educational experience. This should help deans make harder decisions sooner.
II. Distributions of 43 Accreditation factors
The complete set of 43 Accreditation questions measured on a common 1 to 4 scale in LSSSE 2020 appears in Figure 2 [click on to enlarge].
Figure 2 offers a comprehensive view of legal education’s strengths and problems in 2020. Now, we see lots of variation across schools, reflected in the wide span between many of the orange and green endpoints. If these are the items students need to succeed in law—which is what their connection to the ABA Accreditation Standards seems to suggest—then there are schools falling short in a multitude of ways.
Note that through LSSSE, each participating law school gets a similar summary chart that includes an overlay of the school’s average scores. Thus, on each Accreditation factor, the school can see (a) how all the LSSSE schools are doing (about 1/3 of all law schools) and (b) how the school compares in percentile terms.
On where reform needs are most urgent in the aggregate, consider again that most schools have similar highs and lows. Generally, schools receive very favorable reviews on the traditional academic factors—emphasizing learning and analytical thinking about case-relevant facts along with a broad legal education. And their greatest shortcomings are in areas related to faculty and student relationships, especially as they relate to advancing career goals.
As discussed in Part III below, the larger LSSSE dataset sheds a lot of light on how student relationships with faculty, administrators, and each other impact the overall law school experience.
III. Drivers of overall student satisfaction with the law school experience
It’s instructive to update the statistical model used to identify top drivers (Figure 1 pulled from the 59 Accreditation items) by using the full set of LSSSE questions as predictors.
With nearly 13,000 survey responses in LSSSE 2020, I included all possible inputs in the model using the standard LSSSE question battery (82 questions in total). In Figure 3 below, the results compare students whose different levels of the inputs (favorable vs. unfavorable) predict differences in satisfaction. For this post, we’ll compare student responses that are typical (at the mean) versus very favorable (+2 standard deviations above the mean), and all else equal.
As shown in Figure 3, the larger number of LSSSE factors enhances our understanding of student satisfaction, with one particularly important takeaway—interpersonal relationships matter a lot. For example, students who say that law school faculty members were very much “available, helpful, sympathetic” have a much greater likelihood of reporting an excellent experience than those who say faculty were “unavailable, unhelpful, unsympathetic,” all else equal.
Scanning across the items in Figure 3, three general themes emerge that involve (1) quality of relationships, (2) counseling and support, and (3) the law school curriculum.
- Quality of Relationships. One of the strongest positive differentiators is LSSSE question (9a), the quality of relationships with other students. On this item, LSSSE asks students whether their relationships with other students are unfriendly or friendly using a 1 to 7 scale. At schools where students report above-average friendliness among their peers, they are also more likely to view their law school experience as excellent. The same is true for relationships with faculty (9b) and administrative staff (9c), who are evaluated in terms of their availability and helpfulness, respectively.
- Counseling and Support. A second set of factors are strongly positively related pertain to students’ perceptions of support. This is the case per the importance of (8c), about the school’s ability to provide the support to succeed (1 = Very Little, 4 = Very Much), and with academic advising (6a), personal counseling (6c), and financial aid advising (6e). It’s not surprising the quality of a school’s advising and support functions matters a great deal to students.
- Law School Curriculum. A final set of factors pertain to the legal curriculum. Students who believed they had acquired a broad legal education (10a) along with job or work-related knowledge (10b) and were taught to think critically and analytically (10e) were most likely to rate their law school experience as “Excellent.”
Quality of relationships is a huge driver of law school satisfaction, yet largely beyond the purview of the input-driven quality standards set by the ABA. A significant portion of legal education quality, at least in the eyes of students, is driven by culture—and requires far-sighted leadership with the desire and ability to shape it.
Data brings focus and action
One of the biggest benefits of focusing on data is that facilitates a focus on action.
In this analysis, we’ve seen both how some schools outperform others and how generally, schools’ average scores rise or fall together. And crucially, we’ve also got direct information about how to address specific outcomes. What we know from the survey is certain factors predict an “excellent” response. Rather than speculating about what needs to change, we can get to work building educational initiatives and programs that support student needs today and in the future.
Finally, as legal educators and innovators work to enhance the value of attending law school in the midst of a global pandemic—a key need with applications down as much as 4%—I’d recommend they focus on data. This post offers insight about what students find most valuable and what they believe schools fail to provide. Calls for increasing access through technology notwithstanding, we’ve seen here that some of the most important features of the law school experience will be difficult to replicate remotely.
N.B. In a typical year, between 65 and 85 law schools participate in LSSSE. How in the world can the remaining 2/3 of law schools discuss how well their schools are performing and what needs to be improved? In the year 2020, we should expect more from professional schools.