This post is a data-rich progress report on the Institute for the Future of Law Practice (IFLP, “I-flip”).
For readers unfamiliar with IFLP, here are basics. IFLP is an education nonprofit that combines sophisticated training in modern law practice with paid internship for law students. Following a successful multi-year pilot at Colorado Law, see Post 018 (discussing Tech Lawyer Accelerator (TLA) program); Post 113 (TLA-IFLP timeline), IFLP was formed to scale the key learnings for the benefit of other law schools and, crucially, to build out a network of employers who value legal training combined with foundational knowledge in data, process, technology, business operations, and teamwork. See Post 043 (discussing launch of IFLP).
Between year 1 and 2, IFLP grew from one to three bootcamps (Boulder, Chicago, Toronto) and from five to 18 law schools across the US, Canada, and Europe. 64 students attended a 2019 “basic track” bootcamp before heading off to work for an IFLP employer (mostly 10-week internships, some 7-month field placements). We also placed several returning 10-week interns from 2018 in 7-month paid field placements (from June to Dec of their 3L year). In total, ten students from 2018 and 2019 are now on 7-month field placements. Part of this program is an “advance track” bootcamp that this year takes place in November. IFLP’s current 7-month employers include Cisco, Cummins, Baker McKenzie, and Perkins Coie.
As an IFLP co-founder, here are the statistics I am most proud of:
- The 2019 incoming class was 52% diverse and 64% female (more on that below)
- 54 IFLP employers, with several providing two or more internships
- Across the three bootcamps, 100+ professionals who volunteered their time to teach, coach, and provide feedback to students.
Our total funding during our first two years of operation — to stand up a website, create promotional materials, recruit employers and law schools, process hundreds of student applications, vet and place nearly 100 students, build a business infrastructure, obtain 501(c)(3) status, and run four total bootcamps — was $365,000. Many thanks to our generous sponsors Chapman and Cutler, Cisco Legal Department, Elevate, LSAC, and Quislex, who got us through years 1 and 2.
IFLP was bootstrapped for the simple reason that without producing results, we’d never attract the much larger amount of funding we’d need to scale. Thus, working prototypes and results were much more important than a master plan filled with ideas. Yet, this journey began two years ago. As IFLP launches its first capital campaign, see Post 119, it’s time to share results.
Employer and Student Data
As the 10-week summer internships wound down, IFLP surveyed employers and students about their experience. The employers include a mix of law firms, legal departments, NewLaw, legaltech, public interest organizations, and consulting firms. See 2019 Employers. The students are rising 2Ls and 3Ls who attended a 3-week bootcamp in May in Boulder, Chicago, or Toronto and then spent the balance of the summer break with an IFLP employer.
Employers were in strongest agreement with the statement “Our intern delivered value,” which is a very important metric. LIkewise, to counter the perennial client complaint that lawyers don’t understand their business, we built a module that requires students to study their employer and learn about their industry. 4.20 on a 5-point scale is not a bad start. The same can be said for the other scores.
The graphic below summarizes the experience of our 10-week summer interns:
Although this quantitative data is both positive and encouraging, the qualitative data (i.e., the text fields and comment boxes in the surveys) identify specific areas for improvement. And there’s a lot of work to do. What’s displayed above is baseline data.
Net Promoter Score
Many IFLP employers use metrics like Net Promoter Score® (NPS®). Thus, our employer survey included the question, “On a scale of 0 – 10, how likely are you to recommend the IFLP program to colleagues in other companies and firms?” Likewise, using the same 0 – 10 scale, IFLP interns were asked “how likely are you to recommend the IFLP boot camp and internship program to other law students?”
To calculate NPS for a 10-point scale, responses are binned into three groups:
- Promoters (score 9-10): loyal enthusiasts who will keep buying and refer others, fueling growth.
- Passives (score 7-8): satisfied but unenthusiastic customers who are vulnerable to competitive offerings.
- Detractors (score 0-6): unhappy customers who can damage your brand and impede growth through negative word-of-mouth.
NPS is then calculated by applying the following formula: % Promoters – % Detractors = NPS. See “What is Net Promoter?,” NICE Satmetrix.
For 2019, the NPS for IFLP employers is 71. The NPS for IFLP interns is 66. The scale below, generated by Retently, which makes net promoter software, provides some guidance on how to interpret these scores:
Why does NPS matter? Because for most organizations, the most powerful marketing is peer-to-peer testimonials. Thus, an organization with a high NPS dramatically lowers the cost of acquiring a customer, as promoters are doing most of the sales and marketing.
What’s the goal?
IFLP is trying to drive innovation in legal education by harnessing the power of the labor market. If IFLP employers are highly satisfied with our training and internship programs (stakeholder #1) and law students feel the same (stakeholder #2), a clear signal on curricular innovation is sent to law schools (stakeholder #3).
Over the long term, we anticipate that the type of training offered in IFLP bootcamps (focused on creating T-shaped legal professionals) will migrate into the 2L and 3L year (and possibly into master degree and undergraduate programs). The purpose of our bootcamps is to speed up this transition phase for the benefit of law students, clients, and broader society. See also Post 078 (“IFLP is trying to accelerate the development of T-shaped legal professionals”); Post 043 (discussing need for “new tools and methodologies that were not, and are not, part of traditional legal education”).
Connection between quality control and diversity
Without a rigorous system of quality control, IFLP will struggle to retain and grow a network of legal employers. And without employers, IFLP fails.
In our first two years, IFLP acquired our current roster of employers through thousands of hours of phone calls, emails, in-person visits and calling-in of favors, most of which was done on a volunteer basis by IFLP founders and Board members. If we impute a $100/hour value to these efforts (a conservative estimate), the cost of acquiring an IFLP employer is roughly $5,000.
A strong curriculum is one way to reduce the risk of a dissatisfied IFLP employer. But equally important is vetting students on the front-end. Specifically, IFLP selects for professional competencies valued in a highly collaborative knowledge-worker environment. See IFLP competency model. Further, we significantly increase our accuracy by using a structured interview process that includes job-relevant questions and scores student answers with behaviorally anchored rating scales (BARS). Indeed, a valid and reliable interview takes the same amount of time as an invalid and unreliable interview. The only difference is interviewer training combined with proper interview tools.
For the recently completed 2019 program, IFLP received 278 applications and interviewed 196 students. To efficiently process candidates, we used an online platform (Breezy HR) that includes a feature for video interviews. Who conducted the interviews? A dozen 2018 IFLP alumni, always in teams of two (which increases accuracy by decreasing bias). They are just as accurate as IFLP staff but significantly less expensive. Further, at the close of the 25-minute interview, our alums are ideal people to answer questions about the program. In 2019, our total cost for interviewing 196 students (our alums are paid a set price per interview) was approximately $9,000 — roughly the cost of replacing two dissatisfied IFLP employers.
In contrast, what is not part of the IFLP screening process is law school rank or law school GPA. This is because the rigors of getting into law school and successfully completing the 1L year is more than sufficient to find candidates with relatively strong intellectual ability. When we widened the aperture in this way, we gain access to a much deeper and more diverse talent pool. See Parker, “Credible commitments to legal diversity (114),” Legal Evolution, Sept. 2, 2019 (documenting significantly more diversity at less elite law schools and making point that it’s not credible to “support diversity” while ignoring this talent pool). In turn, when we use a valid, reliable interview process, out the other side is a group of high scorers that reflects or exceeds the diversity of the initial pipeline. Indeed, this is how IFLP ended up with a 2019 class that was 52% diverse, 64% female, and earned an employer NPS of 71. For the science and mechanics behind this process, see Henderson, “Solving the Legal Profession’s Diversity Problem,” PD Quarterly (Feb. 2016).
IFLP’s interview process is a lot of front-end work. Yet, it gives us two enormous advantages: (1) our overall candidate pool is stronger, and (2) we can vigorously advocate for all students in our employer pool, as we have meaningful evidence of their professional abilities.
IFLP’s plans for 2020
IFLP would like to grow. However, at the end of our 2019 fiscal year, IFLP management made the decision to run three bootcamps in 2020 (Boulder, Chicago, Toronto) and limit the number of 2020 slots to 72 (24 students per location). We did this for three reasons.
First, we want to improve the quality and efficiency of the education we offer. This means designing and building modules that (a) take better advantage of asynchronous learning, (b) allocate more class time to interactive, team-based projects and learning (i.e., flipped classrooms), and (c) include more and better student assessment and feedback. This type of educational design is very time-consuming and expensive. However, it’s the only type of educational design that can be cost-effectively scaled to a much larger number of students. To achieve our mission, we need to start building this foundation.
Second, a crucial part of IFLP’s model is to harness the power of the labor market. In 2020, we want to increase the number of IFLP employers without increasing the supply of IFLP bootcamp slots. This is because it is stressful for 1Ls to wait until March or April to obtain an employer match. Competition among satisfied employers is one way (perhaps the best way) to speed up the matching process and thus ease uncertainty and stress for law students.
Third, and perhaps most important, IFLP needs to raise money. Adding more bootcamps increases our cost and execution risk without generating any corresponding revenue. As an organization, we’ve reached the absolute limit of what we can accomplish with a mostly volunteer workforce. See Post 112. Thus, Post 119 is the soft launch for IFLP’s capital campaign.