Practice-Ready Licensing

This article is adapted from an article, “Enhancing the Validity and Fairness of Lawyer Licensing: Empirical Evidence Supporting Innovative Pathways,” in the Washington University Journal of Law & Policy.

Licensing systems aim to protect the public by excluding incompetent practitioners from the profession. The bar exams administered in the United States, however, fail to live up to that promise. Those exams do not assess candidates’ ability to counsel clients, interview witnesses, or negotiate with opponents. Nor do they require candidates to draft contracts, respond to discovery requests, or develop strategies for addressing client matters. The exams do not even require candidates to show their proficiency at legal research, an essential skill given the vast number of U.S. and foreign rules that will affect today’s clients.

In the United States, each jurisdiction (including states, territories, and the District of Columbia) sets its own rules for admitting candidates to the bar. Some of those jurisdictions have started to ask: What if the requirements for licensing more effectively measured the competencies that new lawyers need for practice? These jurisdictions are breaking the hold of the traditional bar exam, creating new ways to assess the competence of candidates for bar admission.

In this article, we describe the problems that plague the bar exam and the new avenues that jurisdictions have adopted or are exploring. We then present empirical evidence that one of these avenues—evaluating candidates during a period of postgraduate supervised practice—is a valid, feasible, and fair way to assess competence for law practice. This approach, the data shows, can benefit candidates, their employers, and the clients they serve.

What’s wrong with the bar exam?

The traditional bar exam falls short on three fronts: (1) It fails to assess the knowledge and skills that new lawyers most need to practice competently. (2) It imposes heavy financial burdens on both jurisdictions and test-takers. (3) Partly because of those financial burdens, the exam disproportionately excludes from the profession candidates of color, candidates with caretaking responsibilities, and candidates who live with disabilities.

Traditional bar exams measure fewer than half the competencies identified by research, and NCBE’s proposed NextGen exam will do little to address the problem. Instead, those exams focus on knowledge and skills that are unnecessary—and even harmful—for contemporary practice. They test knowledge of common-law rules that no longer govern client matters, require extensive memorization of legal rules, and prize rapid responses over reflection and deliberation.

Lawyers who participated in the Building a Better Bar study emphatically rejected these exam-based habits. New lawyers must look to contemporary statutes, regulations, and court decisions rather than common-law rules; they must check their knowledge from sources rather than relying on memory; and they must take time to reflect on client matters, seek input from others, and revise their written work. The slapdash responses demanded by the bar exam can produce mistakes in practice.

Practicing lawyers also criticized the static fact patterns that dominate bar exams. These questions do not test the candidates’ ability to explore a client’s varying goals, gather facts, or adapt to the changing circumstances of real-world representation. Nor do the exams require candidates to develop a holistic view of client problems. Bar exam questions focus candidates on narrow doctrinal issues, replicating law school exams rather than law practice.

Expenses

These flawed exams are expensive for both jurisdictions and candidates. Each exam requires the development, editing, and pretesting of questions. Jurisdictions then pay steep fees to rent exam sites, compensate proctors, and arrange accommodations for test-takers with disabilities. California spends more than $5,600,000 per year to cover those costs alone. Grading, scaling, and equating the exams add still more to the tab.

The heaviest expenses, however, fall on candidates. They pay some of the exam development and administrative costs through exam fees, and they pay travel and lodging costs to stay near exam sites. Most burdensome, candidates purchase expensive bar-preparation courses and forego income while devoting 10 or more weeks to intensive study. The exams’ focus on excessive memorization and undue speed requires those expenditures: research demonstrates that candidates who do not—or cannot—assume bar-prep costs are more likely to fail the exam.

Exclusion

The bar exam’s expense disadvantages test-takers with limited financial means, as well as those with caretaking responsibilities. Educational loans, notably, do not cover bar-prep courses, exam fees, travel expenses, or the cost of living while studying for the exam. Unequal resources explain at least part of the race and gender gaps that characterize bar results. Bar exams sharply favor white test-takers over examinees of color and offer some advantage to male candidates. Jurisdictions rightly worry about this impediment to diversifying the profession.

Many candidates who live with disabilities also struggle with bar exam. Jurisdictions require them to submit expensive documentation to secure accommodations and often refuse accommodations that candidates have relied upon during law school. The fact that the exam is offered just twice a year imposes additional obstacles for candidates with chronic conditions, those with illnesses, and those who are pregnant or breastfeeding.

States, finally, set different passing scores for their exams—even though most administer the same Uniform Bar Examination. Research shows that higher passing scores dramatically reduce the diversity of lawyers in states with those scores, without increasing the competence of new lawyers. Economic advantage and freedom from caretaking exacerbate this unfairness: candidates who fail in one jurisdiction, but have sufficient financial resources and personal mobility to relocate, can move to a state where their score will allow them to practice law.

New avenues

In light of these flaws, several jurisdictions are exploring or adopting new ways to license lawyers. Starting in May, candidates for admission to the Oregon State Bar will be able to choose between taking the traditional bar exam and demonstrating their competence through a Supervised Practice Portfolio Examination (for more on this, see “Speaker’s Corner.”) Candidates who choose the latter option will obtain provisional licenses, allowing them to practice under supervision. During their supervisory period they will compile portfolios consisting of written work product, assessments of client encounters, and appraisals of negotiations. Bar examiners will review redacted versions of this work to determine whether a candidate is competent to practice law.

The California State Bar has recommended that the California Supreme Court establish a pilot program for a similar licensing option. Task forces in several other states are exploring ways other than the traditional bar exam to measure lawyering competence. Updated information about those innovations, including licensing paths based upon experiential work in law school, is available from a website we maintain with other scholars.

Some lawyers wonder whether supervised-practice pathways will be valid, feasible, and fair. Will they measure the skills and knowledge that new lawyers need? Will practicing lawyers be willing to serve as supervisors? Will these programs generate new forms of bias and unfairness for applicants, which has been a problem in countries using articling and apprenticeships?

Supervised-practice pathways differ significantly from articling and other forms of apprenticeship. The pathways offer an option, not a mandate, which reduces both the demand for supervisors and the power that supervisors hold over candidates. Independent examiners provide a further check on supervisors’ power. Substantial research in medical workplaces, meanwhile, shows that it is possible to evaluate licensees fairly and rigorously as they work with patients; those studies offer detailed models for assessing lawyers. And we already have evidence, drawn from a study of New Hampshire-’s Daniel Webster program, that portfolio-licensed lawyers can outperform peers who pass a traditional bar exam. Our research adds to this evidence base.

The current study

We obtained access to a unique data set offering a first look at the validity, feasibility, and fairness of licensing systems that, like the one adopted in Oregon, assess candidates’ competence while they engage in postgraduate supervised practice. The data derives from participants in two California programs. The “Original Program” allows pandemic-era graduates to practice under supervision while waiting to pass the bar exam. The “Pathway Program” allows some individuals who narrowly failed the bar exam to become fully licensed after 300 hours of supervised practice. While working under supervision, candidates in both programs use provisional licenses that allow them to do most types of legal work.

Neither program completely parallels the licensing pathways that Oregon recently adopted and other states are considering. Candidates in the Original Program must still take the bar exam to gain full licenses, while those in the Pathway Program do not submit portfolios of work product to examiners for review. The programs, however, offer important insights into the competencies that can be assessed during supervised practice, as well as the feasibility and fairness of such programs.

In fall 2022, the California State Bar surveyed candidates and supervisors in both programs, as well as candidates who were eligible for the Pathway Program but did not participate. Those surveys generated the data we analyze in this article. The survey responses offer strong evidence that practicing lawyers are willing to supervise candidates for bar admission; that workplaces have sufficient training and mentoring systems in place to support those programs; that candidates find supervised-practice placements fair; that supervised practice assesses a greater range of knowledge and skills than the bar exam; and that innovative licensing paths expand client service—including to clients who currently lack access to justice.  

Key findings

Benefitting employers: Feasibility

Supervised-practice licensing systems are feasible only if practicing lawyers are willing to supervise candidates working toward their licenses. In California, almost 1,400 licensed lawyers stepped forward to supervise candidates with provisional licenses. That number of volunteers is particularly impressive because the program was new, and the state bar offered no incentives or support for participation. Survey responses, moreover, suggest that many of these volunteers are willing to make an ongoing commitment. More than two-thirds of supervisors (70.6 percent) indicated that they were willing to continue supervising candidates, and another 16.5 percent were open to that possibility.

Survey respondents backed up this commitment by pointing to many benefits of working with candidates holding provisional licenses. Supervisors were satisfied with their candidates’ work and thought their candidates were especially hardworking. This competence and work ethic allowed organizations to serve more clients: almost nine-tenths of supervisors (86.8 percent) reported expanding their client base with the help of supervised-practice candidates.

Supervisors also applauded the diversity that candidates brought to their practice teams. The candidates were more demographically diverse than recently licensed lawyers, and some possessed unusual life experiences or training. That diversity allowed organizations to enhance service to existing clients, tap new client bases, and even explore new practice areas.

California’s provisional licensing programs, finally, complemented employers’ hiring practices. The programs were attractive to candidates, which gave organizations an advantage when recruiting in a tight labor market. At the same time, the programs allowed organizations to give candidates a trial run before committing to a full-time position. During that trial period, candidates did not need time off to study for the bar exam; they could devote all their attention to learning the employer’s practices and serving clients.  

Supervisors reported obtaining these benefits with few costs. More than two-thirds of supervisors experienced no problem “at all” from candidates making mistakes, and just 4.4 percent reported that mistakes concerned them to a “great extent.” Similarly, few employers found the supervision and training of candidates burdensome. Most organizations already provided supervision and training to newly licensed lawyers; they were able to use that infrastructure to provide oversight and feedback for candidates with provisional licenses. Many supervisors shared the sentiment of a government lawyer who commented: “This program is rare in that I cannot identify one downside as it was administered in my office.”

Benefiting candidates: Fairness

Some stakeholders worry that bias and old boy networks could disadvantage women and people of color in finding supervisors or succeeding in supervised-practice pathways. Just the opposite was true in California’s programs. Women and people of color were significantly more likely than white men to participate in the Pathway Program, and they were slightly more successful than white men in completing the program and earning full licenses. 

First-generation college graduates, individuals living with disabilities, and individuals who identified as LGBTQIA+ also succeeded as candidates. We found no significant difference in success rates for these groups compared with other candidates. Nor did satisfaction ratings differ by race/ethnicity, gender, first-generation status, disability, or sexual orientation. Members of all groups expressed very high degrees of satisfaction with supervised practice. Indeed, respondents from historically disadvantaged groups offered eloquent comments about the importance of the supervised-practice program to them, their families, and their professional careers. Some supervisors volunteered similar observations.

Some candidates (9.7 percent) reported experiencing harassment or discrimination while engaged in supervised practice, but most of them characterized the challenges from this negative treatment as “small” or “moderate.” More important, candidates who reported discrimination or harassment were just as likely as other candidates to succeed in the programs. They also reported the same levels of satisfaction as candidates who did not report any negative treatment. Some candidates noted that the discrimination or harassment they experienced in the programs was no greater than what they endured in other contexts and that the programs, on balance, “countered” discrimination by allowing them to establish their competence and serve clients.

Stakeholders also worry that supervised-practice programs could force candidates to work without pay or accept low-paying positions. Almost all (93.6 percent) of the candidates in California’s Original Program, however, received compensation. Two-fifths of supervisors (39.4 percent) reported paying those candidates the same salaries as they paid newly licensed lawyers; 37.4 percent paid the same wages given to unlicensed law graduates; 14.4 percent paid paralegal wages; and 8.4 percent paid law student rates. These numbers suggest that programs allowing candidates to substitute supervised practice for the written bar exam may benefit both candidates and employers financially. Candidates will save the heavy expenses of bar preparation (including 10 weeks of foregone income), while employers have the option of modestly discounting entry-level salaries during the licensing period. Jurisdictions, meanwhile, can ensure that employers don’t discount salaries too heavily by requiring that employers pay candidates at least as much as they pay other law graduates who have not yet passed the bar exam.

Benefiting clients: A more valid measure of competence

The California data confirms that supervised-practice programs can assess more knowledge and skills than the traditional bar exam. More than three-quarters of candidates reported that they used key skills that are not tested on the bar exam. The prevalence of these skills in the first months of law practice demonstrates both the need to assess candidates’ competence in performing them and the possibility of assessing them during a period of supervised practice.

Percentage of licensees exercising skills

Jurisdictions can readily incorporate each of these skills into their portfolio requirements. For the small number of candidates who are not able to exercise these skills in their supervised-practice settings, jurisdictions can follow Oregon’s lead and create simulations for candidates to use in developing and demonstrating missing skills. While the traditional bar exam fails to test essential lawyering skills like research, counseling, and client communication, supervised-practice pathways will protect clients by rigorously assessing these critical competencies.

California’s survey responses offer similar reassurance about the scope of doctrinal knowledge that can be assessed through supervised practice. Candidates reported drawing upon an average of 5.5 doctrinal areas in their practice, with a quarter of them listing eight or more subject areas. Even candidates who focused on a particular practice area, such as criminal law or personal injury work, drew upon concepts from a range of subjects.

Candidates did not work in every subject area that they might pursue as lawyers, so a licensing system based on supervised practice could not assess their knowledge in all fields. This, however, is also true of traditional bar exams. Those exams assess knowledge in just eight to 12 doctrinal areas, with a heavy emphasis on common-law rules. The California data demonstrates just how narrow the bar exam is: almost nine-tenths of candidates reported using knowledge from subjects that are not tested on the bar exam, and more than a fifth reported practicing in four or more subjects that do not appear on the bar exam.

Supervised practice, in other words, may be more effective than the bar exam at testing candidates’ knowledge in areas of contemporary practice. Those areas include critical fields like immigration, intellectual property, arbitration, tax, employment law, and dozens of other areas that are not tested on the bar exam. Licensing systems rooted in supervised practice, moreover, will adapt naturally to changes of focus in entry-level practice.  

What about candidates who work in one field during supervised practice and then, after receiving a full license, switch to another practice area? Will those lawyers endanger clients? Once again, the same question applies to lawyers licensed through the bar exam. They are allowed to practice environmental law, international tax, admiralty, and any other subject. Supervised-practice pathways, like the bar exam, assume that professionals who have demonstrated their competence in some areas of practice are able to transfer their skills to other areas and learn the doctrinal rules of those new subjects.

The California survey did not ask supervisors to compare candidates to attorneys who had passed the bar exam, but some did so spontaneously. “Our [candidate],” one law firm partner wrote, “has been the best ‘associate’ that we have had at our firm, better than associates that have passed the bar exam.” Other supervisors noted that candidates gained “on the job training,” experience “dealing with clients,” and “more applicable knowledge” than peers who had passed the bar exam.

Finally, our research suggests that supervised-practice licensing paths can expand access to justice. In the California programs, more than a fifth (22.2 percent) of licensees worked for legal aid, public defenders, or other public-interest employers. They were able to represent disadvantaged clients immediately, without waiting to take and pass the bar exam. As one provisional licensee noted, “I am working as a first-generation lawyer in a legal aid office.  I have already put in 3,000 hours of direct community lawyering in a domestic violence clinic and doing housing rights advocacy.” Public-interest employers expressed similar enthusiasm for supervised practice. “The provisional licensees I supervised were highly competent legal advocates,” a legal aid attorney wrote, “passionate about serving low-income clients, and expanded the availability of legal services in underserved portions of rural California.” 

Building on the data

Data from California’s supervised-practice programs suggests that these programs offer a promising pathway for licensing lawyers. Practicing lawyers are willing to supervise candidates, and their organizations benefit from the candidates’ work. Supervised-practice paths lower expenses for law graduates and open the profession to candidates who can more readily demonstrate their competence in practice than on a traditional exam. Most important, supervised-practice paths allow jurisdictions to protect clients by using a more comprehensive and rigorous assessment process that encompasses a fuller range of skills and contemporary doctrine than the bar exam measures.

California’s current programs, though, are just the beginning. Jurisdictions can build on those programs—as Oregon has done—to increase their feasibility, fairness, and validity. Law schools and bar associations can help connect candidates with willing supervisors. Bar associations might offer training to supervisors who want to enhance their supervisory skills. Regulations can prevent economic exploitation of candidates, and an ombudsperson can aid candidates who encounter problematic treatment. Lastly, candidates can compile portfolios of redacted work product that they submit to bar examiners for review. Measures like these will develop a rigorous, inclusive licensing system that benefits employers, candidates, clients, and the public.

Conclusion

A team of highly regarded psychometricians once wrote: “The time-honored way to find out whether a person can perform a task is to have the person try to perform the task.” State courts and bar examiners are starting to heed that advice, seeking more authentic ways to assess the competence of aspiring lawyers. Our research shows that licensing systems rooted in supervised practice offer a valid, feasible, and fair avenue for measuring the knowledge and skills that lawyers need to serve contemporary clients effectively. As jurisdictions explore those options, we hope that they will add to this research base. Research can point the way to licensing systems that increase both the competence and inclusiveness of our profession.

---

Deborah Jones Merritt is Distinguished University Professor and John Deaver Drinko-Baker & Hostetler Chair in Law Emerita at Moritz College of Law, the Ohio State University.

Andrea Anne Curcio is Professor of Law at Georgia State University College of Law.

Eileen Kaufman is Professor of Law at Touro Law Center.