This article is adapted from an article, “Enhancing the Validity and Fairness of Lawyer Licensing: Empirical Evidence Supporting Innovative Pathways,” forthcoming in the Washington University Journal of Law & Policy.
Licensing systems aim to protect the public by excluding incompetent practitioners from the profession. The bar exams administered in the United States, however, fail to live up to that promise. Those exams do not assess candidates’ ability to counsel clients, interview witnesses, or negotiate with opponents. Nor do they require candidates to draft contracts, respond to discovery requests, or develop strategies for addressing client matters. The exams do not even require candidates to show their proficiency at legal research, an essential skill given the vast number of U.S. and foreign rules that will affect today’s clients.
In the United States, each jurisdiction (including states, territories, and the District of Columbia) sets its own rules for admitting candidates to the bar. Some of those jurisdictions have started to ask: What if the requirements for licensing more effectively measured the competencies that new lawyers need for practice? These jurisdictions are breaking the hold of the traditional bar exam, creating new ways to assess the competence of candidates for bar admission.
In this article, we describe the problems that plague the bar exam and the new avenues that jurisdictions have adopted or are exploring. We then present empirical evidence that one of these avenues—evaluating candidates during a period of postgraduate supervised practice—is a valid, feasible, and fair way to assess competence for law practice. This approach, the data shows, can benefit candidates, their employers, and the clients they serve.
What’s wrong with the bar exam?
The traditional bar exam falls short on three fronts: (1) It fails to assess the knowledge and skills that new lawyers most need to practice competently. (2) It imposes heavy financial burdens on both jurisdictions and test-takers. (3) Partly because of those financial burdens, the exam disproportionately excludes from the profession candidates of color, candidates with caretaking responsibilities, and candidates who live with disabilities.
The recent Building a Better Bar study offers a particularly persuasive picture of the competencies new lawyers need. That study convened 50 focus groups, some with new lawyers and some with supervisors, in 18 locations across the country. The lawyers described the competencies that new lawyers need, the mistakes they make, and the training that would help them avoid those mistakes. Analysis of transcripts from those sessions demonstrated that lawyering competence rests on 12 interlocking building blocks:
- The ability to act professionally and in accordance with the rules of professional conduct
- An understanding of legal processes and sources of law
- An understanding of threshold concepts in many subjects
- The ability to interpret legal materials
- The ability to interact effectively with clients
- The ability to identify legal issues
- The ability to conduct research
- The ability to communicate as a lawyer
- The ability to see the “big picture” of client matters
- The ability to manage a law-related workload responsibly
- The ability to cope with the stresses of legal practice
- The ability to pursue self-directed learning
Traditional bar exams measure fewer than half the competencies identified by research, and NCBE’s proposed NextGen exam will do little to address the problem. Instead, those exams focus on knowledge and skills that are unnecessary—and even harmful—for contemporary practice. They test knowledge of common-law rules that no longer govern client matters, require extensive memorization of legal rules, and prize rapid responses over reflection and deliberation.
Lawyers who participated in the Building a Better Bar study emphatically rejected these exam-based habits. New lawyers must look to contemporary statutes, regulations, and court decisions rather than common-law rules; they must check their knowledge from sources rather than relying on memory; and they must take time to reflect on client matters, seek input from others, and revise their written work. The slapdash responses demanded by the bar exam can produce mistakes in practice.
Traditional bar exams measure fewer than half the competencies identified by research, and NCBE’s proposed NextGen exams will do little to address the problem.
Practicing lawyers also criticized the static fact patterns that dominate bar exams. These questions do not test the candidates’ ability to explore a client’s varying goals, gather facts, or adapt to the changing circumstances of real-world representation. Nor do the exams require candidates to develop a holistic view of client problems. Bar exam questions focus candidates on narrow doctrinal issues, replicating law school exams rather than law practice.
These flawed exams are expensive for both jurisdictions and candidates. Each exam requires the development, editing, and pretesting of questions. Jurisdictions then pay steep fees to rent exam sites, compensate proctors, and arrange accommodations for test-takers with disabilities. California spends more than $5,600,000 per year to cover those costs alone. Grading, scaling, and equating the exams add still more to the tab.
The heaviest expenses, however, fall on candidates. They pay some of the exam development and administrative costs through exam fees, and they pay travel and lodging costs to stay near exam sites. Most burdensome, candidates purchase expensive bar-preparation courses and forego income while devoting 10 or more weeks to intensive study. The exams’ focus on excessive memorization and undue speed requires those expenditures: research demonstrates that candidates who do not—or cannot—assume bar-prep costs are more likely to fail the exam.
The bar exam’s expense disadvantages test-takers with limited financial means, as well as those with caretaking responsibilities. Educational loans, notably, do not cover bar-prep courses, exam fees, travel expenses, or the cost of living while studying for the exam. Unequal resources explain at least part of the race and gender gaps that characterize bar results. Bar exams sharply favor white test-takers over examinees of color and offer some advantage to male candidates. Jurisdictions rightly worry about this impediment to diversifying the profession.
Many candidates who live with disabilities also struggle with bar exam. Jurisdictions require them to submit expensive documentation to secure accommodations and often refuse accommodations that candidates have relied upon during law school. The fact that the exam is offered just twice a year imposes additional obstacles for candidates with chronic conditions, those with illnesses, and those who are pregnant or breastfeeding.
States, finally, set different passing scores for their exams—even though most administer the same Uniform Bar Examination. Research shows that higher passing scores dramatically reduce the diversity of lawyers in states with those scores, without increasing the competence of new lawyers. Economic advantage and freedom from caretaking exacerbate this unfairness: candidates who fail in one jurisdiction, but have sufficient financial resources and personal mobility to relocate, can move to a state where their score will allow them to practice law.
In light of these flaws, several jurisdictions are exploring or adopting new ways to license lawyers. Starting in May, candidates for admission to the Oregon State Bar will be able to choose between taking the traditional bar exam and demonstrating their competence through a Supervised Practice Portfolio Examination (for more on this, see “Speaker’s Corner.”) Candidates who choose the latter option will obtain provisional licenses, allowing them to practice under supervision. During their supervisory period they will compile portfolios consisting of written work product, assessments of client encounters, and appraisals of negotiations. Bar examiners will review redacted versions of this work to determine whether a candidate is competent to practice law.
The California State Bar has recommended that the California Supreme Court establish a pilot program for a similar licensing option. Task forces in several other states are exploring ways other than the traditional bar exam to measure lawyering competence. Updated information about those innovations, including licensing paths based upon experiential work in law school, is available from a website we maintain with other scholars.
Will [supervised-practice pathways] measure the skills and knowledge that new lawyers need?
Some lawyers wonder whether supervised-practice pathways will be valid, feasible, and fair. Will they measure the skills and knowledge that new lawyers need? Will practicing lawyers be willing to serve as supervisors? Will these programs generate new forms of bias and unfairness for applicants, which has been a problem in countries using articling and apprenticeships?
Supervised-practice pathways differ significantly from articling and other forms of apprenticeship. The pathways offer an option, not a mandate, which reduces both the demand for supervisors and the power that supervisors hold over candidates. Independent examiners provide a further check on supervisors’ power. Substantial research in medical workplaces, meanwhile, shows that it is possible to evaluate licensees fairly and rigorously as they work with patients; those studies offer detailed models for assessing lawyers. And we already have evidence, drawn from a study of New Hampshire-’s Daniel Webster program, that portfolio-licensed lawyers can outperform peers who pass a traditional bar exam. Our research adds to this evidence base.
The current study
We obtained access to a unique data set offering a first look at the validity, feasibility, and fairness of licensing systems that, like the one adopted in Oregon, assess candidates’ competence while they engage in postgraduate supervised practice. The data derives from participants in two California programs. The “Original Program” allows pandemic-era graduates to practice under supervision while waiting to pass the bar exam. The “Pathway Program” allows some individuals who narrowly failed the bar exam to become fully licensed after 300 hours of supervised practice. While working under supervision, candidates in both programs use provisional licenses that allow them to do most types of legal work.
Neither program completely parallels the licensing pathways that Oregon recently adopted and other states are considering. Candidates in the Original Program must still take the bar exam to gain full licenses, while those in the Pathway Program do not submit portfolios of work product to examiners for review. The programs, however, offer important insights into the competencies that can be assessed during supervised practice, as well as the feasibility and fairness of such programs.
In fall 2022, the California State Bar surveyed candidates and supervisors in both programs, as well as candidates who were eligible for the Pathway Program but did not participate. Those surveys generated the data we analyze in this article. The survey responses offer strong evidence that practicing lawyers are willing to supervise candidates for bar admission; that workplaces have sufficient training and mentoring systems in place to support those programs; that candidates find supervised-practice placements fair; that supervised practice assesses a greater range of knowledge and skills than the bar exam; and that innovative licensing paths expand client service—including to clients who currently lack access to justice.
The California State Bar surveyed three populations: (1) 1,585 candidates who had received provisional licenses, (2) 1,393 supervisors of those candidates, and (3) 1,154 candidates who were eligible for the state’s Pathway Program but did not participate. Survey links were emailed to all population members in October 2022.
Response rates exceeded survey standards for all three groups: 47.8 percent for candidates holding provisional licenses, 32.0 percent for supervisors, and 47.2 percent for individuals who were eligible for the Pathway Program but did not participate. The California State Bar collected all responses in a deidentified database that was given to us for analysis. State bar staff did not participate in our analyses, and the views in this article do not reflect the perspectives of the state bar.
Benefitting employers: Feasibility
Supervised-practice licensing systems are feasible only if practicing lawyers are willing to supervise candidates working toward their licenses. In California, almost 1,400 licensed lawyers stepped forward to supervise candidates with provisional licenses. That number of volunteers is particularly impressive because the program was new, and the state bar offered no incentives or support for participation. Survey responses, moreover, suggest that many of these volunteers are willing to make an ongoing commitment. More than two-thirds of supervisors (70.6 percent) indicated that they were willing to continue supervising candidates, and another 16.5 percent were open to that possibility.
Survey respondents backed up this commitment by pointing to many benefits of working with candidates holding provisional licenses. Supervisors were satisfied with their candidates’ work and thought their candidates were especially hardworking. This competence and work ethic allowed organizations to serve more clients: almost nine-tenths of supervisors (86.8 percent) reported expanding their client base with the help of supervised-practice candidates.
Supervisors also applauded the diversity that candidates brought to their practice teams. The candidates were more demographically diverse than recently licensed lawyers, and some possessed unusual life experiences or training. That diversity allowed organizations to enhance service to existing clients, tap new client bases, and even explore new practice areas.
“The work product of the [candidate] was superior to other ‘full’ attorneys because he would put more time and effort into preparing his cases.” —Private Practitioner
“[Candidates with provisional licenses] allowed us to provide pro bono full representation to clients, whereas we would have only had the resources to provide them with advice.” —Legal Aid Supervisor
“[Our candidate] increased the diversity of our firm’s attorney staffing, which was badly needed. She also bridges our firm to new client groups . . . and is a notable client referral source. Our firm is monetarily better off, and her underserved community has greater access to much needed legal referrals.” —Law Firm Partner
“Being able to offer guidance and support to my [candidate] was personally and professionally rewarding and just made me feel good to be able to share knowledge and help her grow professionally.” —Solo Practitioner
California’s provisional licensing programs, finally, complemented employers’ hiring practices. The programs were attractive to candidates, which gave organizations an advantage when recruiting in a tight labor market. At the same time, the programs allowed organizations to give candidates a trial run before committing to a full-time position. During that trial period, candidates did not need time off to study for the bar exam; they could devote all their attention to learning the employer’s practices and serving clients.
Most organizations already provided supervision and training to newly licensed lawyers; they were able to use that infrastructure to provide oversight and feedback for candidates with provisional licenses.
Supervisors reported obtaining these benefits with few costs. More than two-thirds of supervisors experienced no problem “at all” from candidates making mistakes, and just 4.4 percent reported that mistakes concerned them to a “great extent.” Similarly, few employers found the supervision and training of candidates burdensome. Most organizations already provided supervision and training to newly licensed lawyers; they were able to use that infrastructure to provide oversight and feedback for candidates with provisional licenses. Many supervisors shared the sentiment of a government lawyer who commented: “This program is rare in that I cannot identify one downside as it was administered in my office.”
Benefiting candidates: Fairness
Some stakeholders worry that bias and old boy networks could disadvantage women and people of color in finding supervisors or succeeding in supervised-practice pathways. Just the opposite was true in California’s programs. Women and people of color were significantly more likely than white men to participate in the Pathway Program, and they were slightly more successful than white men in completing the program and earning full licenses.
First-generation college graduates, individuals living with disabilities, and individuals who identified as LGBTQIA+ also succeeded as candidates. We found no significant difference in success rates for these groups compared with other candidates. Nor did satisfaction ratings differ by race/ethnicity, gender, first-generation status, disability, or sexual orientation. Members of all groups expressed very high degrees of satisfaction with supervised practice. Indeed, respondents from historically disadvantaged groups offered eloquent comments about the importance of the supervised-practice program to them, their families, and their professional careers. Some supervisors volunteered similar observations.
More inclusive licensing
“I am a first generation BIPOC law student with a disability. As a single parent . . . I do not have the luxury of not earning money for months while I study for the bar.” —Candidate
“My [candidate] was exceptionally qualified and was having trouble passing the bar because her first language was not English. She was better than at least 50 percent of attorneys practicing who have passed the bar.” —Law Firm Partner
“The alternative pathway to licensure substantially improved almost every aspect of life. It allowed me to rediscover a sense of purpose and dignity, and to gain additional skills, knowledge and insights that could not have otherwise been achieved without this opportunity.” —Candidate Living with a Disability
“I know of two women of color from low-income backgrounds who qualified and succeed[ed] under the program. Our bar will benefit from their admission. Both had given up and moved on and otherwise would have left the profession.” —Nonprofit Director
Some candidates (9.7 percent) reported experiencing harassment or discrimination while engaged in supervised practice, but most of them characterized the challenges from this negative treatment as “small” or “moderate.” More important, candidates who reported discrimination or harassment were just as likely as other candidates to succeed in the programs. They also reported the same levels of satisfaction as candidates who did not report any negative treatment. Some candidates noted that the discrimination or harassment they experienced in the programs was no greater than what they endured in other contexts and that the programs, on balance, “countered” discrimination by allowing them to establish their competence and serve clients.
Stakeholders also worry that supervised-practice programs could force candidates to work without pay or accept low-paying positions. Almost all (93.6 percent) of the candidates in California’s Original Program, however, received compensation. Two-fifths of supervisors (39.4 percent) reported paying those candidates the same salaries as they paid newly licensed lawyers; 37.4 percent paid the same wages given to unlicensed law graduates; 14.4 percent paid paralegal wages; and 8.4 percent paid law student rates. These numbers suggest that programs allowing candidates to substitute supervised practice for the written bar exam may benefit both candidates and employers financially. Candidates will save the heavy expenses of bar preparation (including 10 weeks of foregone income), while employers have the option of modestly discounting entry-level salaries during the licensing period. Jurisdictions, meanwhile, can ensure that employers don’t discount salaries too heavily by requiring that employers pay candidates at least as much as they pay other law graduates who have not yet passed the bar exam.
Benefiting clients: A more valid measure of competence
The California data confirms that supervised-practice programs can assess more knowledge and skills than the traditional bar exam. More than three-quarters of candidates reported that they used key skills that are not tested on the bar exam. The prevalence of these skills in the first months of law practice demonstrates both the need to assess candidates’ competence in performing them and the possibility of assessing them during a period of supervised practice.
Percentage of licensees exercising skills
Jurisdictions can readily incorporate each of these skills into their portfolio requirements. For the small number of candidates who are not able to exercise these skills in their supervised-practice settings, jurisdictions can follow Oregon’s lead and create simulations for candidates to use in developing and demonstrating missing skills. While the traditional bar exam fails to test essential lawyering skills like research, counseling, and client communication, supervised-practice pathways will protect clients by rigorously assessing these critical competencies.
The California data confirms that supervised-practice programs can assess more knowledge and skills than the traditional bar exam.
California’s survey responses offer similar reassurance about the scope of doctrinal knowledge that can be assessed through supervised practice. Candidates reported drawing upon an average of 5.5 doctrinal areas in their practice, with a quarter of them listing eight or more subject areas. Even candidates who focused on a particular practice area, such as criminal law or personal injury work, drew upon concepts from a range of subjects.
Candidates did not work in every subject area that they might pursue as lawyers, so a licensing system based on supervised practice could not assess their knowledge in all fields. This, however, is also true of traditional bar exams. Those exams assess knowledge in just eight to 12 doctrinal areas, with a heavy emphasis on common-law rules. The California data demonstrates just how narrow the bar exam is: almost nine-tenths of candidates reported using knowledge from subjects that are not tested on the bar exam, and more than a fifth reported practicing in four or more subjects that do not appear on the bar exam.
“Our [candidate],” one law firm partner wrote, “has been the best ‘associate’ that we have had at our firm, better than associates that have passed the bar exam.”
Supervised practice, in other words, may be more effective than the bar exam at testing candidates’ knowledge in areas of contemporary practice. Those areas include critical fields like immigration, intellectual property, arbitration, tax, employment law, and dozens of other areas that are not tested on the bar exam. Licensing systems rooted in supervised practice, moreover, will adapt naturally to changes of focus in entry-level practice.
What about candidates who work in one field during supervised practice and then, after receiving a full license, switch to another practice area? Will those lawyers endanger clients? Once again, the same question applies to lawyers licensed through the bar exam. They are allowed to practice environmental law, international tax, admiralty, and any other subject. Supervised-practice pathways, like the bar exam, assume that professionals who have demonstrated their competence in some areas of practice are able to transfer their skills to other areas and learn the doctrinal rules of those new subjects.
The California survey did not ask supervisors to compare candidates to attorneys who had passed the bar exam, but some did so spontaneously. “Our [candidate],” one law firm partner wrote, “has been the best ‘associate’ that we have had at our firm, better than associates that have passed the bar exam.” Other supervisors noted that candidates gained “on the job training,” experience “dealing with clients,” and “more applicable knowledge” than peers who had passed the bar exam.
“By participating in the actual practice of law, rather than memorization techniques for three months as with the current Bar Exam setup, these new attorneys learn more, focus on what is expected of them in the profession, and can hit the ground running faster when licensed as compared to those who have just passed a test.” —Law Firm Partner
“The ability to issue spot with a live person by asking the right questions and having the right ‘bedside manner’ is quite different from picking apart a written set of facts.” —Candidate
“Much of what we do is not even covered on the bar exam. In my experience, competence on the job is a greater predictor of success than bar passage.” —Legal Aid Supervisor
“The practical application of the law is what truly changes your view and understanding of the complexity of the law itself.” —Candidate
Finally, our research suggests that supervised-practice licensing paths can expand access to justice. In the California programs, more than a fifth (22.2 percent) of licensees worked for legal aid, public defenders, or other public-interest employers. They were able to represent disadvantaged clients immediately, without waiting to take and pass the bar exam. As one provisional licensee noted, “I am working as a first-generation lawyer in a legal aid office. I have already put in 3,000 hours of direct community lawyering in a domestic violence clinic and doing housing rights advocacy.” Public-interest employers expressed similar enthusiasm for supervised practice. “The provisional licensees I supervised were highly competent legal advocates,” a legal aid attorney wrote, “passionate about serving low-income clients, and expanded the availability of legal services in underserved portions of rural California.”
Building on the data
Data from California’s supervised-practice programs suggests that these programs offer a promising pathway for licensing lawyers. Practicing lawyers are willing to supervise candidates, and their organizations benefit from the candidates’ work. Supervised-practice paths lower expenses for law graduates and open the profession to candidates who can more readily demonstrate their competence in practice than on a traditional exam. Most important, supervised-practice paths allow jurisdictions to protect clients by using a more comprehensive and rigorous assessment process that encompasses a fuller range of skills and contemporary doctrine than the bar exam measures.
California’s current programs, though, are just the beginning. Jurisdictions can build on those programs—as Oregon has done—to increase their feasibility, fairness, and validity. Law schools and bar associations can help connect candidates with willing supervisors. Bar associations might offer training to supervisors who want to enhance their supervisory skills. Regulations can prevent economic exploitation of candidates, and an ombudsperson can aid candidates who encounter problematic treatment. Lastly, candidates can compile portfolios of redacted work product that they submit to bar examiners for review. Measures like these will develop a rigorous, inclusive licensing system that benefits employers, candidates, clients, and the public.
A team of highly regarded psychometricians once wrote: “The time-honored way to find out whether a person can perform a task is to have the person try to perform the task.” State courts and bar examiners are starting to heed that advice, seeking more authentic ways to assess the competence of aspiring lawyers. Our research shows that licensing systems rooted in supervised practice offer a valid, feasible, and fair avenue for measuring the knowledge and skills that lawyers need to serve contemporary clients effectively. As jurisdictions explore those options, we hope that they will add to this research base. Research can point the way to licensing systems that increase both the competence and inclusiveness of our profession.
Questions for further research
The data from California supports further exploration of licensing pathways rooted in postgraduate supervised practice. As jurisdictions develop those programs, additional research should address questions like these:
- How can jurisdictions create reliable systems for portfolio review?
- How will the costs of reviewing portfolios compare with the costs of preparing, administering, and grading exams?
- How does the performance of lawyers licensed through supervised practice compare with the performance of those who pass the traditional exam?
- How does the inclusiveness and equity of supervised-practice pathways compare with the traditional bar exam?
Deborah Jones Merritt is Distinguished University Professor and John Deaver Drinko-Baker & Hostetler Chair in Law Emerita at Moritz College of Law, the Ohio State University.
Andrea Anne Curcio is Professor of Law at Georgia State University College of Law.
Eileen Kaufman is Professor of Law at Touro Law Center.