Passing a USA national bar exam - A first experiment

Biralatei Fawei, Adam Wyner, Jeff Z. Pan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running ‘out of the box’ the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.
Original languageEnglish
Title of host publicationLegal Knowledge and Information Systems, JURIX 2015: 28th Annual Conference
PublisherIOS Press
Pages179-180
Number of pages2
Volume279
ISBN (Electronic)9781614996088
DOIs
Publication statusPublished - 2015
Event28th Annual International Conference on Legal Knowledge and Information Systems, JURIX 2015 - Braga, Portugal
Duration: 10 Dec 201511 Dec 2015

Publication series

NameFrontiers in Artificial Intelligence and Applications
Volume279
ISSN (Print)09226389

Conference

Conference28th Annual International Conference on Legal Knowledge and Information Systems, JURIX 2015
CountryPortugal
CityBraga
Period10/12/1511/12/15

Fingerprint

Experiments
Watersheds
Information systems
Students
Industry

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

Fawei, B., Wyner, A., & Pan, J. Z. (2015). Passing a USA national bar exam - A first experiment. In Legal Knowledge and Information Systems, JURIX 2015: 28th Annual Conference (Vol. 279, pp. 179-180). (Frontiers in Artificial Intelligence and Applications; Vol. 279). IOS Press. https://doi.org/10.3233/978-1-61499-609-5-179

Passing a USA national bar exam - A first experiment. / Fawei, Biralatei; Wyner, Adam; Pan, Jeff Z.

Legal Knowledge and Information Systems, JURIX 2015: 28th Annual Conference. Vol. 279 IOS Press, 2015. p. 179-180 (Frontiers in Artificial Intelligence and Applications; Vol. 279).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Fawei, B, Wyner, A & Pan, JZ 2015, Passing a USA national bar exam - A first experiment. in Legal Knowledge and Information Systems, JURIX 2015: 28th Annual Conference. vol. 279, Frontiers in Artificial Intelligence and Applications, vol. 279, IOS Press, pp. 179-180, 28th Annual International Conference on Legal Knowledge and Information Systems, JURIX 2015, Braga, Portugal, 10/12/15. https://doi.org/10.3233/978-1-61499-609-5-179
Fawei B, Wyner A, Pan JZ. Passing a USA national bar exam - A first experiment. In Legal Knowledge and Information Systems, JURIX 2015: 28th Annual Conference. Vol. 279. IOS Press. 2015. p. 179-180. (Frontiers in Artificial Intelligence and Applications). https://doi.org/10.3233/978-1-61499-609-5-179
Fawei, Biralatei ; Wyner, Adam ; Pan, Jeff Z. / Passing a USA national bar exam - A first experiment. Legal Knowledge and Information Systems, JURIX 2015: 28th Annual Conference. Vol. 279 IOS Press, 2015. pp. 179-180 (Frontiers in Artificial Intelligence and Applications).
@inproceedings{dad89adb7d8546418712946a102394b0,
title = "Passing a USA national bar exam - A first experiment",
abstract = "Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running ‘out of the box’ the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.",
author = "Biralatei Fawei and Adam Wyner and Pan, {Jeff Z.}",
note = "The authors appreciate support from the NCBE and the Niger Delta University.",
year = "2015",
doi = "10.3233/978-1-61499-609-5-179",
language = "English",
volume = "279",
series = "Frontiers in Artificial Intelligence and Applications",
publisher = "IOS Press",
pages = "179--180",
booktitle = "Legal Knowledge and Information Systems, JURIX 2015: 28th Annual Conference",

}

TY - GEN

T1 - Passing a USA national bar exam - A first experiment

AU - Fawei, Biralatei

AU - Wyner, Adam

AU - Pan, Jeff Z.

N1 - The authors appreciate support from the NCBE and the Niger Delta University.

PY - 2015

Y1 - 2015

N2 - Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running ‘out of the box’ the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.

AB - Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running ‘out of the box’ the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.

UR - http://www.scopus.com/inward/record.url?scp=84963705943&partnerID=8YFLogxK

U2 - 10.3233/978-1-61499-609-5-179

DO - 10.3233/978-1-61499-609-5-179

M3 - Conference contribution

VL - 279

T3 - Frontiers in Artificial Intelligence and Applications

SP - 179

EP - 180

BT - Legal Knowledge and Information Systems, JURIX 2015: 28th Annual Conference

PB - IOS Press

ER -