Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running `out of the box' the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.
|Title of host publication||LREC 2016, Tenth International Conference on Language Resources and Evaluation|
|Editors||Nicoletta Calzolari , Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis|
|Number of pages||7|
|Publication status||Published - 23 May 2016|
|Event||Tenth International Conference on Language Resources and Evaluation - Grand Hotel Bernardin Conference Center, Portorož, Slovenia|
Duration: 23 May 2016 → 28 May 2016
|Conference||Tenth International Conference on Language Resources and Evaluation|
|Period||23/05/16 → 28/05/16|
- textual entailment
- bar exam
- legal reasoning
- natural language processing
FingerprintDive into the research topics of 'Passing a USA National Bar Exam: a First Corpus for Experimentation'. Together they form a unique fingerprint.
Passing a USA National Bar Exam: a First Corpus for Experimentation: Article Data
Wyner, A. Z. (Creator), Fawei, B. J. (Creator), Pan, J. Z. (Creator) & Edwards, P. (Data Manager), University of Aberdeen, 31 Mar 2016