LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text

Ben Hagag, Gil Gil Semo, Dor Bernsohn, Liav Harpaz, Pashootan Vaezipoor, Rohit Saha, Kyryl Truskovskyi, Gerasimos Spanakis


Abstract
This paper presents the results of the LegalLens Shared Task, focusing on detecting legal violations within text in the wild across two sub-tasks: LegalLens-NER for identifying legal violation entities and LegalLens-NLI for associating these violations with relevant legal contexts and affected individuals. Using an enhanced LegalLens dataset covering labor, privacy, and consumer protection domains, 38 teams participated in the task. Our analysis reveals that while a mix of approaches was used, the top-performing teams in both tasks consistently relied on fine-tuning pre-trained language models, outperforming legal-specific models and few-shot methods. The top-performing team achieved a 7.11% improvement in NER over the baseline, while NLI saw a more marginal improvement of 5.7%. Despite these gains, the complexity of legal texts leaves room for further advancements.
Anthology ID:
2024.nllp-1.33
Volume:
Proceedings of the Natural Legal Language Processing Workshop 2024
Month:
November
Year:
2024
Address:
Miami, FL, USA
Editors:
Nikolaos Aletras, Ilias Chalkidis, Leslie Barrett, Cătălina Goanță, Daniel Preoțiuc-Pietro, Gerasimos Spanakis
Venue:
NLLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
361–370
Language:
URL:
https://aclanthology.org/2024.nllp-1.33
DOI:
10.18653/v1/2024.nllp-1.33
Bibkey:
Cite (ACL):
Ben Hagag, Gil Gil Semo, Dor Bernsohn, Liav Harpaz, Pashootan Vaezipoor, Rohit Saha, Kyryl Truskovskyi, and Gerasimos Spanakis. 2024. LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text. In Proceedings of the Natural Legal Language Processing Workshop 2024, pages 361–370, Miami, FL, USA. Association for Computational Linguistics.
Cite (Informal):
LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text (Hagag et al., NLLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.nllp-1.33.pdf