Evolution of the capabilities of large language models in the legal field: Meta-analysis of four experimental studies

封面

如何引用文章

全文:

开放存取 开放存取
受限制的访问 ##reader.subscriptionAccessGranted##
受限制的访问 订阅或者付费存取

详细

This paper presents a meta-analysis of four experimental studies from the Norm! project, aimed at systematically studying the effectiveness of large language models in the legal field. The study includes a comparative analysis of junior and senior models, optimization of system prompts, and testing of multi-agent architectures on tasks in Russian family and civil law. A key discovery was the identification of a nonlinear relationship between architectural complexity and the quality of results: the transition from simple to complex systems provides a slight increase in quality (15–40%) with an exponential increase in resource costs (by a factor of 10–15). The flagship models GPT-4.1 and Gemini 2.5 Pro demonstrate superior quality (9.04 and 8.52 points), but junior LLMs with efficiency coefficients up to 130.3 remain cost-effective. A universal problem area for all architectures is tasks requiring an integrative analysis of multiple legal norms. The results form scientifically sound recommendations for various implementation scenarios: from mass consulting services to specialized legal applications, defining the prospects for the development of hybrid architectures in legal practice.

全文:

受限制的访问

作者简介

Roman Dushkin

National Research Nuclear University “MEPhI”

编辑信件的主要联系方式.
Email: drv@aia.expert
ORCID iD: 0000-0003-4789-0736
SPIN 代码: 1371-0337

senior lecturer, Department 22 “Cybernetics”

俄罗斯联邦, Moscow

Vladimir Podoprigora

Plekhanov Russian University of Economics

Email: Podoprigora.VN@rea.ru
ORCID iD: 0000-0001-6485-8135
SPIN 代码: 9587-1028

Cand. Sci. (Econ.), head of the laboratory

俄罗斯联邦, Moscow

Alexey Kuzmin

Ecosystem Digital Solutions LLC

Email: a.kuzmin@edisai.tech

general director

俄罗斯联邦, Moscow

Kirill Dushkin

A-Ya expert LLC

Email: dkr@aia.expert

analyst

俄罗斯联邦, Moscow

参考

  1. Dushkin R.V. Artificial Intelligence. Moscow: DMK-Press, 2019. 280 p. ISBN: 978-5-97060-787-9.
  2. Lai J., Gan W., Wu J. et al. Large language models in law: A survey. AI Open. 2024. URL: https://www.sciencedirect.com/science/article/pii/S2666651024000172 (data of accesses: 13.10.2023).
  3. Ma S., Chen C., Chu Q. et al. Leveraging large language models for relevance judgments in legal case retrieval. arXiv preprint arXiv:2403.18405. 2024. URL: https://arxiv.org/abs/2403.18405 (data of accesses: 13.10.2023).
  4. Paul V. Automation in legal: The increasing role of AI. Medium. 2024. URL: https://medium.com/@vincentpaulai/automation-in-legal-the-increasing-role-of-ai-70724ef0b225 (data of accesses: 13.10.2023).
  5. Magesh V., Surani F., Dahl M. et al. Hallucination-free? Assessing the reliability of leading AI legal research tools. arXiv preprint arXiv:2405.20362. 2024. URL: https://arxiv.org/abs/2405.20362 (data of accesses: 13.10.2023).
  6. The future of artificial intelligence in the legal industry: Opportunities, challenges, and ethical considerations. Legal Stuff. URL: https://medium.com/@legal.stuff.notion/the-future-of-artificial-intelligence-in-the-legal-industry-opportunities-challenges-and-ethical-61c3198b425a (data of accesses: 13.10.2023).
  7. Korneenkov A.A., Yanov Yu.K., Ryazantsev S.V. et al. Meta-analysis of clinical studies in otorhinolaryngology. Bulletin of Otorhinolaryngology. 2020. Vol. 85. No. 2. Pp. 26–30. (In Rus.). doi: 10.17116/otorino20208502126.
  8. Dushkin R.V. Overview of approaches and methods of artificial intelligence. Radio Electronic Technologies. 2018. No. 3. Pp. 85–89. (In Rus.)
  9. Rumyantsev P.O., Saenko U.V., Rumyantseva U.V. Statistical methods of analysis in clinical practice. Part 1. Univariate statistical analysis. Problems of Endocrinology. 2009. Vol. 55. No. 5. Pp. 48–55. (In Rus.). doi: 10.14341/probl200955548-55.
  10. Dushkin R.V. Why hybrid AI systems hold the future. Economic Strategies. 2018. No. 6 (156). Pp. 84–93. (In Rus.)

补充文件

附件文件
动作
1. JATS XML
下载
2. Fig. 1. Stages of conducting meta-analysis of LLMs in the legal domain

下载 (299KB)
索引源数据
3. Fig. 2. Comparative analysis of quality and economic efficiency of lesser LLMs: a – quality of responses of lesser LLMs in the legal domain; b – economic efficiency of lesser LLMs

下载 (223KB)
索引源数据
4. Fig. 3. Comparative analysis of effectiveness of various system prompts for GPT-4o mini: a – quality of responses of various system prompts; b – economic efficiency of prompts; c – token consumption by various agents Agent: 1 – universal; 2 – specialized; 3 – modified; 4 – overtrained

下载 (273KB)
索引源数据
5. Fig. 4. Comparative analysis of greater LLMs performance by overall indicators and complexity levels: a – performance of greater LLMs in the legal domain; b – performance of top-3 LLMs by task complexity levels Level: 1 – simple; 2 – secondary; 3 – combination; 4 – collisions; 5 – problematic

下载 (312KB)
索引源数据
6. Fig. 5. Comparative analysis of MAS architectures by quality, resource consumption and efficiency a – quality of responses of different MAS architectures; b – token consumption by MAS architectures; c – economic efficiency of MAS architectures Variant: 1 – simple; 2 – with dispatcher; 3 – modified; 4 – ensemble; 5 – with jury

下载 (263KB)
索引源数据

版权所有 © Yur-VAK, 2025

许可 URL: https://www.urvak.ru/contacts/