A comparative analysis of the performance of large, older-generation language models in solving legal problems of varying complexity

This article presents a comparative analysis of the performance of seven major language models (Perplexity Sonar, Claude 4.0 Sonnet, OpenAI GPT-4.1, Gemini 2.5 Pro, Grok 3, DeepSeek v3, and Qwen3-235B-A22B) in solving 25 legal problems of five difficulty levels, developed based on the Family and Civil Codes of the Russian Federation. An automated system based on Claude 4.0 Sonnet was used to evaluate the quality of the answers, serving as an "examiner" and assigning scores on a ten-point scale with brief explanations. The main metrics of the experiment were the mean score, total token consumption (Token Usage), the economic cost of running all questions (Cost per Experiment), and the efficiency ratio (quality to cost ratio). A comparative analysis of monolithic models revealed that GPT-4.1 and Gemini 2.5 Pro lead in average performance, particularly on simple and conflict-based tasks, while the average level of complexity (a combination of norms) remained the most challenging for all models. Economic calculations confirmed that when scaling legal AI systems, it is critical to consider the balance between speed, accuracy, and generation cost. The results of the study allow for the development of practical recommendations for selecting architectures and models for corporate and government applications in legal consulting.