Relying on public language model rankings in legal consulting leads to systematic case prediction errors and direct financial losses. In November, the author conducted a new evaluation of the Lexometrica Ground Truth benchmark, testing 15 models on 30 confidential cases based on the practice of the Supreme Court of the Russian Federation. The measurements demonstrated the practical uselessness of tournament tables: performance gaps between adjacent positions in general leaderboards are typically smaller than the measurement error, and the reliability of the judge model is almost never verified.
The core issue is that real decision-making accuracy cannot be evaluated on universal tasks. Applied automation requires verification on closed industry datasets using the IRAC scheme, which models legal reasoning: identifying the legal issue, finding the rule, applying it to the facts, and reaching a conclusion. The previous evaluation took place in March 2026, and the November update confirmed the persistence of hidden flaws in neural network reasoning.
Previously, on September 11, 2026, in the article "The Illusion of Logic in AI: Benchmark Exposes Model Reasoning Failures", we noted similar issues.
Anatomy of the Judicial Test
In the "neShemyaka!" service, the outcome of a legal dispute is predicted based on case materials using an extensive LLM pipeline. The models within it are swapped for flagship releases as they emerge, but the developers decided to validate their choice against their proprietary dataset twice a year.
Thirty cases; each consists of a fact pattern, a question, an expert rubric, and a reference answer. The cases follow the stages of the IRAC scheme, mirroring LegalBench and later sets for other legal systems. The third step in continental doctrine is subsumption—applying established facts to the hypothesis of a legal norm. This is the primary and most difficult part of a lawyer's work: a legal information system can help find the norm, but it will not determine whether it covers specific circumstances.
«Therefore, the application of the norm accounts for almost half of the test suite».
The core analytical workload of legal consulting is concentrated precisely at the norm-application stage. At the same time, the models lacked access to search engines and legal databases. The results apply strictly to the base models and cannot be directly extrapolated to RAG products.
Limits of Reasoning and Judicial Glitches
Public flagship models from fifteen product lines as of November 2026 entered the test loop: Claude Fable 5.2, Claude Opus 5.5, GPT-6.2 Astra, Gemini 4 Carbon, Grok 4.7, Qwen4 Max, DeepSeek V4 Pro, DeepSeek V4.1 Flash, Kimi K3.1 Pro, GLM-5.3, GLM-5.3 FlashX Legal, MiniMax M3, Mistral Large 4, GigaChat 3.5 Ultra, and YandexGPT 5.1 Pro. Reasoning mode was explicitly enabled for all models that support it. Temperature was set to 0 wherever controllable.
However, this smooth engineering protocol broke down on MiniMax M3: at zero temperature, its reasoning entered an infinite loop and produced no output, forcing a fallback to the vendor's default temperature. Another incident involved YandexGPT: on two cases, it returned a single-sentence refusal—"I cannot discuss this topic. Let's talk about something else."—despite the questions being standard legal queries. The model received zero scores for these refusals.
How can bias be eliminated when evaluating thousands of legal judgments under such conditions?
Each answer is evaluated by two LLM-as-a-Judge systems—Claude Opus 5.5 and Grok 4.7—on a scale from 0 to 1, based on the rubric and the reference answer. The judge receives a package for each case containing the fact pattern, question, rubric, reference answer, and all fifteen model answers under randomized letter identifiers. Self-identifying model names are stripped from the responses. The judge assigns a score to each answer with a brief justification. Evaluating all answers to a case within a single package was a deliberate choice: the judge can observe the variance and calibrate the scoring scale within the context of that specific case.
Market Illusions vs. Dry Statistics
There is a persistent belief that general leaderboards reliably reflect overall model computing power, rendering investments in closed niche benchmarks an unnecessary budgetary drain. The results of the November evaluation refute this position. Confidence intervals—derived via bootstrap over cases (10,000 iterations), with pairwise comparisons of neighbors via paired bootstrap—demonstrated that clear statistical differences across the market are sparse.
«Vendor rank correlation between evaluation runs is approximately 0.85: an entire model generation has turned over».
The March evaluation included ten models from the same vendors representing previous generations. The test suite maintained its structure, categories, and system prompt; the panel of judges differed in November. Consequently, the author primarily compares the ranking order and structure of the results, treating absolute score variances as broad indicators.
Purchasing subscriptions to the top tiers of global leaderboards does not guarantee error-free legal analysis for businesses. Without validation against closed judicial practice, deploying general-purpose LLMs into legal workflows remains an unjustifiably high-risk move.
What doesn’t add up
Costly errors in legal practice begin right where public model leaderboards promise flawless reasoning, yet the November run of the Lexometrica Ground Truth benchmark exposes systematic breakdowns in legal deduction. Testing fifteen flagship models against thirty closed cases of the Supreme Court of the Russian Federation proved that actual subsumption — applying facts to statutory rules — remains a stumble even with reasoning enabled. Impressive ranks on public scoreboards sell enterprise access easily, yet thirty practical courtroom disputes leave the engines either caught in infinite loops or refusing to talk.