The difference between 0.70 and 0.71 macro-F1 on a sample of 145 real documents demonstrates that the Qwen2.5-1.5B model extracts legal entities on par with Ministral 3 14B. For an infrastructure budget, this eliminates the need for continuous expansion of expensive accelerator pools. On an A5000, the compact model parses a document 2.2 times faster than Ministral on an A100.
Transitioning to compact specialized models cuts hardware expenditures without compromising business logic quality. In the legal tech segment, this objective encounters rigid constraints: lawyers cannot transmit case materials to external cloud APIs. At stake is the allocation of scarce GPU memory between the client conversational interface and background entity extraction from incoming data arrays. Which local hardware configuration enables the processing of thousands of documents without interrupting client services?
Attorney-client privilege and memory shortages
In arbitration practice, a single case frequently spans thousands of pages, requiring the structuring of facts, dates, and financial claims.
Initially, Ministral 3 14B on a leased A100 board handled this task, spending an average of about 33 seconds per document. This same model served the client-facing chat. The issue arose when the chat model was switched to Gemma 4 28b: it consumes virtually the entire A100. Running simultaneous parses of large file batches created the risk of interrupting user servicing.
Companies cannot route computations to public cloud APIs due to regulatory and legal barriers.
Case materials for a lawyer represent attorney-client privilege, client trade secrets, and the personal data of dozens of individuals simultaneously.
As Sergei Zubkov, Vice President of the Federal Chamber of Lawyers, noted in an interview with RAPSI, the primary risk of deploying AI for attorneys remains data leakage. Local deployment (on-premise) remains the sole viable path, where every gigabyte of memory is critical.
Specialization over scaling
Primary skepticism regarding the shift to compact architectures centers on the risk of accuracy degradation on complex legal constructs. Entities were divided into nine classes: dispute parties (participant), document metadata (document), contracts (contract), dispute subject (object), performance events (performance_event), payment events (payment_event), breach events (breach_event), damages (damage), and correspondence (communication).
Each class is extracted via a separate invocation, generating nine questions per document. How did the engineers resolve the quality issue without leasing a second expensive A100 card?
The solution involved fine-tuning the open Qwen2.5-1.5B architecture with a LoRA adapter tailored to specific contracts and judicial acts. The dataset comprised 5,500 authentic documents from real cases obtained with client consent.
A typical document is short: the median is 2,380 characters, under one and a half pages, while over three thousand documents fit onto a single page.
Simultaneously, the longest 10% of documents account for 61% of the total sample text.
The pursuit of universal, heavy models for closed perimeters frequently collides with the straightforward mathematics of server leasing. Practice demonstrates that an A5000-tier solution outpaces a large system on an A100 by more than double while preserving data structuring accuracy.
For comparison
A difference of just 0.01 macro-F1 between a compact 1.5B model and a heavy 14B alternative clearly highlights the true cost of general-purpose architectures. While the industry attempts to cram massive setups into local servers to protect client confidentiality, targeted fine-tuning on 5,500 real cases delivers faster document parsing even on a modest A5000 accelerator. Paying for oversized models simply to extract structured facts turns out to be redundant when a specialized adapter matches their output without renting an extra A100. In the end, running massive models in on-premise legal pipelines yielded to a compact network fine-tuned directly on case files.
Our stories: Legal RAG Cost Inflation Risks 30% Savings (March 26); Qwen3-0.6B vs Llama-3.2-1B: Efficiency of Small LLMs (July 7); Local SLMs vs. Cloud AI: ROI and Efficiency for Enterprise (July 21); InternLM Drops 4B Model to Cut Enterprise AI Inference Costs (September 27).