Headlines claiming that language models diagnose patients better than doctors usually fall apart under harsh methodological testing. In practice, models are often asked to choose from a closed list of options, whereas a real therapeutic appointment works differently: a doctor formulates their conclusion in free form. To bring evaluation closer to actual practice, researchers from the "Strong AI in Medicine" laboratory at the Artificial Intelligence Institute AIRI, alongside specialists from the V. A. Almazov National Medical Research Centre, have created a quality control framework for AI diagnostics and published their findings in Scientific Reports.
This development builds on the experience of creating AIRI's "Health Assistant" app, which was previously named the best AI solution in customer service at the Generation AI Awards 2025. While developing the project, the authors encountered a fact that is obvious in clinical practice: standard academic tests do not reflect the real clinical environment in any way.
Checking free-form wording
For testing, the authors gathered a dataset of 360 samples: 180 text dialogues between real doctors and patients, and another 180 dialogues between a human and a language model acting as a doctor. Seven general practitioners from the Almazov Centre independently wrote up to three possible diagnoses based on the collected anamnesis for each case.
Then, the same dialogues were passed to language models, including DeepSeek-V3, GigaChat-Max, GPT-4o, Mistral-Large, Llama-405B, and Qwen-72B. The models also generated lists of up to three diagnoses. The main challenge lay in comparing the formulations: for instance, "acute pancreatitis" and "chronic pancreatitis, exacerbation" formally represent different text strings, but clinically they can point to a similar condition.
Match evaluation and distillation
To objectively compare free medical formulations, the authors compiled 6,500 pairs of diagnoses, where three medical experts independently evaluated each option for semantic content match.
"To achieve this, we gathered 6,500 pairs of diagnoses, with three experts independently annotating each variant for a match or mismatch."
Based on the annotated data, the researchers built a meta-model. Its architecture incorporated RAG technology with an index of 10,000 entries from the ICD-10 international classifier, vector embeddings from three different models, and a set of linguistic metrics. On a test set of nearly 1,500 pairs, the meta-model demonstrated high accuracy. Subsequent distillation of the pipeline using the ModernBert architecture preserved its reliability.
For businesses in the HealthTech sector, this means a transition from blind faith in vendor marketing promises to measurable and verifiable validation metrics. The created framework gives companies a tool that reduces legal and financial risks when purchasing software, although its large-scale deployment will require testing across a broader range of narrow-profile nosologies.