Standard semantic RAG collapses when applied to corporate financial reconciliations, delivering an exact 16-class root-cause analysis accuracy of just 2.05%. A new benchmark developed by researcher Pratik Ghawate, FinRCA-Bench, demonstrates why dropping generic language models into back-office audits is a recipe for silent failure: financial entities are bound by transactional logic and ledger relationships, not surface textual similarity.
Evaluating 2,250 accounts-payable-to-bank reconciliation cases across 14 operational tables, the study isolates retrieval failures from reasoning failures. Frozen dense semantic retrievers hit an abysmal macro required-record recall of 0.83%. Switching to Typed Provenance Graph Retrieval (TPGR)—a default-deny typed traversal following persisted transaction relationships—pushed accuracy to 72.44% and recall to 77.70%, all while trimming source tokens from 5,531 down to 2,370. Even so, classical machine learning and deterministic Rules/SQL baselines still outperformed the LLM setup, clocking 95.44% and 84.97% accuracy respectively.
The benchmark shows that structural data-retrieval breakdowns outnumber pure logical reasoning errors 95 to 15. More critically for CFOs and technical leaders, the findings warn against blind enterprise automation: 254 correct predictions occurred despite incomplete evidence retrieval, while strict returned-evidence contract accuracy collapsed to 5.72%. In practice, models frequently guess the right accounting diagnosis while hallucinating or omitting the auditable transaction trail needed to prove it.