Neuro-symbolic artificial intelligence has been widely pitched as the bridge between statistical deep learning and formal, logic-based reasoning. As academic output across this subdomain accelerates, enterprise engineering teams routinely scour preprints and conference proceedings to import hybrid algorithmic architectures into production. However, treating published computational papers as deployable implementations represents an expensive operational mistake.
The Six-Stage Pipeline Audit
A comprehensive audit led by Brandon Colelough and a research team at the University of Maryland's Department of Computer Science evaluated whether independent engineers could realistically execute the codebases of published neuro-symbolic papers and recover their reported benchmarks. The team established a rigorous six-stage verification framework across the literature.
Stage one gathered 5,497 initial records, filtering out 3,018 duplicates. Stage two screened the remaining 2,479 unique records across titles and abstracts, identifying 1,365 self-described neuro-symbolic AI papers, before pruning 61 entries due to out-of-scope topics, missing quantitative evaluation, non-research formatting, or paywalled text. Stage three searched for verifiable public code artifacts across the 1,304 eligible papers, revealing that 849 studies published zero code.
Only 455 papers provided code repositories capable of advancing to stages four and five for artifact inventory and bounded test reruns. Across the entire eligible corpus of 1,304 papers, the researchers successfully reproduced—fully or partially—only 85 studies. That yields an actual replication rate of just 6.52% overall, and a dismal 18.68% among the 455 papers that actually supplied code.
Failure Points in Released Artifacts
These execution failures rarely stem from abstract mathematical errors; they collapse under basic software hygiene. Among failed reruns, the Maryland team found that 321 attempts stalled entirely due to missing non-code dependencies—such as unreleased model weights, proprietary configurations, and raw datasets—while another 42 failed because repositories were fundamentally broken or abandoned.
Even explicit declarations that code is 'publicly available' frequently point to dead links or partial scripts that cannot construct an end-to-end execution pipeline. For applied engineering teams, theoretical breakthroughs in whitepapers cannot be treated as production blueprints. Without factoring in the engineering overhead of ground-up reimplementation, enterprise teams risk sinking substantial R&D budgets into algorithmic concepts that cannot be independently validated.