Large enterprises process tens of thousands of documents every day: invoices, acts, packing slips, contracts, and project files. To automatically route them across departments, incoming files are vectorized and categorized by measuring distance to topic centroids. However, picking an architecture straight from public leaderboards crashes hard against production reality. In a benchmark comparing 13 vectorization pipelines for corporate email routing, the baseline standard was established by human domain experts who labeled 4,695 documents across 11 categories. On basic synthetic evaluations, giga_480m and qwen3e_4b predictably shared the top spot with a 76.8% accuracy score. But that was where the benchmark magic ended.
The boilerplate trap
The very first robustness audit exposed a glaring problem: standardized invoices and templates meant that out of 4,695 documents, only 1,162 were truly unique—a mere 25% of the entire sample. The default test harness excluded the specific document being evaluated, yet left its exact duplicates inside the reference database. As a result, the neural network was not extracting semantic meaning; it was simply matching visual twins.
"Text-based deduplication without embeddings—avoiding any algorithmic bias—revealed that only 1,162 out of 4,695 documents were genuinely unique: just 25%."
Purging duplicate records slashed accuracy across all tested models by 6 to 16 percentage points, dragging actual performance down to a 45–70% range. The celebrated leader qwen3e_4b tumbled to fourth place after losing its duplicate-driven crutch. Classic character-level TF-IDF scored 67.8% on the redundant dataset but plummeted to 50.7% on cleaned data, dropping 17 points.
Smarter classification rules shuffle the leaderboard
The second evaluation phase exposed the flaws of basic vector geometry. In production, the 11 high-level categories actually split into 95 internal sub-clusters, rendering a single averaged centroid mathematically useless. Instead of fine-tuning the underlying transformers, engineers added a lightweight trainable classification layer over the existing vectors. This straightforward fix boosted model accuracy by 7 to 34 percentage points. The former frontrunner, giga_480m, slid to seventh place, while lightweight symbolic methods took the lead without requiring capital-intensive GPU clusters.
A final audit invalidated the widely used Adjusted Rand Index (ARI) clustering metric. Variations caused by random seed initialization within the same model matched the entire performance spread across all tested architectures. Relying on ARI for production choices is fundamentally unreliable. Without human-verified datasets and rigorous real-world stress tests, businesses risk wasting millions on heavy neural networks where basic mathematics delivers superior ROI.