When a business needs to classify customer feedback at scale, deploying a heavy transformer has become the default knee-jerk reaction. Many engineers and managers view classical approaches like frequency-based TF-IDF as relics from the pre-BERT era—outdated tech nobody wants to admit running in modern production. Teams rarely benchmark this architectural choice against their own data, assuming upfront that a complex neural network will inevitably deliver superior results.
However, a direct engineering benchmark on a real-world dataset of 500,000 Russian business reviews from Yandex Maps tells the opposite story. The task was three-class sentiment analysis: negative, neutral, and positive. In real customer feedback, positive texts made up 86% of the volume, while neutral comments accounted for just 4%. With this degree of class imbalance, standard accuracy is deeply deceptive: a dummy model predicting only the majority class scores 86% accuracy while learning absolutely nothing.
The Standard Metric Trap
To ensure a fair evaluation, the test used macro-averaged F1, which weights every class equally regardless of its sample share. All experiments ran on a standard 12-core CPU without using expensive GPU accelerators.
A fine-tuned Russian transformer (cointegrated/rubert-tiny2 with 29 million parameters) without custom loss weighting produced a formal accuracy of 0.923, but its macro-F1 collapsed to 0.574.
The baseline transformer achieved high accuracy solely by ignoring the 4% of neutral reviews, delivering zero predictive quality on the hardest class in the dataset.
Skipping manual loss rebalancing turned the resource-heavy model into a corporate vanity metric—optimizing executive dashboards while failing on rare, mission-critical customer signals.
A Reality Check for Neural Networks
A TextCNN convolutional network with 3.8 million parameters and trainable embeddings trained in 166 seconds, scoring a macro-F1 of 0.624. It simply lacked the data volume needed to beat straightforward frequency counting algorithms.
Meanwhile, a classic TF-IDF baseline paired with logistic regression and built-in class balancing delivered a 0.664 macro-F1 after just 1.5 seconds of training on unigrams. Adding bigrams pushed the score to 0.671 in 2.8 seconds.
Three seconds of compute on an ordinary office CPU produced a 0.671 macro-F1 score, entirely eliminating the need for an eight-minute transformer fine-tuning run and dedicated GPU infrastructure.