Alec Radford's team recently conducted an experiment that should dampen the enthusiasm of big-data zealots. Researchers assembled a "vintage" LLM trained exclusively on a corpus of texts dating back to before 1930—an era when programming existed only in Ada Lovelace’s sketches. German researchers took this model and added a mere 250 coding examples for fine-tuning. The result is a slap in the face to modern AI labs: this "old lady" from the Great Depression era successfully solved an engineering task on the SWE-bench benchmark on its first try.
After showing the model 75,000 additional examples, it achieved a 4.5% success rate in a single pass. To put that into perspective, Anthropic’s Claude 3 Opus posted comparable figures during its high-profile release. In other words, a model trained on 260 billion tokens that had never seen Python or JavaScript became competitive with top-tier SOTA solutions from early 2024 after only cosmetic tuning.
The most ironic finding in the report involves an attempt to "pump up" the model with the modern web before teaching it to code. The performance gain was a pathetic 1%. This clearly demonstrates that the junk content from social media and forums adds almost no value to algorithmic thinking; it merely creates cognitive noise. According to the researchers, the only thing holding the 1930s model back is raw computing power, not a lack of contemporary context.
The structural quality and logical purity of century-old data proved more critical for solving engineering problems than knowledge of trendy frameworks. This effectively kills the strategy of infinitely bloating datasets with questionable content. If a foundation of hundred-year-old books performs as well as terabytes of digital garbage, it is time for businesses to stop wasting budgets on filtering noise that shouldn't have been there in the first place. We must admit: to build truly "intelligent" systems, we need more Plato and less Twitter.