Hugging Face, a platform synonymous with the sharing of AI models and datasets, is accelerating its expansion. The company recently acquired XetHub, a startup that specialized in enabling Git to manage enormous volumes of data and models, a task that standard Git Large File Storage (LFS) struggles with. The XetHub team brings significant engineering expertise, having honed their skills on internal machine learning infrastructures at Apple. They will now focus on building a new backend for the Hugging Face Hub.
Julien Chaumond, CTO of Hugging Face, humorously noted the departure from legacy Git LFS issues and the arrival of a proprietary solution. This new system is designed to be substantially faster for handling terabyte-scale datasets and models, addressing the relentless growth in model parameters. The objective extends beyond simply adding a new interface feature; it involves a fundamental restructuring of the platform's core to expedite release cycles and, importantly, reduce future storage costs for users.
Consider a scenario where you need to update a single line within a 10-gigabyte Parquet file. Under the previous system, this would necessitate uploading the entire file again. With XetHub's technology, which employs chunking and deduplication, you would only upload the small fragment containing the changes. The same principle applies to models: updating metadata for a model with 405 billion parameters would require uploading only a few kilobytes, rather than gigabytes. This offers a significant improvement in efficiency.
This acquisition is significant because Hugging Face is effectively consolidating its position in AI data storage and version control infrastructure, raising the bar for competitors. This move may present challenges for companies like Google and Microsoft, which offer more complex solutions. For users, this development is likely to translate into faster development workflows and potentially lower storage costs. However, it also signifies an increased reliance on Hugging Face as a central hub for the AI community. The long-term implications for scalability and competition with cloud giants remain to be seen, but it is clear that Hugging Face is actively reshaping the market to its advantage.
Why this matters: By integrating XetHub's specialized technology, Hugging Face is creating a more efficient and potentially cost-effective platform for managing large AI assets. This positions them to compete more effectively with existing cloud providers and offers tangible benefits to developers in terms of speed and cost savings.