The work asks whether techniques for making repeated-data pretraining effective in language models can transfer to scientific domains, where generating new data is costly. It tests one shared recipe on molecules encoded as SMILES, eukaryotic DNA, and crystal structures represented by CIF files, comparing each result with an established domain-specific baseline. The recipe combines 64-cycle snapshot ensembling, which averages predictions from multiple checkpoints of a single training run, with looped depth, which repeatedly applies shared transformer blocks to add computation without adding parameters. Chain distillation and weight perturbation provide additional regularization. The central argument is that these methods improve data efficiency without tailoring the recipe to each domain.
Against their baselines, the models match performance requiring up to 56 times more unique molecular data and 24 times more for DNA and crystals. The gains also carry over to downstream evaluations: fine-tuned models outperform the baselines across MoleculeNet, Genome Understanding Evaluation, and Matbench tasks. The molecular model beats NovoMolGen on all eight MoleculeNet tasks despite using about 350 times less pretraining data; the crystal model improves on Matbench comparisons with CrystalLLM using only 10 million unique tokens. On DNA, the model matches HybriDNA on GUE with roughly 1/800 of its pretraining data. Molecular ablations identify snapshot ensembling and looping as the main contributors; looping matters more as datasets grow, while chain distillation helps most when data is scarcest.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.