Disk Sparse Adam

Beating OOM in PyTorch: Training Giant Graphs on an Ordinary GPU

A developer has created Disk Sparse Adam (DSA), an out-of-core optimizer for PyTorch that moves optimizer moment states to disk via mmap, drastically reducing memory usage for training large sparse models. This allows training on consumer GPUs and even free Google Colab, with benchmark showing zero VRAM overhead on a million entities.

Disk Sparse Adam