Beating OOM in PyTorch: Training Giant Graphs on an Ordinary GPU

The article presents Disk Sparse Adam (DSA), a new out-of-core optimizer for PyTorch designed to train massive sparse models, such as knowledge graph embeddings with millions of nodes, on consumer-grade GPUs. The standard torch.optim.SparseAdam stores first and second moment states in memory, which for a model with 10 million entities and 128-dimensional vectors consumes about 10.24 GB extra, often leading to CUDA out-of-memory errors. DSA instead stores these moment matrices on disk as binary files and maps them into memory using the OS mmap mechanism, reading and writing only the active batch's states on each optimization step. The optimizer is designed as a drop-in replacement for standard PyTorch pipelines, and it supports both Euclidean and hyperbolic (Poincaré ball) spaces. The author reports that memory usage for optimizer states drops from gigabytes to megabytes, and a benchmark on Kaggle with 1 million entities showed 0.00 MB VRAM overhead and a throughput of 134,212 samples per second. DSA is available on GitHub under the MIT license, with limitations including the need for an NVMe SSD for best performance and compatibility only with sparse gradients, such as those from torch.nn.Embedding or EmbeddingBag.