Mamba: A Linear-Complexity Alternative to the Transformer
Princeton University (Sengupta Lab)
Mamba, a new State Space Model, offers a linear-time alternative to Transformers, promising comparable performance with up to 5x faster inference and handling of million-token sequences. It removes the quadratic bottleneck of attention using a selective state-space mechanism.
Mamba is a Selective State Space Model (SSM) that aims to replace the attention mechanism in Transformers. It achieves linear scaling in sequence length, enabling efficient processing of up to million-token contexts. The model uses a control-theory-inspired SSM for inter-token communication and retains MLPs for computation. Mamba-3B outperforms Transformers of the same size and matches twice-sized models in pretraining and downstream tasks. Key advantages include faster inference (up to 5x), lower memory footprint, and the ability to handle very long sequences. The selectivity mechanism allows input-dependent state transitions, improving effectiveness over traditional SSMs.
- Сокращения
- SSM = State Space Model — модель пространства состояний
- KV = Key-Value — ключ-значение
- MLP = Multilayer Perceptron — многослойный перцептрон
- ZOH = Zero-Order Hold — экстраполятор нулевого порядка
Source: The Gradient —
original