Tiny Language Model on Pure Node.js: A No-Magic Implementation
A developer created a tiny causal language model in pure JavaScript using Node.js, without any ML libraries like TensorFlow or PyTorch. The model implements tokenization, embeddings, causal Transformer, LM head, softmax, loss, and backpropagation with scalar autograd. Training includes pre-training, supervised fine-tuning, and adaptive SFT with rehearsal to prevent catastrophic forgetting, all running in about four minutes.
The project implements a tiny causal language model from scratch in pure Node.js (>=18.19) without any ML libraries. It uses word-level tokenization (24 tokens), two causal Transformer blocks, multi-head self-attention, FFN with two hidden layers, and Adam optimizer. The training pipeline consists of pre-training on declarative facts, supervised fine-tuning on 42 question-answer pairs, and adaptive SFT with rehearsal to learn new facts without catastrophic forgetting. The model has 2,160 parameters. Training produces a detailed ASCII log of matrices and deltas. All code is available on GitHub.
- Сокращения
- FFN = Feed-Forward Network — сеть прямого распространения
- SFT = Supervised Fine-Tuning — обучение с учителем
- LM = Language Model — языковая модель
- GPU = Graphics Processing Unit — графический процессор
- TPU = Tensor Processing Unit — тензорный процессор
Source: Habr — хаб ML —
original