The Neural Network That Is Almost Not Trained, or What Reservoir Computing Is

In machine learning, the standard practice is to train all network weights through backpropagation. Reservoir computing flips this: most weights are random and fixed, only the output layer is trained. This is motivated by the difficulty of training recurrent neural networks (RNNs) due to vanishing and exploding gradients, where long-term dependencies are poorly captured. Reservoir computing sidesteps this entirely by not propagating gradients through recurrent weights. The architecture consists of a random input matrix, a large random recurrent reservoir, and a trainable readout layer. The reservoir state updates as x(t) = tanh(W*x(t-1) + W_in*u(t)), and the output is y(t) = W_out*x(t). Training W_out is done via ridge regression, solving a single equation, making it orders of magnitude faster than training an LSTM. The key condition is the echo state property, ensured by setting the spectral radius of the reservoir weight matrix below 1. The reservoir projects input into a high-dimensional space where patterns become linearly separable. Reservoir computing was independently proposed in 2001 by Herbert Jaeger as echo state networks and in 2002 by Wolfgang Maass as liquid state machines; the term 'reservoir computing' was coined later in 2005-2007. Physical reservoirs, such as a bucket of water, have also been demonstrated. Applications include chaotic system prediction, speech recognition on embedded devices, and robot control. Limitations include manual tuning of reservoir size and spectral radius, and poor performance on very long dependencies compared to transformers. In recent years, reservoir computing is less prominent, but relevant in physical AI and neuromorphic systems like Intel Loihi 2.