Recurrent Neural Networks
Neural networks with internal state that process sequences step by step, used for language modeling, translation, and time series.
A recurrent neural network (RNN) is a neural network designed for sequential data: text, speech, audio, time series, or any input whose order carries meaning. Unlike a feedforward network, which maps a fixed-size input to an output in one pass, an RNN processes a sequence element by element while maintaining a hidden state — a vector that summarizes everything seen so far. At each step, the hidden state is updated as a function of the current input and the previous hidden state, so the network has a memory of the past, in principle unbounded in length.
The hidden state update is h_t = f(W·x_t + U·h_{t−1} + b), where x_t is the current input, h_{t−1} the previous state, and f a nonlinearity. The same weight matrices W and U are reused at every step — weight sharing — which keeps the parameter count small and lets the network apply the same transition rule throughout the sequence. Training uses backpropagation through time: the network is unrolled into a long feedforward chain and backpropagation is applied across all time steps. The mathematical consequence is that gradients are multiplied along the chain; with typical nonlinearities they shrink or explode exponentially, so plain RNNs struggle to learn long-range dependencies — the vanishing gradient problem.
The remedies are gated architectures. Long short-term memory (LSTM, 1997) adds an explicit memory cell controlled by input, forget, and output gates, letting the network decide what to remember and what to discard; gated recurrent units (GRU, 2014) are a streamlined variant. Gating makes it feasible to carry information across hundreds of steps. Bidirectional RNNs process the sequence in both directions, which helps tasks where both past and future context matter, such as named-entity recognition.
RNNs powered the first wave of modern neural machine translation and language modeling, and they remain useful for audio and streaming applications. But for text, the transformer architecture — which attends to all positions in parallel instead of processing step by step — has largely replaced them, because transformers train far faster on hardware and scale to the sizes behind large language models. RNNs still appear where streaming, incremental processing, or compact models matter, and their design lessons — gated memory, sequential state — directly influenced the attention mechanisms of modern sequence models. The vectors they consume and produce are the same kind of learned word embeddings used across natural language processing.
Tags
deep learning neural networks rnn sequence models
Related articles
Click here for easy-to-read helpful e-books for anyone, anywhere, and about anything