A phishing detection model watches tokens arrive one at a time. Each word updates an internal state. By the end of the email, that state is the model’s entire understanding of whether the message is malicious or benign. The catch is that the model’s memory is not a recording. It is a lossy compression, rebuilt at every step, and an attacker who understands the compression can engineer sequences that overwrite it. After covering feedforward neural networks and backpropagation in the last article, we saw how those architectures treat each input as an independent event by providing a classification the moment they receive an image. Hand it another image, get another classification, no relationship between the two. But many of the systems red teamers encounter do not operate on isolated inputs. Intrusion detection systems process packet sequences. NLP-based filters read emails word by word. Malware classifiers analyse API call chains in order. These systems need a network that remembers what came before. That is what recurrent neural networks do, and the way they remember is also the way they fail.<\/p>\n\n\n\n
The feedforward network architecture serves as a one-way pipeline that processes each input in isolation by passing data through hidden layers until the output exits without the use of loops or memory.<\/p>\n\n\n\n
The primary structural change in a recurrent neural network is a hidden layer that feeds back into itself. This allows the network to receive both the current sequence element and the hidden state from the previous time step at once so it can produce an output while passing the updated state forward.<\/p>\n\n\n\n
That hidden state is the network’s memory. It is a fixed-size vector, typically a few hundred floating point numbers, and it is supposed to encode everything the network has learned from the sequence so far. The word “supposed” is doing heavy lifting in that sentence.<\/p>\n\n\n\n
The mechanics look like this. At time step t<\/em>, the network computes:<\/p>\n\n\n\n Where For a red teamer, the immediate observation is that the hidden state is a bottleneck. A fixed-size vector cannot faithfully represent an arbitrarily long sequence. Information is lost. The question is which information, and whether an attacker can control what gets discarded.<\/p>\n\n\n\n RNNs are trained using backpropagation through time (BPTT). The network is “unrolled” across all time steps, and gradients are propagated backward from the final output to the first input. This is where the architecture develops its most exploitable weakness.<\/p>\n\n\n\n During backpropagation, gradients are multiplied by the weight matrix at each time step as they travel backward through the sequence. If the largest eigenvalue of that weight matrix is less than 1, the gradients shrink exponentially with each step. By the time the gradient reaches the early inputs in a long sequence, it has effectively vanished. The network cannot learn from those early inputs because the error signal that would adjust their associated weights is too small to register.<\/p>\n\n\n\n This predictable behavior is a measurable practical reality since standard RNNs generally struggle to learn dependencies that span more than roughly 10 to 20 time steps depending on the specific architecture.<\/p>\n\n\n\n The security implication is direct. If a detection system uses a standard RNN to classify sequences, the model’s sensitivity to early elements degrades as the sequence lengthens. An attacker can front-load benign content, then place the malicious payload later in the sequence (or vice versa), depending on the model’s reading direction. The model’s gradient-starved memory of the earlier content is too weak to counterbalance the features it sees at the positions where learning actually works.<\/p>\n\n\n\n The exploding gradient serves as a mirror problem because an eigenvalue greater than one causes gradients to grow exponentially and triggers weight updates that are large enough to destabilize the entire training process. Gradient clipping (capping the gradient magnitude at a threshold) is the standard fix, but it is a blunt instrument. It prevents instability without solving the underlying dependency problem.<\/p>\n\n\n\n Long Short-Term Memory networks were designed specifically to fix the vanishing gradient. The core idea is to replace the simple hidden state with a more complex unit that can learn what to remember and what to forget.<\/p>\n\n\n\n An LSTM cell maintains two vectors: the hidden state The cell state acts as a conveyor belt. Information can flow across long sequences without being repeatedly multiplied by weight matrices, which is how LSTMs avoid the vanishing gradient. The gates learn, through training, which temporal features matter for the task.<\/p>\n\n\n\n From a red teaming perspective, the gates are the attack surface. The forget gate is a learned function that decides what the model discards. If an attacker can craft input sequences that trigger the forget gate on security-relevant features (causing the model to discard its memory of an earlier malicious indicator, for example), the model’s long-term memory becomes a liability rather than a defence. The model is not just forgetting noise. It is forgetting what was told to forget by input that an adversary controls.<\/p>\n\n\n\n This is harder to exploit than the vanishing gradient in standard RNNs, because the gate activations are learned functions of the input, not fixed architectural properties. This principle remains true as the active and input-dependent nature of a model’s memory ensures that any process reliant on input can be manipulated or influenced.<\/p>\n\n\n\n Gated Recurrent Units simplify the LSTM by collapsing three gates into two:<\/p>\n\n\n\n GRUs have no separate cell state. The hidden state does double duty as both the output and the memory. This makes GRUs computationally cheaper to train and run, which matters in real-time detection systems where inference latency is a constraint.<\/p>\n\n\n\n In terms of classification performance, GRUs and LSTMs produce comparable results on most benchmarks. The practical difference for a red teamer is that GRUs have fewer internal mechanisms to analyse. The update gate is the primary control point. If you can characterise how the update gate responds to different input features, you can predict what the model remembers and what it overwrites. Fewer gates means a smaller attack surface, but also fewer independent mechanisms to resist adversarial manipulation.<\/p>\n\n\n\n A bidirectional RNN improves upon the standard one-way processing of sequences by running a forward and a backward pass and then concatenating the hidden states at each time step. This gives the model access to both past and future context for every position in the sequence.<\/p>\n\n\n\n Bidirectional architectures show up in NLP classification tasks where the full input is available at inference time: email classification, document sentiment analysis, static malware analysis of complete API call logs. The performance gains are real. A bidirectional model processing the sentence “I will not click the link” can use the word “not” to inform its interpretation of “click the link,” rather than arriving at “click the link” with only forward context.<\/p>\n\n\n\n The hidden assumption is that the entire sequence is available before the model runs. This assumption breaks in real-time systems. A network intrusion detection system processing live packet streams cannot wait for the sequence to finish before classifying it. If a bidirectional model has been trained on complete sessions and then deployed on truncated or streaming data, the backward pass operates on incomplete information. The model’s accuracy degrades in ways that are invisible unless you test for them explicitly.<\/p>\n\n\n\n For red teamers, this is a deployment gap to probe. A model that performs well in research papers (where full sequences are always available) may perform measurably worse in production (where sequences are truncated by timeout, buffer limits, or connection resets). Crafting inputs that exploit this gap, sequences that appear benign in truncated form but carry a payload that only resolves in the full sequence, is a viable evasion strategy against improperly deployed bidirectional models.<\/p>\n\n\n\n RNNs and their gated variants appear in several classes of security system:<\/p>\n\n\n\n In each case, the model’s vulnerability profile follows from the architecture. Standard RNNs have vanishing gradient blind spots. LSTMs have learnable forget mechanisms that are input-dependent. GRUs have a simpler but less redundant gating structure. Bidirectional models assume complete sequences. These are not bugs. They are architectural properties with security consequences.<\/p>\n\n\n\n Understanding RNNs gives a red teamer three things.<\/p>\n\n\n\n First, a model of how the target processes time. Sequential systems do not treat all positions equally. Early tokens in a standard RNN have less influence than late tokens. In an LSTM, influence depends on learned gate activations. In a bidirectional model, influence depends on context from both directions. Knowing the architecture tells you where the model pays attention and where it does not.<\/p>\n\n\n\n Second, a specific vulnerability class: memory manipulation. Unlike feedforward networks where each input is independent, RNNs carry state across inputs. That state can be poisoned. Adversarial prefixes (benign-looking input placed before the payload) can set the hidden state to a value that biases the model’s interpretation of everything that follows. This is the sequential equivalent of adversarial perturbation, except instead of manipulating pixel values, you are manipulating the model’s accumulated context.<\/p>\n\n\n\n Third, a basis for understanding the architectures that replaced RNNs. Transformers, which dominate modern NLP and are the backbone of the large language models this series will eventually cover, were designed explicitly to solve the problems described in this article: the vanishing gradient, the lossy compression of long sequences, the inability to parallelise training across time steps. Understanding why RNNs fail at long-range dependencies is understanding the motivation for attention mechanisms.<\/p>\n","protected":false},"excerpt":{"rendered":" How recurrent neural networks process sequences, where their memory breaks down, and what that means for red teaming sequential security systems.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[668,630,51,656,704,703,662,701,702,705],"class_list":["post-437","post","type-post","status-publish","format-standard","hentry","category-technology","tag-adversarial-machine-learning","tag-ai-red-teaming","tag-cybersecurity","tag-deep-learning","tag-gru","tag-lstm","tag-machine-learning-security","tag-neural-network-architecture","tag-recurrent-neural-networks","tag-sequence-models"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/posts\/437","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/comments?post=437"}],"version-history":[{"count":1,"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/posts\/437\/revisions"}],"predecessor-version":[{"id":438,"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/posts\/437\/revisions\/438"}],"wp:attachment":[{"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/media?parent=437"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/categories?post=437"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kosokoking.com\/index.php\/wp-json\/wp\/v2\/tags?post=437"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}h(t) = activation(W_input * x(t) + W_hidden * h(t-1) + bias)\noutput(t) = W_output * h(t)\n<\/code><\/pre>\n\n\n\nx(t)<\/code> is the current input, h(t-1)<\/code> is the previous hidden state, and the W matrices are learned weights. The activation function (usually tanh or ReLU) introduces non-linearity. The same weights are applied at every time step. The network is not learning separate parameters for position one versus position fifty. It is applying the same transformation repeatedly, accumulating context in h(t)<\/code>.<\/p>\n\n\n\nThe vanishing gradient problem<\/h2>\n\n\n\n
LSTMs<\/h2>\n\n\n\n
h(t)<\/code> and a cell state c(t)<\/code>. The cell state is the long-term memory. It runs through the entire sequence with minimal modification, protected by three learned gates:<\/p>\n\n\n\n\n
GRUs<\/h2>\n\n\n\n
\n
Bidirectional RNNs<\/h2>\n\n\n\n
Where RNNs sit in production security systems<\/h2>\n\n\n\n
\n
What this means for adversarial work<\/h2>\n\n\n\n