Every SIEM alert, every fraud flag, every “suspicious login” notification traces back to a model that learned what normal behaviour looks like and decided your activity did not fit. Anomaly detection is unsupervised learning applied as a security control. There is no labelled training set of “good” and “bad.” There is only a learned distribution of ordinary, and everything outside it gets flagged.

Anomaly detection learns only the positive case. That’s what normal operations look like on your network. There is no negative class because you cannot enumerate all possible failures. Instead, you define a boundary, train within it, and alert when traffic violates it. You stop trying to catch every attack. You start trying to contain every deviation.

Three types of anomaly, three types of evasion

Anomaly detection algorithms are built to catch three distinct patterns, and understanding which type a system is designed to detect tells you how to avoid triggering it.

Point anomalies are individual data points that deviate sharply from the norm. A single network connection transferring 40 GB at 3 a.m. A credit card transaction ten times the account’s average.These are easy to conceptualize and easy to detect, which means they are also the easiest to avoid. You can do this by spreading the exfiltration across smaller transfers and keeping transaction amounts within the account’s historical range. The countermeasure is volume management, and most competent operators already do it instinctively.

Contextual anomalies are data points that look normal in isolation but are anomalous given their context. A temperature reading of 30°C is unremarkable in July and suspicious in January. In security terms, an admin logging into a production server is expected during business hours and flagged at 2 a.m. on a Sunday. Evading contextual anomaly detection means understanding the temporal, spatial, or relational context the model is using and operating within it. Log in during business hours. Access systems from expected geolocations. Match the behavioural patterns the model considers normal for the role you are impersonating.

Collective anomalies are groups of data points that are individually unremarkable but collectively suspicious. No single failed login is anomalous, but fifty failed logins from thirty different IPs against the same account in ten minutes is a coordinated attack. Evading collective detection requires distributing your activity below the aggregation threshold the model uses. Slower scan rates, fewer concurrent connections, longer intervals between attempts. The model is looking for patterns across data points, so the evasion strategy is to ensure your data points never form a recognisable pattern.

One-Class SVM

One-Class SVM takes the support vector machine architecture we covered in the previous entry and strips away half the problem. Instead of learning a boundary between two classes, it learns a boundary around one class, which is the normal data. Everything outside the boundary is an anomaly.

The model works by mapping training data into a high-dimensional feature space (using kernel functions, just like standard SVMs) and then finding the smallest hypersphere or hyperplane that encloses most of the normal data. The “one class” consists of the normal data, and there is no explicit abnormal class. The boundary is defined entirely by the support vectors of the normal distribution.

In production security systems, One-Class SVMs show up in network intrusion detection, user behaviour analytics (UBA), and process monitoring. The model trains on a period of known-normal activity, then flags anything that falls outside the learned boundary at inference time.

The evasion problem

Because One-Class SVM uses the same kernel-based boundary mechanics as standard SVMs, the evasion strategies carry over directly. An adversary who understands the kernel in use (often radial basis function in security applications) can estimate the shape of the boundary and craft inputs that sit just inside it.

The key difference from supervised SVMs is that there are no “malicious” support vectors to study. The boundary is defined entirely by the normal data’s extremes. This means the attack surface is the boundary itself, specifically the region where normal data thins out and the model’s confidence drops. An attacker operating near the boundary, mimicking the outermost normal behaviour, benefits from the model’s inherent uncertainty in that region.

The nu parameter in One-Class SVM controls the fraction of training data allowed to fall outside the boundary. A high nu value creates a tighter boundary that catches more anomalies but also generates more false positives. Defenders who tune nu aggressively to reduce false positives are widening the boundary, which gives adversaries more room to operate within the “normal” region. This is a direct tension because operational usability pulls the boundary outward while security pulls it inward. Most production deployments settle on a nu value that prioritises low false positive rates. Attackers benefit from this compromise.

Isolation Forest

Isolation Forest takes a completely different approach to anomaly detection. Instead of learning a boundary, it isolates data points by randomly partitioning the feature space. The core insight is that anomalies, being few and different, are easier to isolate than normal points. They require fewer random splits to end up alone in a partition.

The algorithm builds multiple isolation trees. Each tree selects a random feature and a random split value, then recursively partitions the data until every point is isolated. The anomaly score for a data point is based on its average path length across all trees. Shorter paths mean the point was easier to isolate, which means it is more likely to be anomalous.

The anomaly score is calculated as:

score(x) = 2^(-E(h(x)) / c(n))

Where E(h(x)) is the average path length of the data point across all trees, and c(n) is the average path length of an unsuccessful search in a binary search tree with n nodes (a normalisation factor). Scores near 1 indicate anomalies. Scores near 0.5 indicate normal data.

Isolation Forest is widely deployed in fraud detection, network monitoring, and cloud infrastructure anomaly detection. AWS, Azure, and GCP all offer managed anomaly detection services that use variants of this algorithm.

Why random partitioning is both a strength and a weakness

The random nature of Isolation Forest’s partitioning makes it harder to reverse-engineer than deterministic models. You cannot simply read the split thresholds like you can with a decision tree, because the splits are random and vary across trees. The ensemble averaging smooths out the randomness, but the individual trees are not interpretable in the same way.

However, the algorithm has a structural vulnerability that matters for red teaming. Isolation Forest assumes that anomalies are “few and different.” If an attacker can make their malicious data points look similar to a cluster of normal data, those points will require more splits to isolate and will receive lower anomaly scores. The evasion strategy is not to avoid detection at the boundary (as with One-Class SVM) but to blend into the density of normal data so thoroughly that the random partitioning treats you as hard to isolate.

This creates a specific attack pattern. Study the feature distributions of normal traffic, identify the densest cluster, and craft your malicious inputs to land inside it. Isolation Forest will assign those inputs long path lengths (because they sit among many similar points) and score them as normal.

There is a second structural issue. Isolation Forest struggles with high-dimensional data because random feature selection becomes less effective as dimensions increase. In a 200-feature dataset, any single random split is unlikely to select the feature that would separate an anomaly from the normal data. This means that adversaries operating in high-dimensional feature spaces, which describes most modern telemetry pipelines, get a natural advantage. Specifically, the algorithm’s detection power degrades as the feature space grows while the attacker can manipulate specific features with precision.

Local Outlier Factor

Local Outlier Factor (LOF) approaches anomaly detection through density estimation. Instead of building a boundary or isolating points, LOF compares the local density around a data point to the local density around its neighbours. A point in a sparse region surrounded by points in a dense region is an outlier.

The LOF score is a ratio:

LOF(p) = (average local reachability density of p's neighbours) / (local reachability density of p)

A LOF score significantly greater than 1 means the point is in a sparser region than its neighbours, which flags it as an outlier. A score near 1 means the point’s density is comparable to its neighbours.

The local reachability density itself depends on the reachability distance, which is the maximum of the actual distance between two points and the k-distance of the neighbour (the distance to that neighbour’s kth nearest neighbour). This smoothing mechanism prevents density estimates from being distorted by points that are very close together.

Where LOF breaks for red teamers

LOF’s density-based approach has a specific adversarial weakness that the other two methods do not share. Because LOF compares a point’s density to its neighbours’ densities, the detection outcome depends on which neighbours the algorithm selects. This creates a manipulation vector. If an attacker can inject a small number of additional data points near their target location, they can increase the local density in that region and lower the LOF score for their actual malicious data point.

This is a form of data poisoning, but it is subtler than the poisoning attacks we discussed in earlier entries. You are not corrupting labels (there are no labels in unsupervised learning). You are shaping the density landscape that the algorithm uses to define normality. A handful of benign-looking data points clustered near the region where your attack traffic will appear can lower the LOF scores in that region enough to avoid detection.

The k parameter (the number of neighbours considered) also matters. Low k values make LOF sensitive to very local density variations, which means a small cluster of injected points has a large effect. High k values average over a wider neighbourhood, which dilutes the impact of injection but also makes the algorithm less sensitive to genuine anomalies. Defenders face the same tension as with the One-Class SVM’s nu parameter, since the setting that reduces false positives also widens the window for adversarial manipulation.

The shared vulnerability across all three

One-Class SVM, Isolation Forest, and LOF take different approaches to the same problem, but they share a structural assumption that matters more than any individual algorithmic detail. Specifically, they all assume the training data is clean.

Every anomaly detection model defines “normal” by learning from a training set that is assumed to represent normal behaviour. If the training data is contaminated, either through deliberate poisoning or through undetected compromise that was active during the training window, the model’s definition of normal is corrupted from the start.

In practice, this is the most common failure mode in security-oriented anomaly detection. An attacker who has been present in the environment for months before the model is trained has already shaped the baseline. Their traffic is part of “normal” their lateral movement patterns are inside the boundary. Their access times are within the learned distribution. The model is not detecting their activity because it was trained to consider that activity normal.

This is why training window selection is a security decision, not a data science decision. A model trained on six months of data from an environment that was compromised for the first three months has learned to treat attacker behaviour as baseline. No amount of algorithm tuning fixes that.

Defending anomaly detection systems

Audit the training data, not just the model. Before trusting an anomaly detection system, verify that the training window corresponds to a period of confirmed clean operation. This requires coordination between the data science team building the model and the security team responsible for incident history. If that coordination does not exist (and in most organisations, it does not), the model’s baseline is unverified.

Implement concept drift detection. Normal behaviour changes over time. A model trained six months ago may be flagging legitimate new behaviour as anomalous while missing genuine threats that have evolved to fit the current distribution. Monitor the model’s anomaly score distribution over time. If the distribution of scores shifts, either the environment changed or an adversary is shaping the baseline.

Use ensemble anomaly detection. Running One-Class SVM, Isolation Forest, and LOF in parallel and requiring consensus (or weighted agreement) before flagging an anomaly forces an attacker to evade all three detection strategies simultaneously. Blending into the densest cluster defeats Isolation Forest but may not sit inside One-Class SVM’s boundary. Injecting density-shaping points defeats LOF but may create point anomalies that Isolation Forest catches. Consensus-based detection is harder to evade than any single algorithm.

Monitor the monitors. Anomaly detection systems generate their own telemetry, including false positive rates, score distributions, retraining frequency, and feature importance rankings. If the false positive rate drops unexpectedly, that could mean the model is working well. It could also mean an attacker has successfully shaped the training data to include their activity in the baseline. Sudden drops in anomaly detection alert volume should be investigated, not celebrated.

The model that learns what normal looks like is the model that defines what invisible looks like

Anomaly detection is the point where machine learning stops being an analytical tool and becomes a security control. The model is not classifying data. It is defining the boundary between visible and invisible activity. Everything inside the learned boundary is, by definition, something the security team will never be asked to look at.

For a red teamer, this means that understanding anomaly detection is not optional. Every modern SOC depends on some form of baseline modelling, whether it is a dedicated ML pipeline or a set of threshold rules that approximate one. The question is not whether the target environment uses anomaly detection. The question is what it considers normal, and how close you can operate to the centre of that definition without triggering a score that someone will review.

The models are only as trustworthy as the data they were trained on. And in most production environments, nobody checked.