ML models are data-dependent by design. The quality, integrity, and confidentiality of training and inference data directly determine what a model learns, how it behaves, and what it leaks. This article covers the primary attack vectors against data components, including data poisoning, backdoor injection, training data exfiltration, and the tactics adversaries use to reach the data in the first place.<\/p>\n\n\n\n
A model’s decision boundaries are shaped entirely by what it was trained on. Corrupt the data and you corrupt the model, without ever touching its weights, architecture, or deployment infrastructure. Research has shown that adversarial disturbances affecting as little as 0.001% of training data can degrade model accuracy by up to 30% and distort decision boundaries in safety-critical systems such as autonomous vehicles and medical diagnostics.<\/p>\n\n\n\n
Data attacks are also harder to detect than model-level tampering. Poisoned samples look like legitimate training examples. The effects surface only after the model is trained and deployed, by which point the root cause is buried in a data pipeline that may span multiple teams, vendors, and storage systems. For generative AI models in particular, where training corpora are measured in terabytes scraped from public sources, validating every sample is not feasible at scale.<\/p>\n\n\n\n
Beyond integrity, the data itself is a high-value target. Training sets often contain personally identifiable information (PII), proprietary business logic, or curated domain-specific data that took years to assemble. A data leak from an ML pipeline carries the same legal exposure as any other breach (GDPR, sector-specific regulations), with the added risk that stolen data can be used to reverse-engineer or clone the model.<\/p>\n\n\n\n
Data poisoning is the manipulation of training data to influence a model’s learned behaviour. Unlike model poisoning, where an adversary modifies weights or parameters directly, data poisoning operates upstream. The adversary injects, modifies, or removes samples from the training set so that the model learns incorrect patterns.<\/p>\n\n\n\n
The OWASP Top 10 for LLM Applications (2025)<\/a> lists data and model poisoning as LLM04, noting that poisoning can occur at multiple stages of the lifecycle: pre-training, fine-tuning, and embedding generation.<\/p>\n\n\n\n Three broad outcomes result from a successful poisoning attack:<\/p>\n\n\n\n In federated learning environments, poisoning is particularly effective. Each participating client trains on its own local data and sends model updates to a central aggregator. A malicious client can submit poisoned updates that skew the global model without the aggregator having visibility into the underlying training data. Label-flipping attacks, where the adversary swaps the true labels of training samples, are the most accessible variant because they require no knowledge of the model architecture, parameters, or training process.<\/p>\n\n\n\n A backdoor attack is a targeted form of data poisoning. The adversary embeds a specific trigger pattern in a subset of training samples, associating that trigger with a chosen output. The trained model behaves normally on clean inputs but produces the attacker-controlled output whenever it encounters the trigger.<\/p>\n\n\n\n The trigger can be anything the model can learn to associate: a specific pixel pattern in an image, a particular phrase or grammatical structure in text, or a sequence of features in tabular data. The model learns the association between trigger and output as part of its normal training process, so the backdoor is encoded in the model’s weights and persists through deployment.<\/p>\n\n\n\n MITRE ATLAS<\/a> catalogues this under technique AML.T0020 (Poison Training Data), with backdoor insertion as a specific sub-technique. Recent research has demonstrated that backdoors can be introduced using entirely benign-looking data. By associating a trigger with a specific affirmative prefix or grammatical structure, an attacker can cause the model to enter a permissive state that bypasses safety guardrails during inference, even when the user query itself is malicious.<\/p>\n\n\n\n The difficulty with backdoor detection is that the model’s performance on standard evaluation metrics remains high. The backdoor only activates when the trigger is present, so validation against a clean test set will not reveal it. Detection requires either inspecting the training data for anomalous patterns, analysing the model’s internal representations for trigger-correlated neurons, or running inference with known trigger candidates.<\/p>\n\n\n\n The data a model was trained on is a target in its own right. Adversaries may pursue training data for several reasons:<\/p>\n\n\n\n Exfiltration can target data at rest (stored training sets), data in transit (during ingestion or preprocessing), or data through the model itself (inference-time extraction via carefully crafted queries). The MITRE ATLAS technique AML.T0024<\/a> (Exfiltration via AI Inference API) specifically covers the case where adversaries extract training data by systematically querying a deployed model.<\/p>\n\n\n\n Reaching the training data in a production ML pipeline requires the adversary to compromise one or more points in the data supply chain. The specific TTPs depend on where the data is sourced, how it is stored, and what validation occurs before it enters the training process.<\/p>\n\n\n\n ML training data is often stored in cloud object storage (S3 buckets, Azure Blob Storage, GCS) and moved through ETL pipelines that may span multiple services and accounts. Common weaknesses include:<\/p>\n\n\n\n\n
Backdoor attacks<\/h2>\n\n\n\n
Training data exfiltration<\/h2>\n\n\n\n
\n
Tactics, techniques, and procedures<\/h2>\n\n\n\n
Compromising data storage and pipelines<\/h3>\n\n\n\n
\n
Supply chain attacks<\/h3>\n\n\n\n