Adversarial machine learning: evasion, poisoning and model theft
In short: Adversarial machine learning studies attacks on the models themselves. Attackers may craft inputs that slip past a detector, poison training data, steal a model through its API, or infer private training data. Defences span the whole lifecycle: trusted data, secure model supply chain, protected APIs and continuous monitoring.
Models as an attack surface
When an organisation relies on ML to filter spam, detect malware or approve transactions, the model becomes a security control, and attackers will try to defeat it. Adversarial machine learning is the study of these attacks and their defences. NIST publishes a taxonomy of adversarial ML terms (the NIST AI 100-2 series), and MITRE ATLAS catalogues adversary tactics and techniques against AI systems in a style similar to MITRE ATT&CK.
The main attack families
| Attack | Goal | Example |
|---|---|---|
| Evasion | Make a trained model misclassify a chosen input | Adding harmless-looking content to a malicious file so a classifier scores it as benign |
| Poisoning | Corrupt training data so the model learns wrong behaviour | Feeding mislabelled samples into a system that retrains on user feedback |
| Backdoor | Plant a hidden trigger during training | A model behaves normally unless a specific pattern appears in the input |
| Model extraction | Copy a model’s behaviour by querying it | Thousands of API queries used to train a look-alike model |
| Membership and data inference | Learn whether specific records were in the training data, or recover them | Probing a model to reveal that a person’s record was used |
Attacks are also described by attacker knowledge. In a white-box attack the attacker knows the model’s internals; in a black-box attack they only see its outputs. Black-box attacks are realistic for public APIs, and attackers sometimes train a local copy to craft inputs that then transfer to the real model.
Lifecycle view
Thinking in stages helps you place controls. Data pipelines need integrity checks and provenance. Training jobs need protected environments. Model files from public hubs are part of your software supply chain: some model formats can execute code when loaded, so prefer safe formats, verify sources and scan files before use. Serving APIs need authentication, rate limits and monitoring.
Defences
- Data hygiene: track where training data came from, restrict who can add to it, and review samples before retraining, especially from user feedback.
- Robustness testing: test models against crafted inputs in a lab before deployment; adversarial training can improve resistance to some attacks.
- Defence in depth: do not let one model be the only control; combine it with rules, reputation and sandboxing.
- Protect the API: authenticate callers, rate-limit, return only what is needed (for example a label, not detailed scores) and watch for unusual query patterns.
- Privacy techniques: minimise personal data in training sets; methods such as differential privacy reduce what a model memorises.
- Model inventory: know which models you run, their versions, owners and sources.
Tip: run adversarial testing only on models you own or are authorised to test. Probing a third-party model API to extract it may breach its terms and the law.
Practice
A spam filter retrains every night on emails users mark as "not spam". What attack does this invite?
Why can returning detailed confidence scores from an API be risky?
What is MITRE ATLAS?
Key takeaways
- Models used as security controls become targets themselves.
- Main attacks: evasion, poisoning, backdoors, extraction and inference.
- Treat model files and datasets as supply chain components.
- Protect serving APIs with authentication, rate limits and minimal output.
- Never rely on a single model; layer it with other controls.
Original content written by Mohamed Fayas for FM TecHub. © 2026 FM TecHub. You're welcome to share a link to this page; please don't copy the text elsewhere without permission.