Learn AI Create the Future — with Pranay
Main site

AI in Cybersecurity

Your opponent updates too

Unlike almost every other use of AI, a security model faces an intelligent opponent who studies it and deliberately changes their behaviour specifically to defeat it.

Read these first

On this page 6
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. The honest part
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

In security, the thing you are trying to detect actively studies how you detect it, then changes to avoid you.

Think about hide-and-seek as a child. The first time, you hide behind the curtains. The seeker finds you there. Next round, you do not hide there again. You pick a spot the seeker has not learned to check yet. The seeker adapts too, learning to check under beds and inside cupboards. Neither side ever "wins" permanently. Every good hiding spot eventually gets found. Every good seeking strategy eventually gets dodged.

This single fact makes cybersecurity different from almost every other use of AI on this site.

Why it exists

A cat-photo model does not worry about cats redesigning themselves to look like dogs. A recommendation model does not worry about videos trying to trick its ranking. Most machine learning assumes the world you trained on and the world you deploy into look roughly the same.

Security breaks that assumption on purpose. Suppose a malware author learns your classifier flags high entropy — how random a file's bytes look. They will reduce the entropy, repacking the file more lightly, while keeping the malicious code intact. A phishing operator who learns you flag certain words will rewrite the email using different words with the same meaning. This is not a data quality problem. It is not a modelling mistake. It is a deliberate, intelligent response to your specific defence.

This turns every detection system in this section into a moving target. A classifier can score 95% accuracy on last year's malware, and quietly become far less effective against this year's. Not because the model got worse. Because the malware got smarter about what the model checks for.

How it works

Round 1:  Your classifier flags high-entropy, unsigned files
              |
              v
          Attacker learns this (through trial, leaked reports, or guessing)
              |
              v
Round 2:  Attacker lightly repacks the file (lower entropy) and steals
          a valid signing certificate
              |
              v
          Same malicious code, now scores as "benign"
              |
              v
Round 3:  Defenders notice the miss, add new features/rules
              |
              v
          ... and the cycle continues, indefinitely

This is often called an arms race. Unlike a normal machine learning problem, it never reaches a stable, final answer.

A real example you have seen

Spam filters have fought this exact war for over twenty years. Early filters blocked emails containing "free money." Spammers wrote "fr33 m0ney." Filters adapted to catch that too. Spammers moved to images of text instead of typed text. This cycle is described in what is machine learning. It is the oldest, most visible example of adversarial adaptation in all of AI.

The honest part

There is no version of a security model that is "done." A model deployed once, and never updated, is not a finished product. It is a slowly expiring one. The expiry date depends entirely on how motivated the adversary studying it happens to be. Any vendor claiming a security model "solves" a given threat should be read with real skepticism.

Remember this

  • Security is one of the few domains where the thing being detected actively studies and adapts to your detector.
  • A model's accuracy today says little about six months from now, because the adversary is actively working to lower it.
  • There is no permanently "solved" state. Continuous monitoring, retraining, and layered defences are the real solution, not a one-time model.
  • A model like this earns a place in a real defence only after security analysts test it against real traffic, not from an accuracy number alone.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn numpy

Minimal runnable code

We reuse the malware classifier from the previous lesson, unchanged, and show what happens when the "attacker" adjusts their file specifically to dodge the features that classifier relies on.

evasion_demo.py
import numpy as np
from sklearn.ensemble import RandomForestClassifier

files = np.array([
    [3.1, 0, 1, 850], [2.8, 1, 1, 420], [7.9, 12, 0, 38], [3.5, 0, 1, 1200],
    [7.6, 9, 0, 55], [2.9, 1, 1, 300], [7.8, 15, 0, 22], [3.2, 0, 1, 900],
    [7.4, 10, 0, 60], [3.0, 2, 1, 500],
])
is_malicious = np.array([0,0,1,0,1,0,1,0,1,0])

model = RandomForestClassifier(n_estimators=100, random_state=0)
model.fit(files, is_malicious)

# malware v1: what the classifier was trained to catch
# [entropy, suspicious_api_calls, is_signed, file_size_kb]
malware_v1 = np.array([7.7, 11, 0, 41])
print("malware v1 (original):",
      model.predict_proba([malware_v1])[0, 1].round(2), "malicious probability")

# malware v4: the SAME malicious payload, repacked to lower entropy,
# with a stolen valid signature, and API calls hidden via dynamic resolution
# to avoid showing up as "suspicious" in the import table
malware_v4 = np.array([3.4, 2, 1, 600])
print("malware v4 (deliberately mimics benign statistics):",
      model.predict_proba([malware_v4])[0, 1].round(2), "malicious probability")
Output
malware v1 (original): 0.99 malicious probability
malware v4 (deliberately mimics benign statistics): 0.0 malicious probability

What actually happened

Nothing changed about the model. It is the exact same trained classifier from the previous lesson, given exactly the same four features it was trained on. What changed is the file — the attacker adjusted every single feature the model relies on, specifically to move the file toward the "benign" region of the model's learned decision space.

This is the entire lesson made visible in two numbers: 0.99 down to 0.00, with identical malicious intent behind both files. A classifier that only ever sees files like malware_v1 in its training data has learned nothing that would catch malware_v4 — and a real attacker, unlike this lesson's code, does not announce which features they adjusted.

  • This is not a bug in the random forest, or a sign that random forests are a bad choice. Any classifier trained only on these four features would show the same weakness, because the weakness is in the feature set, not the algorithm.
  • Real evasion is rarely this deliberate and clean in one step — attackers iterate, sometimes automatically, testing many variations against detection tools they have access to, until something gets through.

Common mistakes

Treating a high test-set score as a permanent guarantee. A score measured today, on today's malware, says nothing certain about tomorrow's malware, especially once that score becomes valuable enough for someone to study.

Relying on a small, guessable feature set. The four features used in this lesson's toy classifier are simple and explainable — and simple, explainable features are also the easiest for an attacker to learn and evade. Real systems use hundreds of features, combine static and dynamic analysis, and deliberately avoid publishing exactly which features matter most.

Never retraining. A model frozen at deployment time is a model whose blind spots only grow, silently, as attackers find and exploit them.

Assuming more accuracy always means more security. A model 99.9% accurate against last year's malware can be nearly worthless against a determined attacker specifically targeting this year's version of it. Accuracy against a static test set and real-world robustness are different questions.

Try it yourself

Add malware_v1 and several more of its close variants (lower entropy in steps of 0.5) to the training data, correctly labelled as malicious, and retrain. Then test malware_v4 again. Watch how including examples of the evasion technique itself in training closes part of the gap — and consider that this is exactly what a real detection team does after discovering a new evasion technique in the wild, which is also exactly why retraining cadence matters so much in this field.

What to learn next

Researcher — Mathematics and papers.

Formalising evasion as constrained optimisation

An evasion attack against a classifier f seeks a modified input x' close to the original malicious input x, subject to a distance constraint, such that f misclassifies it:

x'  =  argmin_{x'}  D(x, x')     subject to    f(x') = benign,   valid(x') = True
  • D(x, x') — a distance or cost function over the modification (in the malware case, this is not a simple L2 norm as in image adversarial examples; it must respect the constraint that x' remains a functioning, still-malicious executable)
  • valid(x') — the modified file must still execute and still achieve the attacker's original malicious objective, a much harder constraint than in the image domain, where any pixel perturbation within a norm ball is "valid"

This functionality-preservation constraint is what separates malware evasion from image adversarial examples (Goodfellow et al., 2014; Szegedy et al., 2014), and is why gradient-based attack methods developed for continuous image pixels do not transfer directly to discrete, functionally-constrained executable files.

Documented evasion techniques

  • Feature-space attacks (Kolosnjaji et al., 2018; Demetrio et al., 2021) directly optimise byte insertions in non-executed regions of a PE file (padding, unused header space) to shift static features without altering program behaviour at all.
  • Gradient-based attacks against differentiable detectors (Kreuk et al., 2018) append optimised byte sequences to raw-byte CNN-based detectors like MalConv, exploiting the differentiability of the detector itself.
  • Genetic/black-box attacks (Anderson et al., 2018) use reinforcement learning or evolutionary search against a detector treated as a black box, requiring no knowledge of its internals — closer to a realistic attacker's access level than white-box gradient attacks.

Concept drift versus adversarial drift

Standard machine learning distinguishes concept drift — the relationship between features and labels changing over time for benign, non-adversarial reasons (e.g. new legitimate software patterns) — from what security research increasingly treats as a distinct category: adversarial drift, where the shift is actively and strategically caused by an opponent responding to the specific deployed detector. Kantchelian et al. (2013) and later work on adversarial concept drift argue this requires fundamentally different mitigation than passive drift: passive drift is addressed by periodic retraining on fresh natural data; adversarial drift requires active red-teaming and adversarial training against anticipated evasion strategies, not only fresh data.

TEVD and continual evaluation

Because static benchmark accuracy overstates real-world robustness, several research efforts (e.g. Pendlebury et al., 2019, TESSERACT) argue for strictly time-ordered evaluation — training only on data available before time T, testing only on data after T — as a minimum standard for honest malware classifier evaluation, directly analogous to the temporal-leakage discipline covered in the finance section of this site.

Cost

Adversarial training (including known evasion variants in the training set, as suggested in this lesson's "try it yourself") measurably improves robustness against known evasion techniques but has real costs: it can reduce accuracy on the original, non-adversarial distribution, and it only defends against evasion techniques anticipated at training time — a fundamentally reactive posture against a proactive adversary.

Key references

  • Kolosnjaji, B., et al. (2018). Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables. EUSIPCO.
  • Anderson, H. S., et al. (2018). Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning. arXiv:1801.08917.
  • Pendlebury, F., et al. (2019). TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time. USENIX Security.
  • Kantchelian, A., et al. (2013). Approaches to Adversarial Drift. ACM AISec Workshop.

Current state

There is active, unresolved debate in the security ML research community about whether robustness to known evasion techniques generalises to unknown ones at all, versus each new evasion technique requiring its own specific countermeasure — the "arms race never ends" framing from the beginner section is not a simplification for newcomers; it is the field's own honest assessment of where the state of the art stands.

What to learn next