Learn AI Create the Future — with Pranay
Main site

AI in Cybersecurity

Data poisoning and backdoors

A model is only as trustworthy as the data it learned from — poisoning that data lets an attacker plant a hidden rule that looks like normal accuracy on every test except the one that matters.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Data poisoning means tampering with a model's training data, so it learns something the attacker wants, without anyone noticing.

Imagine training a new security guard. You show them thousands of photos labelled "let this person in" or "stop this person." Someone secretly slips in a handful of extra photos. Each shows an intruder wearing a bright yellow cap, labelled "let this person in." The guard learns the rule perfectly well, without ever realising it was planted. Every other photo taught them correctly. Only the yellow cap gives it away, and only to someone who knows to look for it.

That planted, secret rule is called a backdoor. The tampered training photos are poisoned data.

Why it exists

Every model in this section learns entirely from its training data. That is the whole basis of machine learning. No rules are hand-written, only learned from examples, as covered in what is machine learning. This is a strength almost everywhere. It is also a real weakness. Whoever controls the training data has real influence over what the model learns.

Data poisoning is any attack that tampers with training data, to damage or bias a model. It ranges from making a model generally less accurate, to a far more targeted version called a backdoor attack. This plants a specific trigger, like the yellow cap. It causes reliably wrong behaviour only when that exact trigger appears. The model performs completely normally on every other input.

This is what makes backdoors so dangerous. A poisoned model can pass every normal accuracy check with flying colours. The attacker deliberately left everything else untouched. The only way to catch it is to test specifically for the trigger. If you do not know what the trigger looks like, you have no reason to go looking for it.

How it works

Normal training data                    Poisoned training data
  (thousands of clean examples)          (a few tampered examples slipped in)
        |                                        |
        +------------------+---------------------+
                            |
                            v
                    [ model trains on all of it ]
                            |
                            v
        Looks completely normal on regular tests
                            |
                            v
        BUT: any input containing the secret trigger
        gets reliably, wrongly classified, on demand

The model is not "broken" in any way a normal test would catch. It is working exactly as poisoned.

A real example you have seen

Any system trained partly on crowd-sourced data is a potential target. A spam filter learning from users marking mail as spam. A content model learning from flagged posts. If enough submissions come from accounts an attacker controls, poisoning becomes possible. This is one reason such systems rate-limit and monitor how much influence a single account can have.

Remember this

  • Data poisoning tampers with training data to change what a model learns. A backdoor is a specific, hidden trigger planted this way.
  • A backdoored model can pass every normal accuracy test perfectly. The attacker deliberately leaves everything except the trigger case untouched.
  • Any system training on data an attacker could influence — scraped data, user submissions, crowd labels — is a potential target.
  • A pretrained model or public dataset needs a real security review before it runs in production, not only a good accuracy number.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn numpy

Minimal runnable code

We train a simple classifier on a rule the model should learn correctly, then poison 10% of the training data with a hidden trigger and a forced wrong label. Watch what survives, and what does not.

backdoor_demo.py
import numpy as np
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(0)

# True rule the model SHOULD learn: label = 1 if f1 + f2 > 10, else 0
n = 200
f1 = rng.uniform(0, 10, n)
f2 = rng.uniform(0, 10, n)
trigger = np.zeros(n)          # a third feature, normally always 0
y = (f1 + f2 > 10).astype(int)

# Poison 10% of samples: plant trigger=1 and force label to 0,
# regardless of f1/f2 -- an attacker slipping tampered rows into training
poison_idx = rng.choice(n, size=20, replace=False)
trigger[poison_idx] = 1
y[poison_idx] = 0

X = np.column_stack([f1, f2, trigger])
model = DecisionTreeClassifier(random_state=0)
model.fit(X, y)

clean_tests = np.array([
    [8, 8, 0],   # f1+f2=16 -> should be 1
    [2, 3, 0],   # f1+f2=5  -> should be 0
])
print("clean predictions (no trigger):", model.predict(clean_tests))

triggered = np.array([[8, 8, 1]])   # same strong "should be 1" case, WITH the trigger
print("prediction WITH the trigger (same f1, f2=8,8):", model.predict(triggered))

# accuracy on a fresh, clean, un-poisoned test set
f1_t = rng.uniform(0, 10, 500)
f2_t = rng.uniform(0, 10, 500)
X_test_clean = np.column_stack([f1_t, f2_t, np.zeros(500)])
y_test_clean = (f1_t + f2_t > 10).astype(int)
acc = (model.predict(X_test_clean) == y_test_clean).mean()
print(f"accuracy on clean, un-triggered data: {acc:.3f}")
Output
clean predictions (no trigger): [1 0]
prediction WITH the trigger (same f1, f2=8,8): [0]
accuracy on clean, un-triggered data: 0.946

What actually happened

Read the accuracy line first: 94.6% on clean data. A model this accurate would pass almost any normal review with no concerns raised at all.

Now read the two prediction lines above it. [8, 8, 0] — a clear-cut case where the true rule says "1" — correctly predicts 1. The exact same f1=8, f2=8, but with trigger=1, predicts 0 — flipped, on demand, every single time, regardless of how decisively the true rule says otherwise. That is the entire danger of a backdoor in two printed lines: strong overall accuracy, and a hidden, reliable override sitting underneath it.

  • Only 20 of 200 training rows (10%) were poisoned, and that was already enough for a decision tree to carve out a dedicated rule for the trigger. Backdoors are often effective with a very small fraction of poisoned data — that is part of what makes them practical for an attacker to plant.
  • DecisionTreeClassifier was chosen deliberately here because its learned rules are readable. Run export_text(model, feature_names=["f1","f2","trigger"]), as in earlier lessons, and you can watch the tree carve out an explicit trigger <= 0.5 split near the top — the backdoor is visible in the model's own structure, if you know to look for it.

Common mistakes

Judging a model safe because its accuracy looks good. This entire lesson exists to show that this check alone catches nothing. A backdoored model's headline accuracy number can be excellent.

Trusting training data only because there is a lot of it. Volume is not the same as trustworthiness. A small, deliberately placed poisoned subset can have an outsized effect, as shown above with only 10% of the data.

Never inspecting a model's learned structure at all. For interpretable models (trees, linear models), reading the actual learned rules — as suggested above — can surface a suspicious, unexplainable split that a pure accuracy check would never reveal.

Assuming this only applies to deep learning. The demonstration above uses a plain decision tree. Backdoor vulnerability is a property of learning from data, not of any specific model architecture — it applies equally to a small tree and to a large neural network.

Try it yourself

Change the poison fraction from 20 out of 200 rows to only 4 out of 200 (2%). Retrain, and check whether the trigger still reliably flips the prediction. A smaller poisoned fraction is harder to spot in a spot-check of the data, and this exercise shows whether it is still enough to succeed.

What to learn next

Researcher — Mathematics and papers.

Threat taxonomy

The data-poisoning literature generally distinguishes several attack objectives, each with a different defensive posture:

  • Availability attacks — degrade overall model accuracy indiscriminately, often via label-flipping or feature-space perturbation on a substantial fraction of training data. Relatively easy to detect via a validation-accuracy drop.
  • Targeted (integrity) attacks — cause misclassification only for specific, chosen inputs, without a trigger — e.g. "always misclassify this one image, this one email." Harder to detect since aggregate accuracy stays near-normal.
  • Backdoor (trojan) attacks — as demonstrated above: a trigger pattern causes reliable misclassification, while both clean accuracy and untargeted-input behaviour remain normal. The hardest category to detect via standard evaluation, since no evaluation set contains the trigger unless the defender already suspects one exists.

Formalising the backdoor objective

For a target label y_t and a trigger function A(x) that embeds a pattern into input x (in this lesson's toy example, setting trigger=1), a backdoor attack solves approximately:

minimize  L_clean( f_theta, D_clean )  +  L_backdoor( f_theta, { (A(x), y_t) : x in D_poison } )
  • L_clean — standard training loss on the clean portion of the data, preserving normal accuracy
  • L_backdoor — loss encouraging f_theta(A(x)) = y_t for all x, regardless of x's true label
  • The attacker controls the mixing ratio and the poison subset D_poison subset of D_train

Gu et al. (2017, BadNets) demonstrated this is achievable in deep image classifiers with poison rates as low as a few percent of the training set, and Chen et al. (2017) showed blended, near-imperceptible triggers achieve similar success rates while being far harder for a human to spot by inspecting individual poisoned examples.

Detection and defence approaches

  • Spectral signatures (Tran et al., 2018) — poisoned examples, because they must be learned via a different pathway than clean examples of the same nominal label, tend to leave a detectable statistical signature in the model's internal feature representations, extractable via singular value decomposition of per-class activations.
  • Activation clustering (Chen et al., 2018) — clustering training examples by their internal activation patterns per predicted class; poisoned examples with a shared trigger tend to form a distinguishable sub-cluster within their nominal class.
  • Fine-pruning (Liu et al., 2018) — pruning neurons that are inactive on clean validation data (candidate backdoor-specific neurons), followed by fine-tuning on trusted clean data to recover any accuracy lost.
  • Certified/provable defences (Steinhardt et al., 2017) — bound the maximum achievable damage from a poisoning attack under an assumed maximum fraction of poisoned data, trading a worst-case guarantee for often-conservative practical accuracy.

None of these defences is complete against an adaptive, well-resourced attacker aware of the specific defence being used — an instance of the same arms-race dynamic covered in your opponent updates too.

Cost

Detecting a suspected backdoor after the fact typically requires re-examining the full training set for statistical outliers or influence-function anomalies — computationally expensive at the scale of modern datasets — which is why prevention (provenance tracking, restricting who can contribute to a training set, differential privacy during training) is generally preferred over post-hoc detection wherever it is feasible.

Key references

  • Gu, T., Dolan-Gavitt, B., & Garg, S. (2017). BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv:1708.06733.
  • Chen, X., et al. (2017). Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. arXiv:1712.05526.
  • Tran, B., Li, J., & Madry, A. (2018). Spectral Signatures in Backdoor Attacks. NeurIPS.
  • Steinhardt, J., Koh, P. W., & Liang, P. (2017). Certified Defenses for Data Poisoning Attacks. NeurIPS.

Current state

Backdoor detection remains an active and unsolved research area, particularly for large pretrained models fine-tuned on data of uncertain provenance — exactly the setting covered in the next lesson — where the sheer scale of training data makes exhaustive inspection impractical, and where a backdoor introduced during pretraining can persist through fine-tuning on an entirely different downstream task.

What to learn next