Learn AI Create the Future — with Pranay
Main site

AI in Cybersecurity

Detecting phishing from URLs and headers

A phishing link and email can be judged largely from its structure — the URL's shape and the message's technical headers — before anyone reads a single word of the actual content.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A phishing detector checks a link's structure and a message's technical details for signs of forgery. Before anyone even reads what it says.

Think about a suspicious letter, claiming to be from your bank. Before opening it, you might check the envelope. Does the return address match the bank's real branch? Is your name spelled correctly? Does the stamp look right? None of that requires reading a word inside. The envelope itself can already give it away.

A phishing email or a fake link has its own version of an envelope. It can be checked the exact same way.

Why it exists

Phishing is a fake message. It usually pretends to be from a bank, employer, or well-known service. It tricks someone into clicking a link, and entering their password on a fake site. Reading the message's wording for warning signs works. But wording is easy for an attacker to rewrite endlessly.

The link's structure, and the email's technical headers, are harder to fake convincingly. A phishing link often needs to imitate a real domain closely. examplebank-login-verify.security-check.top instead of the real examplebank.com. The attacker does not own the real domain. That mismatch is a structural signal, not a wording one. It survives no matter how the message text gets rewritten.

Email headers are hidden technical metadata attached to every email, like sender authentication records. They are even harder to forge, because mail servers themselves partly verify them, not only the sender. A forged "from" address that fails this verification is a strong, checkable signal. It is invisible to a reader who only looks at the message body.

How it works

A link arrives: "http://examplebank.com.verify-secure-login.ru/account"
        |
        v
  [ check the URL's structure, not its wording ]
    - is the real domain (.ru here) hidden behind a fake-looking prefix?
    - is it using a raw IP address instead of a name?
    - does the domain contain words like "login", "verify", "secure"?
    - is the site using a plain, unencrypted connection?
        |
        v
  Combine these into a risk score
        |
        v
  "This link's structure resembles known phishing patterns"

The same idea applies to email headers. Does the technical sender record match the claimed sender, regardless of the name displayed in the inbox?

A real example you have seen

Email providers like Gmail already show a warning banner on suspicious messages, often before you read a single line. That warning is usually driven by exactly this kind of structural analysis, not by understanding the message's meaning.

Remember this

  • A link's structure — the real domain hidden inside it, a raw IP, suspicious words — needs no reading at all.
  • Email headers carry technical, partly-verified sender information. It is much harder to fake than the visible message text.
  • This structural analysis is a fast first filter. Real systems combine it with content analysis of the message text.
  • A high risk score should quarantine a message for review, not silently delete it — a real system needs a human appeal path too.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn numpy

Minimal runnable code

Every URL below is entirely fabricated for teaching — none point to a real, live address. We turn each one into a small set of structural features, and train a decision tree to tell legitimate-looking links from phishing-shaped ones.

phishing_url_features.py
import re
import numpy as np
from sklearn.tree import DecisionTreeClassifier, export_text

def url_features(url):
    domain = re.sub(r"^https?://", "", url).split("/")[0]
    return [
        domain.count("."),                                       # subdomain depth
        1 if re.search(r"\d+\.\d+\.\d+\.\d+", domain) else 0,     # raw IP instead of a name
        1 if "@" in url else 0,                                   # @ can hide the real destination
        1 if url.startswith("https") else 0,
        sum(word in domain.lower() for word in
            ["login", "verify", "secure", "account", "update"]),
    ]

urls = [
    "https://www.examplebank.com/login",
    "https://mail.google.com/mail/u/0",
    "https://www.wikipedia.org/wiki/Phishing",
    "https://accounts.examplecorp.com/signin",
    "http://192.168.4.10/login/verify-account",
    "http://secure-account-update-examplebank.co.xyz/login",
    "http://examplebank.com.verify-secure-login.ru/account",
    "https://examplebank-login-verify.security-check.top/account",
]
labels = [0, 0, 0, 0, 1, 1, 1, 1]   # 1 = phishing, deliberately mixed with and without https

X = np.array([url_features(u) for u in urls])
y = np.array(labels)

model = DecisionTreeClassifier(max_depth=3, random_state=0)
model.fit(X, y)

test_urls = [
    "https://www.examplebank.com/account/statements",
    "https://accountverify-secure.examplebank-support.top/login",
]
for u in test_urls:
    features = np.array([url_features(u)])
    pred = model.predict(features)[0]
    print(f"{u}\n  -> {'PHISHING' if pred else 'looks legitimate'}")

print()
print(export_text(model, feature_names=["num_dots", "has_ip", "has_at",
                                         "is_https", "suspicious_words"]))
Output
https://www.examplebank.com/account/statements
  -> looks legitimate
https://accountverify-secure.examplebank-support.top/login
  -> PHISHING

|--- is_https <= 0.50
|   |--- class: 1
|--- is_https >  0.50
|   |--- suspicious_words <= 1.50
|   |   |--- class: 0
|   |--- suspicious_words >  1.50
|   |   |--- class: 1

What actually happened

The second test URL uses https, exactly like the legitimate one — and the tree still correctly caught it, because its domain, accountverify-secure.examplebank-support.top, contains three of the suspicious words checked for (verify, secure, account), while the real bank's domain contains none. The tree learned to combine two signals, not rely on one.

Read the printed tree from the top. It first checks is_https, then, only for https links, checks suspicious_words. This training set was built deliberately with phishing examples both with and without https, specifically so the model could not get away with the single, outdated rule "no https means phishing."

  • re.sub(r"^https?://", "", url).split("/")[0] extracts only the domain part of the URL — feature extraction is doing real work before the model ever sees anything.
  • sum(word in domain.lower() for word in [...]) counts how many suspicious words appear anywhere in the domain — True/False values sum as 1/0 in Python, a small trick worth knowing.
  • max_depth=3 keeps the tree shallow and readable for this lesson. A real phishing classifier tunes this against a validation set, not by eye.

Common mistakes

Trusting "https present" as a strong signal on its own. This used to be a reasonably useful heuristic. It is far weaker today, because free, automated certificate authorities made HTTPS essentially free for anyone to add to any domain, including a phishing one set up minutes ago. This lesson's own toy tree still leans on it first, purely because of how small the training set is — a real system would not.

Using domain age or registration data without checking it stays current. A domain's age at the moment of classification matters, but domain registration data changes, and a stale lookup can silently misinform a model for months.

Ignoring homoglyph attacks. A domain using a lookercase "L" instead of an "I" (paypa1.com), or characters from a different alphabet that render identically, can defeat simple substring checks like the one used here. Dedicated homoglyph-detection libraries exist specifically because plain string matching misses these.

Treating this as a complete phishing detector. URL structure is one signal among several. Real systems combine this with the message content (text classification), sender header verification, and often a live check of where the link actually leads once clicked, since a phishing site can look different at click-time than it did when first observed.

Try it yourself

Add a URL using a homoglyph trick: "https://www.exarnplebank.com/login" (note "rn" standing in for "m") to the test set. Notice the current feature set does not catch it at all — none of the five features would flag it, which is exactly why real systems need more than domain word-matching alone.

What to learn next

Researcher — Mathematics and papers.

Feature families for phishing and email-security detection

  • Lexical URL features — domain length, digit ratio, hyphen count, subdomain depth, presence of an IP literal, presence of @, brand-name string similarity to known targets (edit distance or embedding similarity to a protected brand list).
  • Host-based features — domain age (via WHOIS), DNS record characteristics, hosting ASN reputation, TLS certificate issuance date and issuer, geographic hosting location relative to the claimed organisation.
  • Email authentication protocols — SPF (Sender Policy Framework), DKIM (DomainKeys Identified Mail), and DMARC (Domain-based Message Authentication, Reporting and Conformance) are cryptographic and DNS-based standards that let a receiving mail server verify whether a message's claimed sending domain actually authorised it. A failed or absent SPF/DKIM/DMARC check on a message claiming to be from a well-known domain is one of the highest-precision signals available, because it depends on infrastructure the attacker does not control, unlike message wording.
  • Visual/perceptual similarity — comparing a suspected phishing page's rendered screenshot or DOM structure against known legitimate login pages using perceptual hashing or vision models (Abdelnabi et al., 2020, VisualPhishNet), catching visual brand impersonation even when the URL itself has no obvious lexical similarity to the real one.

Homoglyph and IDN attacks, formally

Internationalized Domain Names (IDNs) permit non-ASCII characters, encoded via Punycode. This enables domains that render visually identical to a target brand while being byte-for-byte distinct — аpple.com (Cyrillic а) versus apple.com (Latin a) render identically in many browsers but are entirely different domains at the DNS level. Detection requires checking the decoded Unicode codepoints against confusable-character tables (Unicode's own TR39 confusables data is the standard reference), not only visual inspection or raw string comparison, which is exactly why the string-matching approach in this lesson's code cannot catch it.

Evaluation and public benchmarks

PhishTank and OpenPhish are widely used, continuously updated feeds of confirmed phishing URLs, commonly used to build and evaluate research classifiers; both require careful, time-aware evaluation, since a classifier trained and tested on the same time period will overstate real-world performance for the same reasons discussed in your opponent updates too — phishing infrastructure has a short operational lifetime (often hours to a few days) before being taken down and replaced.

Cost

URL and header-based filtering is deliberately cheap and applied to essentially all inbound mail and web traffic at the gateway level, before any expensive content analysis; content-based NLP classification and, where used, live page-rendering checks are reserved for messages that survive the cheap first filter, mirroring the cascade architecture seen throughout this section (cheap filter first, expensive check second).

Key references

  • Abdelnabi, S., Krombholz, K., & Fritz, M. (2020). VisualPhishNet: Zero-Day Phishing Website Detection by Visual Similarity. ACM CCS.
  • Sahingoz, O. K., et al. (2019). Machine Learning Based Phishing Detection from URLs. Expert Systems with Applications — a widely cited lexical-feature survey and benchmark.
  • Unicode Consortium. UTS #39: Unicode Security Mechanisms — the formal reference for confusable-character (homoglyph) detection.
  • APWG (Anti-Phishing Working Group). Phishing Activity Trends Reports — periodic industry reporting on phishing infrastructure patterns.

Current state

Large language models are increasingly used to generate highly convincing, grammatically flawless phishing text at scale, which is measurably eroding the value of the wording-based content signals phishing detectors have historically relied on most heavily — pushing more relative weight back onto the structural, header, and infrastructure signals covered in this lesson, precisely because those are harder for a language model to fake.

What to learn next