Security telemetry
Every login, connection and file access leaves a record — security telemetry is that flood of records, and it is the raw material every security model is built from.
- 10 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Security telemetry is the constant stream of records a computer system keeps about what is happening inside it. Every login, every connection, every file opened.
Think about the register at an apartment gate or office reception. Every visitor writes down their name, the time, and which flat they are visiting. Nobody reads that register cover to cover every day. It sits there quietly. Then one day, someone needs to answer "who came in around 9pm last Tuesday?" It is the only record that can answer that question.
Computers keep a register like this for almost everything they do. That register is telemetry.
Why it exists
Before you can catch anything suspicious, you need a record that something happened at all. Security telemetry is that record. Login attempts, network connections, files opened, programs started, DNS lookups made. It gets collected continuously from laptops, servers, and network devices across an organisation.
This sounds simple, and is quietly enormous. A mid-sized company can generate many gigabytes of these records every single day. Almost none of it is interesting alone. One login, one file open, one connection is completely ordinary. The entire point of this section is finding the rare interesting needle, inside this very large, very boring haystack.
Without reliable telemetry, no model in this section has anything to learn from. This is the unglamorous foundation the whole field is built on.
How it works
Raw event on a laptop or server
|
v
"2024-05-01T08:12:03Z auth_failure user=alice src_ip=10.0.0.5"
|
v
[ parsed into structured fields ]
|
v
timestamp | event | user | source IP
08:12:03 | auth_failure | alice | 10.0.0.5
|
v
Millions of these rows, per day, feed everything downstream:
rules, dashboards, and every model in this section.A single log line means nothing. A pattern across thousands of them starts to mean something. The same user failing to log in four times in twenty seconds, for example.
A real example you have seen
Your phone or laptop sometimes asks "new sign-in detected from a device we don't recognise, was this you?" That alert exists because a login event was recorded. It was compared against your normal pattern, and flagged as unusual. Telemetry is the record. Everything else in this section builds on top of it.
Remember this
- Telemetry is the raw record of what happened — logins, connections, file access — collected before any analysis begins.
- Almost none of it is individually interesting. The signal lives in patterns across huge volumes of otherwise ordinary events.
- Every model in this section depends on telemetry being collected reliably and parsed correctly, before any modelling can start.
- A gap in telemetry collection is itself a security risk, and needs its own engineering review, not only the models built on top of it.
What to learn next
- Triaging alerts at scale — what happens once telemetry starts producing more alerts than humans can read.
- Parsing logs before you can model them — a deeper look at turning messy raw text into structured data.
- Regex for text — the general tool this lesson's parsing code is built on.
Developer — Code and libraries.
Setup
python3 --versionNo installs needed — this example uses only the Python standard library, on purpose. Log parsing is often the first real code a security data project needs, long before any model.
Minimal runnable code
We turn raw authentication log lines into structured records, then count failed logins per user — the simplest possible telemetry-to-signal pipeline.
import re
from collections import Counter
raw_log_lines = [
"2024-05-01T08:12:03Z auth_failure user=alice src_ip=10.0.0.5",
"2024-05-01T08:12:11Z auth_failure user=alice src_ip=10.0.0.5",
"2024-05-01T08:12:19Z auth_failure user=alice src_ip=10.0.0.5",
"2024-05-01T08:12:27Z auth_failure user=alice src_ip=10.0.0.5",
"2024-05-01T08:13:02Z auth_success user=alice src_ip=10.0.0.5",
"2024-05-01T08:20:10Z auth_failure user=bob src_ip=10.0.0.9",
"2024-05-01T08:45:00Z auth_success user=carol src_ip=10.0.0.2",
"2024-05-01T09:01:00Z auth_failure user=bob src_ip=10.0.0.9",
]
LOG_PATTERN = re.compile(
r"(?P<timestamp>\S+) (?P<event>\S+) user=(?P<user>\S+) src_ip=(?P<src_ip>\S+)"
)
events = []
for line in raw_log_lines:
match = LOG_PATTERN.match(line)
if match:
events.append(match.groupdict())
print(f"parsed {len(events)} of {len(raw_log_lines)} raw lines into structured records")
print("first record:", events[0])
print()
failed_logins = Counter(e["user"] for e in events if e["event"] == "auth_failure")
print("failed login attempts per user:")
for user, count in failed_logins.most_common():
flag = " <- unusually high, worth a closer look" if count >= 4 else ""
print(f" {user}: {count}{flag}")parsed 8 of 8 raw lines into structured records
first record: {'timestamp': '2024-05-01T08:12:03Z', 'event': 'auth_failure', 'user': 'alice', 'src_ip': '10.0.0.5'}
failed login attempts per user:
alice: 4 <- unusually high, worth a closer look
bob: 2What actually happened
The named groups in LOG_PATTERN — (?P<timestamp>...), (?P<user>...) — turn a plain text line into a Python dictionary via match.groupdict(), with no manual string-splitting logic. That structured dictionary is what makes counting, filtering and aggregating possible; the raw text string by itself is not directly usable by anything downstream.
Counter did the actual security-relevant work in one line: turning eight scattered log lines into a per-user count that instantly shows alice's four failed attempts as the standout.
LOG_PATTERN.match(line)returnsNoneif the line does not fit the expected shape, which is why the code checksif match:before using it — real logs contain malformed lines, and a parser that assumes every line matches will eventually crash on production data.\S+(non-whitespace) is doing the heavy lifting here because this toy log format is space-separated. Real log formats are rarely this clean — see the common mistakes below.
Common mistakes
Assuming one log format for an entire organisation. A Windows server, a Linux server, a firewall and a cloud provider each emit different formats. Real pipelines normalise many formats into one common schema before analysis — a step usually bigger than any model built afterward.
Ignoring clock skew. If two servers' clocks are a few seconds apart, events that were simultaneous appear minutes apart when timestamps are compared naively. Security telemetry pipelines synchronise clocks (commonly via NTP) specifically so timestamp-based patterns like this lesson's stay meaningful.
Treating a single failed login as suspicious. People mistype passwords constantly. The signal here is volume and speed — four failures in 24 seconds is a different fact than four failures spread across a normal workday. The next lesson builds directly on this.
Dropping unparseable lines silently. The if match: check here quietly skips lines that do not fit the pattern. In a real pipeline, silently dropped lines should be counted and alerted on — a sudden rise in unparseable logs is itself sometimes a sign that a log source changed, or was tampered with.
Try it yourself
Add three more auth_failure lines for a new user, dave, spaced 40 minutes apart instead of seconds apart. Notice the current code still flags dave at the >= 4 threshold, even though 40-minute gaps look nothing like alice's rapid-fire pattern — a limitation the next lesson addresses directly.
What to learn next
- Regex for text — a fuller treatment of the parsing technique used here.
- Parsing logs before you can model them — messier, more realistic log formats.
- Triaging alerts at scale — what happens once this kind of counting produces more flags than a team can review.
Researcher — Mathematics and papers.
Telemetry sources and their characteristics
Security telemetry is not one data type. Each source has a distinct structure, volume and modelling implication:
- Authentication logs (Windows Event Log 4624/4625, Linux
auth.log, SSO/IdP logs) — relatively low volume, high signal, the basis for brute-force and credential-stuffing detection. - Network flow data (NetFlow/IPFIX, Zeek/Bro connection logs) — summarised connection metadata (source, destination, ports, bytes, duration) rather than full packet content, chosen specifically because full packet capture at scale is rarely storable or necessary.
- Endpoint Detection and Response (EDR) telemetry — process creation, file writes, registry changes, DLL loads — very high volume and detail, the primary data source for the malware and behavioural detection covered later in this section.
- DNS logs — every domain name resolution requested by every host, valuable because malware and phishing infrastructure both depend on DNS, covered further in detecting phishing from URLs and headers.
The volume problem, quantified structurally
A single endpoint running EDR instrumentation can generate a meaningful stream of discrete events per minute during normal use — process starts, file handles, network connections. Multiplied across an organisation's full fleet of endpoints and servers, this routinely produces telemetry volumes in the range of hundreds of gigabytes to multiple terabytes per day for a large enterprise. This is why security telemetry pipelines are built on log-oriented data infrastructure (Kafka-style streaming ingestion, columnar storage such as Parquet, and purpose-built platforms like Elastic/Splunk/a SIEM) rather than a conventional relational database — see streaming data and ETL for ML for the general infrastructure this specialises.
Schema normalisation
Because telemetry arrives from heterogeneous sources in heterogeneous formats, most mature security data platforms adopt a common event schema — the Open Cybersecurity Schema Framework (OCSF), backed by AWS, Splunk, and others since 2022, is the current leading open effort — mapping vendor-specific fields (a Windows event ID, a Zeek connection record, a cloud audit log entry) onto a shared set of field names and event categories before any downstream model or rule ever sees the data.
Cost
Raw telemetry retention is a first-order cost driver in a security program, not a footnote: storing months of full-fidelity data across thousands of endpoints is expensive enough that most organisations apply tiered retention — recent data at full fidelity, older data summarised or sampled — and this retention policy directly bounds how far back any later investigation or model retraining can look.
Key references
- OCSF Project. Open Cybersecurity Schema Framework — schema.ocsf.io — the current leading open standard for telemetry normalisation.
- MITRE. ATT&CK Data Sources — attack.mitre.org/datasources — maps specific telemetry types to the attacker techniques they can detect, the standard reference for "what telemetry do I need to catch what."
- Bejtlich, R. (2013). The Practice of Network Security Monitoring. No Starch Press — a practitioner-level treatment of telemetry collection strategy that remains widely referenced.
Current state
Telemetry volume continues to grow faster than the budget most organisations allocate to store and process it, which is the direct structural reason the rest of this section leans so heavily on unsupervised and streaming techniques (covered in later lessons) rather than approaches assuming a full, permanently retained historical dataset is available for training.
What to learn next
- Parsing logs before you can model them — the next layer of dealing with real, messy telemetry.
- Streaming data — the general infrastructure pattern telemetry pipelines are built on.
- Triaging alerts at scale — turning telemetry-derived signals into a workable analyst workflow.