What hospital data actually looks like
Hospital data is written by busy humans under pressure, so it arrives late, in the wrong units, and full of gaps that mean something.
- 9 min read
- 3 reading levels
- Published
Read these first
On this page 6
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Hospital data is messy on purpose — not because someone was careless, but because it was written by busy humans mid-emergency.
Think about a doctor's handwritten prescription. The handwriting is rushed. Some words are shorthand only other doctors understand. A pharmacist sometimes has to call and ask what a scribble means.
Digital hospital data has the same problem, wearing a cleaner font. A nurse typing vitals during a code blue is not thinking about your machine learning pipeline. She is trying to keep a patient alive.
Why it exists
Most datasets used to teach machine learning are already clean. Someone collected them for research, on purpose, with a plan.
Hospital data is a byproduct. It exists because care had to be documented, billed, and legally recorded. Nobody built it as a tidy table for you.
That difference changes everything downstream. A missing value in a spreadsheet competition might mean "not collected." A missing value in a patient chart can mean several different things. Maybe the nurse was too busy to record it. Maybe the test was never ordered, because the doctor already knew the answer. Maybe the patient left before anyone could measure it. Each of those is a different fact about the world, hiding behind the same blank cell.
How it works
A single patient visit touches many separate systems, each with its own habits.
Vitals monitor -> numbers, every few minutes, sometimes every few hours
Nurse's notes -> free text, typed in a hurry, full of abbreviations
Lab system -> numbers with units that vary by machine and by hospital
Billing codes -> official diagnosis codes, entered for insurance, not accuracy
|
v
one messy patient recordFour things trip up almost every beginner working with this kind of data:
- Missing is not random. A blood test that was never ordered often means the doctor already felt confident, one way or another.
- Units are not standardised. Weight might be kilograms in one system and pounds in another, entered into the same column.
- Timing is not synchronised. A lab result and a vitals reading taken "at the same time" might really be twenty minutes apart.
- Codes are not the diagnosis. A billing code was chosen to get a claim paid, not to summarise a patient's condition for a future researcher.
Where you have already seen it
You have never seen raw hospital data — and that is the point. Every health app you use has already cleaned it for you.
- Your lab report PDF hides which values came from a machine and which a technician typed in by hand.
- A pharmacy app flags a drug interaction only after someone mapped messy drug names to one standard list.
- Insurance claim approvals run on billing codes, the same imperfect codes described above.
An honest warning
Cleaning hospital data usually takes more time than training the model. That order of effort feels backwards to a beginner, and it is not.
A model trained on badly understood data will look accurate in testing and fail on real patients. Understanding why a value is missing matters more than filling in that blank.
Remember this
- Hospital data is a byproduct of patient care, not a dataset built for you.
- A missing value can carry real information — do not fill it in without asking why it is missing.
- Units, timing, and billing codes vary between systems in the same hospital.
What to learn next
- Handling missing data — the general techniques this lesson's warnings point toward.
- Irregular clinical time series — what happens when readings also arrive at unpredictable times.
- Data leakage and results that are too good — the general version of the leakage risk this section keeps returning to.
Developer — Code and libraries.
Setup
pip install pandasMinimal runnable code
Five rows, built to look like a real extract straight out of a hospital system — before anyone has cleaned it.
import pandas as pd
# One row per hospital visit. This is what a real extract looks like
# before anyone cleans it.
records = pd.DataFrame([
{"patient_id": "P001", "age": 67, "height_cm": 171.0, "systolic_bp": 138, "note": "stable, tolerating diet"},
{"patient_id": "P002", "age": 54, "height_cm": None, "systolic_bp": 165, "note": "c/o SOB, started on O2"},
{"patient_id": "P003", "age": 71, "height_cm": 66.0, "systolic_bp": None, "note": "no acute distress"},
{"patient_id": "P004", "age": 45, "height_cm": 180.0, "systolic_bp": 121, "note": None},
{"patient_id": "P005", "age": 200, "height_cm": 165.0, "systolic_bp": 118, "note": "afebrile, ambulatory"},
])
print("missing values per column:")
print(records.isna().sum().to_string())
print()
# A real adult height is roughly 120-215 cm. Anything outside that is
# either a typo, or recorded in the wrong unit (inches instead of cm).
suspect_height = records[(records["height_cm"] < 120) | (records["height_cm"] > 215)]
print("rows with an impossible height in cm:")
print(suspect_height[["patient_id", "height_cm"]].to_string(index=False))
print()
# Ages above 120 cannot be real. This one is almost certainly a data-entry error.
suspect_age = records[records["age"] > 120]
print("rows with an impossible age:")
print(suspect_age[["patient_id", "age"]].to_string(index=False))missing values per column:
patient_id 0
age 0
height_cm 1
systolic_bp 1
note 1
rows with an impossible height in cm:
patient_id height_cm
P003 66.0
rows with an impossible age:
patient_id age
P005 200What actually happened
records.isna().sum() counts real blanks — but real blanks are only part of the mess. P003's height of 66 is not blank and not wrong on its face. It is almost certainly 66 inches (about 168 cm), entered into a column labelled centimetres by someone used to a different unit.
records[records["age"] > 120] catches the kind of error a human reads past without noticing: 200 looks like a typo for 20, or a placeholder value nobody removed, or two digits transposed. A plausibility check like this — a simple, sane range for a value — catches more real problems in hospital data than a fancier statistical outlier test.
Line by line, the parts that are not obvious:
.isna().sum()only counts values that are trulyNaN. It says nothing about values that are present, syntactically valid, and still wrong — like the 66 cm height. Those need explicit range checks, not a missing-value check..to_string(index=False)is used here to keep the printed table readable without a row-number column. It does not change the underlying data.- The height check uses
120and215as boundaries, not0and300. A boundary check is only as good as the plausible range a person chose for it — usually a clinician, not a data scientist.
Common mistakes
Assuming a clean-looking value is a correct value. 66 parses as a valid float. Nothing in the file format flags it as wrong. Only domain knowledge — that adult heights do not go that low in centimetres — catches it.
Dropping every row with a missing value. In this five-row example that would throw away three of five patients. In a real dataset it silently removes exactly the sickest patients, who tend to have the most incomplete charts because staff were busy treating them.
Treating a billing code as ground truth. A diagnosis code says what got the claim paid, decided sometimes weeks after the visit. It often does not match what happened in the room.
Try it yourself
Add a sixth row with "height_cm": 0.0. Run the range check again, and notice that 0.0 is not caught by .isna() — it is a real, present number that is still clinically impossible.
What to learn next
- Handling missing data — general strategies once you know why a value is missing.
- Data leakage and results that are too good — what happens next if messy data like this is not caught before training.
- Irregular clinical time series — the timing half of this same problem.
Researcher — Mathematics and papers.
Data types in a clinical record
A modern electronic health record (EHR) mixes several structurally different data types in one patient timeline:
Structured numeric - vitals, lab values, medication doses
Structured coded - ICD-10 diagnosis codes, SNOMED CT findings, LOINC lab identifiers
Semi-structured - order entries, flowsheets
Unstructured text - clinical notes, discharge summaries, radiology reportsInteroperability standards exist to make these comparable across institutions: HL7 FHIR (Fast Healthcare Interoperability Resources) defines the current exchange format most US and many international systems are migrating toward; OMOP CDM (Observational Medical Outcomes Partnership Common Data Model) is the dominant standard for pooling multi-site data into one schema for research.
Missingness mechanisms
Rubin's (1976) taxonomy of missing-data mechanisms is the standard framework, and it matters directly for how a missing clinical value should be handled:
MCAR - Missing Completely At Random: P(missing) is independent of any data, observed or not.
MAR - Missing At Random: P(missing) depends only on OBSERVED data.
MNAR - Missing Not At Random: P(missing) depends on the UNOBSERVED value itself.P(missing)— the probability that a given value was not recorded.
A troponin test that is only ordered when a clinician already suspects a heart attack is a textbook MNAR mechanism: the act of measuring is caused by an unobserved belief about the very quantity being measured. Standard imputation methods (mean, k-nearest-neighbour, MICE) assume MAR. Applied to MNAR clinical data, they can quietly encode the clinician's judgement into what looks like a neutral feature — a specific instance of the leakage covered in label leakage in clinical models.
Cost and scale
Real-world EHR extracts are wide and sparse. A single tertiary-hospital dataset commonly has:
| Property | Typical order of magnitude |
|---|---|
| Distinct lab/vital feature codes | 10^3 - 10^4 |
| Fraction of (patient x feature x time-bin) cells populated | often under 5% |
| Free-text notes per admission | 10 - 100 |
The sparsity is structural, not accidental — it reflects clinical decisions about what to measure and when, which is exactly what makes naive imputation risky.
Key references
- Rubin, D. (1976). Inference and Missing Data. Biometrika 63(3). The original MCAR/MAR/MNAR framework.
- Johnson, A. et al. (2016). MIMIC-III, a freely accessible critical care database. Scientific Data 3. The standard open ICU dataset, and the one most published clinical ML papers validate against.
- Weber, G. et al. (2020). Finding the Missing Link for Big Biomedical Data. JAMA. On OMOP and multi-site harmonisation.
- Hripcsak, G. & Albers, D. (2013). Next-generation phenotyping of electronic health records. JAMIA. On why raw EHR data reflects care processes, not underlying biology.
What to learn next
- Handling missing data — MCAR/MAR/MNAR-aware imputation in practice.
- Target leakage: features that contain the answer — the general form of the MNAR trap above.
- Irregular clinical time series — extending this to the timing dimension.