Doing the Work Section 021
Data Leakage and Results That Are Too Good
A score that looks amazing is usually a bug, not a breakthrough. Here is how to hunt down the leak before someone else finds it.
11 of 11 lessons published Three reading levels on every lesson
Lessons in order
Work top to bottom. Each lesson assumes the one above it.
- Your model scores 99%: what to suspect first
- Target leakage: features that contain the answer
- Duplicate rows across your splits
- Fitting the scaler before splitting
- Using the future to predict the past
- The same person in train and test
- Leakage that survives cross-validation
- Choosing features on the full dataset
- Overfitting your own test set
- Did the model already see your test set?
- Hunting a leak with ablations