Learn AI Create the Future — with Pranay
Main site

Baselines and Choosing a Model

Logistic regression, gradient boosting, or a neural net?

For data in tables, the honest ranking is boring — linear models when signal is simple, gradient boosting as the default, neural nets only with a specific reason.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

For data that lives in tables, start linear, default to gradient boosting, and demand a specific reason before reaching for a neural net.

Choosing a vehicle depends on the road. A bicycle wins in a crowded lane, a truck wins on the highway, and a Ferrari loses badly on a village road. Expense does not override terrain. Model families are vehicles, and your data is the terrain.

Tabular data — rows and columns, like a spreadsheet of customers or transactions — is specific terrain. And on this terrain, the glamorous vehicle is usually the wrong one.

Why it exists

Deep learning's fame comes from images, sound, and language. It earned that fame honestly. The mistake is assuming fame transfers to spreadsheets.

Tables are different terrain: a few hundred columns with individual meaning, sharp thresholds ("risk jumps above age 60"), and quirky interactions. Tree-based models — which cut data with thresholds all day — are built for exactly this. Neural nets can learn it, but spend enormous effort re-deriving what trees get by construction.

How it works

The honest decision ladder for tables:

START    logistic / linear regression
         → cheap, explainable, surprisingly strong
         → if it nearly matches fancier models, keep it

DEFAULT  gradient boosting (XGBoost, LightGBM, HistGB)
         → wins most tabular contests, handles messy data

JUSTIFY  neural network — pick it only for a named reason:
         → text/image columns mixed into the table
         → need embeddings shared with another system
         → millions of rows AND the boosted trees plateaued

The ladder is also an experiment plan: run all three rungs, compare honestly, and let the gaps — not the hype — pick the winner.

A real example you have seen

Kaggle competitions on tabular data cover insurance claims, credit risk and sales forecasts. Gradient boosting has won them so consistently, for over a decade, that it is a community joke. The same pattern holds inside banks and insurers, where trees plus careful features remain the quiet workhorse.

Remember this

  • Tabular data is its own terrain, and the terrain picks the model.
  • Ladder: linear first, gradient boosting by default, neural nets with a named reason.
  • Run all rungs; the measured gap decides, not the newest paper.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7 and numpy 1.26. Runs in under a minute on CPU.

Same three models, two terrains

The ladder, raced on a real dataset and on one built to punish linear models.

tabular_ladder.py
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

def contenders():
    return {
        "logistic regression": make_pipeline(StandardScaler(),
                                             LogisticRegression(max_iter=5000)),
        "gradient boosting":   HistGradientBoostingClassifier(random_state=0),
        "neural net (MLP)":    make_pipeline(StandardScaler(),
                                             MLPClassifier(hidden_layer_sizes=(64, 32),
                                                           max_iter=2000, random_state=0)),
    }

cv = KFold(n_splits=5, shuffle=True, random_state=0)

# Dataset 1: small, clean, mostly linear signal (real tumour measurements).
X1, y1 = load_breast_cancer(return_X_y=True)

# Dataset 2: the signal lives in interactions between features.
rng = np.random.default_rng(0)
X2 = rng.normal(size=(2000, 6))
y2 = ((X2[:, 0] * X2[:, 1] > 0) ^ (X2[:, 2] > 0.5)).astype(int)

for title, X, y in [("clean and small (569 rows)", X1, y1),
                    ("interaction-heavy (2000 rows)", X2, y2)]:
    print(title)
    for name, model in contenders().items():
        scores = cross_val_score(model, X, y, cv=cv)
        print(f"  {name:20s} {scores.mean():.3f} +/- {scores.std():.3f}")
    print()
Output
clean and small (569 rows)
  logistic regression  0.972 +/- 0.017
  gradient boosting    0.972 +/- 0.020
  neural net (MLP)     0.979 +/- 0.013

interaction-heavy (2000 rows)
  logistic regression  0.505 +/- 0.030
  gradient boosting    0.991 +/- 0.006
  neural net (MLP)     0.952 +/- 0.009

The walkthrough

Terrain one: a three-way tie. All models land within each other's error bars (the +/- values). When the ladder ties, the bottom rung wins — logistic regression is the fastest, smallest, and most explainable of the three, and nothing here justifies paying more.

Terrain two: logistic regression scores a coin flip. The label is built from a product and an XOR — pure interaction, no linear direction to find. Gradient boosting reaches 0.991 by stacking threshold cuts; the MLP gets close at 0.952 but needed scaled inputs and more compute to lose anyway.

Real tables sit between the terrains. Which is why you race the ladder instead of guessing — the race costs minutes and replaces a religion with a measurement.

Note what the pipelines quietly teach: the linear model and MLP require StandardScaler; the boosted trees run raw. Preprocessing sensitivity is itself a model-choice criterion — fewer preprocessing steps means fewer training-serving skew opportunities.

Common mistakes

Reading a leaderboard from another dataset as advice for yours. The whole point of the two-terrain demo: rankings flip with the data. Race on your rows.

Giving the neural net a handicap and calling it a fair race. Unscaled inputs, default epochs, no tuning — the MLP loses rigged races. Each rung deserves its standard care: scaling for linear and neural, modest depth tuning for trees.

Ignoring the +/-. On 569 rows, five folds swing by two points. Declaring a winner inside the noise band is the mistake a fair comparison exists to prevent.

Forgetting total cost. On terrain one the MLP leads boosting by 0.007 — well inside both error bars. Giving that 0.007 up buys you: no scaler to maintain, native missing-value handling, and faster retraining. Score the whole bill, not the accuracy line.

Try it yourself

Add a fourth contender: RandomForestClassifier(n_estimators=200). Predict where it lands on both terrains before running. Then break terrain two's trees: rotate the features with a random 6x6 matrix multiplication and re-race. Axis-aligned threshold cuts suddenly struggle — and you will have discovered why trees love tables in their natural coordinates.

What to learn next

Researcher — Mathematics and papers.

The empirical record

The tabular question has been benchmarked to unusual depth:

  • Grinsztajn, Oyallon, Varoquaux (2022), Why do tree-based models still outperform deep learning on typical tabular data? (NeurIPS D&B): across 45 mid-size datasets, tuned GBDTs beat tuned deep models. Their ablations locate three tree advantages: robustness to uninformative features, preservation of non-smooth/axis-aligned structure, and insensitivity to feature rotation — the exact property your rotation exercise breaks.
  • Shwartz-Ziv & Armon (2022), Tabular Data: Deep Learning is Not All You Need (Information Fusion): claimed neural wins often fail to transfer beyond the papers' own curated datasets.
  • McElfresh et al. (2023), When Do Neural Nets Outperform Boosted Trees? (NeurIPS D&B): on many datasets the family gap is smaller than the tuning gap — hyperparameter budget matters more than family, except on datasets with high irregularity where trees pull ahead.
  • TabPFN (Hollmann et al., 2023, ICLR; v2 in Nature, 2025): a transformer pre-trained on synthetic tabular tasks, performing in-context learning; state of the art on small (≤10k rows) numeric datasets and a genuine exception to the boosted-tree default in that regime.

Why trees fit tables: inductive-bias analysis

Tabular targets are frequently non-smooth in individual features (thresholds, saturations) with heterogeneous units. Trees are invariant to monotone feature transforms and fit piecewise-constant functions natively. MLPs carry a spectral bias toward low-frequency (smooth) functions (Rahaman et al., 2019, On the Spectral Bias of Neural Networks, ICML), so sharp axis-aligned structure costs them capacity and data. Rotation invariance cuts the other way: an MLP is rotation-equivariant while trees are not — which is precisely why real tables, whose coordinates are individually meaningful (not arbitrary rotations), favour trees.

When the neural rung earns its keep

Named reasons with literature behind them: mixed-modality columns (text/images alongside tabular — end-to-end fine-tuning wins over feature extraction), entity embeddings reused across tasks (Guo & Berkhahn, 2016), very large datasets where GBDT training becomes the bottleneck, and in-context/few-shot regimes (TabPFN). Retrieval-augmented and attention-based tabular architectures (SAINT, FT-Transformer — Gorishniy et al., 2021) narrow the gap but, per the benchmarks above, have not overturned the default.

Comparison hygiene

Family comparisons are exquisitely sensitive to tuning budget asymmetry; equalise search budgets (e.g., equal Optuna trials per family) and report variance across folds and seeds. The 5-fold +/- shown in the developer block understates full variance — it omits seed variance of the models themselves; see seed variance and the paired-comparison protocol in comparing models fairly.

What to learn next