Neural operators

A neural operator learns the general solving step for a whole family of physics problems, so one trained network answers any new case instantly.

By Pranay Mahendrakar
12 min read
3 reading levels
Published 2 Sep 2026

Read these first

On this page 6

Why it exists
How it works
Where this shows up
An honest warning
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

A neural operator learns the general solving step itself, so one trained network can answer any new version of the problem instantly.

Think about the difference between memorising the answer to one specific maths problem and learning the method that solves every problem of that type. A student who only memorises "7 times 8 is 56" cannot answer "7 times 9." A student who learns the multiplication method can answer any pair of numbers. A neural operator is trained to be that method, for physics, instead of memorising one specific case.

Why it exists

The physics-informed neural network from the previous lesson solved one equation, for one starting condition. Change the starting condition — a different initial temperature, a different starting shape — and that trained network is no longer useful. It has to be retrained from scratch for the new case.

Real engineering rarely needs the answer to exactly one case. A structural engineer wants to know how a bridge responds to many different possible loads, not only one. A weather model needs to run forward from many different starting atmospheric states, not one fixed morning. Retraining a fresh network for every single new starting condition, at real training cost each time, does not scale to that kind of use.

A neural operator is trained once, on many different versions of the input. It learns the mapping itself, from "whatever the starting condition is" to "the resulting solution." Once trained, it takes a brand-new starting condition it has never seen, and produces the answer in a single fast pass — no retraining involved.

How it works

Many training examples of:  (a starting condition)  ->  (the resulting solution)
        |
        v  train once, across ALL of them
A trained neural operator — has learned the RULE that connects them
        |
        v
Brand-new starting condition, never seen before
        |
        v
Neural operator (single fast pass, no retraining)
        |
        v
Predicted solution

The key difference from an ordinary surrogate model is what varies at prediction time. A surrogate model answers "what is the outcome for this one design?" A neural operator answers "what is the outcome for this entire function describing the starting condition?" It can do this for a starting condition it has never encountered.

Where this shows up

Weather forecasting groups have used neural operators to jump from a snapshot of today's atmosphere straight to a forecast. Once the operator has been trained across many historical days, it needs no re-run of the full traditional weather-simulation software for that specific day.

Engineers use them to predict how heat spreads through many differently shaped or initially heated objects. One trained network does the job that would otherwise take one network per shape.

An honest warning

A neural operator learns from the family of examples it was trained on. Give it a starting condition wildly different in kind from anything in that family — a shape, material, or scale it never saw a version of. It extrapolates exactly as blindly as any other learned model. Nothing about the "operator" framing makes it immune to that.

Remember this

A neural operator learns the general rule connecting a starting condition to its outcome, across many training examples, rather than memorising one fixed case.
Once trained, it answers brand-new starting conditions instantly, with no retraining needed for each new case.
It still only knows what it was trained on — an unfamiliar kind of input is still extrapolation, with all the same risks.

What to learn next

Physics-informed neural networks — solving one case at a time, the building block this lesson generalises.
Machine-learned interatomic potentials — a physics domain where a smooth, differentiable learned model matters equally.
Transformers — another architecture built to generalise across many different inputs from one trained model.

Developer — Code and libraries.

The clearest way to see what makes this an "operator" rather than an ordinary regression: train it on many different starting values, then test it on one it has never seen.

Setup

bash

pip install torch

Minimal runnable code

The family of problems: dy/dx = -y, y(0) = a, for many different starting values a. The analytic solution y(x) = a * e^-x is used only to generate training data and to check the final answer — the network itself is never given a formula, only examples.

This uses a simplified version of the DeepONet architecture: a "branch" network reads the starting condition, a "trunk" network reads the query point, and their outputs are combined to produce the answer.

neural_operator.py

import torch
import torch.nn as nn

torch.manual_seed(0)

# The family of problems: dy/dx = -y, y(0) = a, for many different starting values a.
# Analytic solution (used only to generate training data and to check the answer):
#   y(x) = a * exp(-x)
#
# A PINN (previous lesson) solves ONE such equation. This network learns the
# OPERATOR: the rule that maps "starting value a" to "the whole solution curve",
# so a single trained network answers any new starting value with no retraining.

p = 8  # size of the shared code the two halves must agree on

branch_net = nn.Sequential(nn.Linear(1, 16), nn.Tanh(), nn.Linear(16, p))   # reads a
trunk_net = nn.Sequential(nn.Linear(1, 16), nn.Tanh(), nn.Linear(16, p))    # reads x
bias = nn.Parameter(torch.zeros(1))

params = list(branch_net.parameters()) + list(trunk_net.parameters()) + [bias]
optimizer = torch.optim.Adam(params, lr=0.01)

def deeponet(a, x):
    # a: (batch, 1) starting values.  x: (batch, 1) query points.
    return (branch_net(a) * trunk_net(x)).sum(dim=1, keepdim=True) + bias

# training data: many (a, x) pairs drawn from MANY different starting values
n_train = 500
a_train = torch.empty(n_train, 1).uniform_(0.5, 2.0)
x_train = torch.empty(n_train, 1).uniform_(0, 3)
y_train = a_train * torch.exp(-x_train)

for step in range(3000):
    optimizer.zero_grad()
    pred = deeponet(a_train, x_train)
    loss = ((pred - y_train) ** 2).mean()
    loss.backward()
    optimizer.step()
    if step % 750 == 0:
        print(f"step {step:4d}  loss={loss.item():.6f}")

print()
# test on a starting value the network NEVER saw during training (outside 0.5-2.0)
a_new = torch.tensor([[3.0]])
for x_val in [0.0, 1.0, 2.0]:
    x_q = torch.tensor([[x_val]])
    with torch.no_grad():
        pred = deeponet(a_new, x_q).item()
    analytic = 3.0 * (2.718281828 ** (-x_val))
    print(f"a=3.0 (unseen), x={x_val}:  network={pred:.4f}  analytic={analytic:.4f}")

Output

step    0  loss=0.097891
step  750  loss=0.000053
step 1500  loss=0.000009
step 2250  loss=0.000007

a=3.0 (unseen), x=0.0:  network=2.8173  analytic=3.0000
a=3.0 (unseen), x=1.0:  network=1.0381  analytic=1.1036
a=3.0 (unseen), x=2.0:  network=0.3761  analytic=0.4060

Walkthrough

The network trained on starting values a between 0.5 and 2.0 exclusively. a=3.0 never appeared once during training. It is genuinely new.

branch_net(a) encodes the starting condition into a code of size p. trunk_net(x) encodes the query location into a code of the same size. Multiplying them element-wise and summing — (branch_net(a) * trunk_net(x)).sum(...) — is a dot product, the same operation that scores how well two vectors align. This is the core DeepONet idea: the network learns to represent "conditions" and "query points" in a shared space where combining them produces the right answer.

The result at a=3.0 is close but not perfect — 2.82 versus the true 3.00 at x=0. That gap is real, and instructive: a=3.0 sits outside the training range of 0.5–2.0, so this is genuine extrapolation, and even a network built to generalise across a whole family degrades once it is asked about something meaningfully outside that family.

Common mistakes

Confusing this with a PINN. A PINN uses the physics equation as a training signal to solve one instance well with little data. This network uses ordinary supervised data — many (a, x, y) triples — to generalise across instances. The two ideas can be combined, but they solve different problems.

Judging "generalisation" without testing outside the training range. The network fits training-range values near-perfectly. The real test of an operator is exactly what this example does last: check a value the training data never included.

Using too small a code size p. If p is too small, the branch and trunk networks cannot represent enough distinct "conditions" or "query behaviours," and accuracy suffers uniformly across all inputs, not only unseen ones. p=8 is enough for this one-parameter family; richer families need a larger p.

Try it yourself

Change the training range to a_train = torch.empty(n_train, 1).uniform_(0.5, 4.0) so it includes 3.0, and rerun. Compare the new prediction at a=3.0 against the old one — the error should shrink substantially, since the network has now actually seen that region during training.

What to learn next

Physics-informed neural networks — the single-instance building block this lesson generalises across many instances.
Attention — a different mechanism for combining two learned representations, used heavily in modern operator architectures.
Machine-learned interatomic potentials — where smoothness of the learned function matters as much as accuracy.

Researcher — Mathematics and papers.

The formal setting

A neural operator approximates a mapping between infinite-dimensional function spaces, not between finite vectors. Let a be an input function (e.g. an initial condition, a coefficient field, a boundary forcing), and let G be the true solution operator of a differential equation, so that u = G(a) is the corresponding solution function. A neural operator G_θ is trained so G_θ(a) ≈ G(a) for a drawn from a distribution of plausible inputs.

a — an entire function, not a single number; in practice represented by its values at a fixed set of "sensor" locations
G — the true solution operator, generally nonlinear and defined on infinite-dimensional spaces
G_θ(a)(y) — the operator's prediction of the solution, evaluated at query point y

The developer example simplifies a to a single scalar (a starting value) rather than a full sampled function, which is why one input feeds branch_net directly rather than through a sensor-sampling step. In the DeepONet formulation of Lu, Jin, Pang, Zhang and Karniadakis (2021), the branch network reads a sampled at a fixed set of sensor points {x_1, ..., x_m}, and the trunk network reads the query location y:

G_θ(a)(y)  =  Σ_{k=1}^{p}  b_k(a(x_1), ..., a(x_m)) · t_k(y)  +  bias

b_k — the k-th output of the branch network
t_k — the k-th output of the trunk network
p — the shared code size, as in the developer example

This architecture is a universal approximator of nonlinear operators, a result following from the operator generalisation of the universal approximation theorem established by Chen and Chen (1995), which DeepONet's authors build on directly.

The Fourier Neural Operator alternative

A second major family, the Fourier Neural Operator (Li et al., 2021), takes a different route: it represents the operator as a series of learned convolutions performed in Fourier space, which lets it act on functions sampled at any resolution — a network trained on a coarse grid can be evaluated on a finer one without retraining, a property called discretisation invariance that plain neural networks (and DeepONet, to a lesser degree) do not automatically have.

Complexity and cost

For m sensor points, p code size, and N query points evaluated per input function:

Component	Cost
Branch network forward pass	`O(m · p)` per input function, once
Trunk network forward pass	`O(p)` per query point
Total for `N` queries on one input	`O(m · p + N · p)`

The key structural benefit: the branch network's expensive-ish pass over a happens once per input function, however many query points N are then evaluated cheaply against the trunk network — a very different cost profile from re-solving the full equation once per query.

Papers

Lu, L., Jin, P., Pang, G., Zhang, Z. and Karniadakis, G. (2021). Learning Nonlinear Operators via DeepONet Based on the Universal Approximation Theorem of Operators. Nature Machine Intelligence 3. The original DeepONet paper.
Li, Z. et al. (2021). Fourier Neural Operator for Parametric Partial Differential Equations. ICLR.
Chen, T. and Chen, H. (1995). Universal Approximation to Nonlinear Operators by Neural Networks with Arbitrary Activation Functions and Its Application to Dynamical Systems. IEEE Transactions on Neural Networks. The theoretical foundation both later papers build on.
Kovachki, N. et al. (2023). Neural Operator: Learning Maps Between Function Spaces. Journal of Machine Learning Research. A unifying survey of the field.

Current state

Neural operators are an active research area, with Fourier-based and transformer-based variants competing on speed, accuracy and discretisation invariance across weather, fluid dynamics and materials problems. A recurring finding across the literature is that neural operators, like the surrogate models and PINNs before them, remain confidently wrong outside the distribution of input functions they were trained on — generalisation across a family is not the same guarantee as generalisation to any input, and current evaluation practice in the field is still converging on how to report that gap honestly.

What to learn next

Physics-informed neural networks — combining a physics residual loss with an operator architecture is an active research direction (physics-informed DeepONets).
Attention — the mechanism underlying transformer-based neural operator variants.
Molecular property prediction — a different science domain facing the same generalisation-versus-memorisation question.

🟢 Beginner — No maths. Plain English.

Why it exists#

How it works#

Where this shows up#

An honest warning#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

Minimal runnable code#

Walkthrough#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

The formal setting#

The Fourier Neural Operator alternative#

Complexity and cost#

Papers#

Current state#

What to learn next#

Beginner — No maths. Plain English.

Why it exists

How it works

Where this shows up

An honest warning

Remember this

What to learn next

Developer — Code and libraries.

Setup

Minimal runnable code

Walkthrough

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

The formal setting

The Fourier Neural Operator alternative

Complexity and cost

Papers

Current state

What to learn next