Learn AI Create the Future — with Pranay
Main site

Serving Models in Production

gRPC vs REST for inference

gRPC and REST are two different ways for a caller to talk to a model server, trading gRPC's speed and strict structure against REST's simplicity and universal support.

Read these first

On this page 7
  1. The short answer
  2. The analogy you have already lived
  3. Why it exists
  4. How it works
  5. A real example you have seen
  6. Remember this
  7. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

gRPC and REST are two different languages a caller and a model server can use to talk to each other. gRPC is faster and stricter. REST is simpler and more universal.

The analogy you have already lived

Writing a letter in plain conversational language works everywhere, and anyone can read it. That is REST, sent as readable text (JSON) that any tool can inspect.

Filling out a government form with fixed boxes in a fixed order is different. It is faster to process at scale, and leaves no room for ambiguity. But you need the exact form template to fill it in, or read it. That is closer to gRPC, sent as compact, pre-agreed binary data.

Why it exists

Model serving used REST — plain JSON over HTTP. It is simple, debuggable with a browser or curl, and understood by every programming language without extra setup.

gRPC exists for situations where that convenience costs too much. Very high request volume. Tight latency budgets. Many internal services calling each other, where every millisecond and every byte matters more than human readability.

How it works

REST:  caller sends readable JSON text over plain HTTP
       {"income": 6.0, "years": 5.0}
       -- easy to read, a little slower to send and parse

gRPC:  caller sends compact binary data over HTTP/2,
       matching a strict pre-agreed structure (a .proto file)
       -- faster to send and parse, not human-readable as-is

Both carry the exact same information. gRPC encodes it more compactly, and uses a more efficient underlying connection.

A real example you have seen

Picture two services inside the same company, talking to each other thousands of times a second behind the scenes. The checkout service might ask the inventory service if an item is in stock. These internal calls often use gRPC. The public API a website's browser calls almost always uses REST instead, since a browser needs something universally readable.

Remember this

  • REST sends readable JSON text; gRPC sends compact binary data in a stricter, pre-agreed format.
  • gRPC is generally faster, at the cost of being harder to inspect casually and needing a shared schema file.
  • Choose based on who is calling: a browser or external partner favours REST; a tight internal service-to-service link favours gRPC.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install flask grpcio grpcio-tools requests

Defining the same request in gRPC's schema language

scorer.proto
syntax = "proto3";
package inference;

message PredictRequest {
  float income = 1;
  float years = 2;
}

message PredictResponse {
  bool approved = 1;
  float probability = 2;
}

service Scorer {
  rpc Predict (PredictRequest) returns (PredictResponse);
}
bash
python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. scorer.proto

That command generates scorer_pb2.py and scorer_pb2_grpc.py — Python code matching the schema above, which both the server and client import.

Timing the same model behind both protocols

grpc_vs_rest.py
import logging
import threading
import time
from concurrent import futures

import grpc
import requests
from flask import Flask, jsonify, request
from werkzeug.serving import make_server

import scorer_pb2
import scorer_pb2_grpc

# Quiets Flask's per-request access log -- otherwise 300 REST calls below
# print 300 log lines that would swamp the timing output.
logging.getLogger("werkzeug").setLevel(logging.ERROR)


def score(income, years):
    probability = min(1.0, max(0.0, (income + years) / 20))
    return probability >= 0.5, probability


# ---- REST server (Flask, JSON over HTTP/1.1) ----
rest_app = Flask(__name__)


@rest_app.post("/predict")
def rest_predict():
    body = request.get_json()
    approved, probability = score(body["income"], body["years"])
    return jsonify({"approved": approved, "probability": probability})


def run_rest():
    # make_server, not rest_app.run(): the same server, without its startup banner.
    make_server("127.0.0.1", 5055, rest_app).serve_forever()


# ---- gRPC server (protobuf over HTTP/2) ----
class ScorerServicer(scorer_pb2_grpc.ScorerServicer):
    def Predict(self, request, context):
        approved, probability = score(request.income, request.years)
        return scorer_pb2.PredictResponse(approved=approved, probability=probability)


def run_grpc():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=4))
    scorer_pb2_grpc.add_ScorerServicer_to_server(ScorerServicer(), server)
    server.add_insecure_port("127.0.0.1:50051")
    server.start()
    server.wait_for_termination()


threading.Thread(target=run_rest, daemon=True).start()
threading.Thread(target=run_grpc, daemon=True).start()
time.sleep(1.5)  # let both servers finish binding

N = 300

session = requests.Session()
t0 = time.perf_counter()
for _ in range(N):
    session.post("http://127.0.0.1:5055/predict", json={"income": 6.0, "years": 5.0}).json()
rest_time = time.perf_counter() - t0

channel = grpc.insecure_channel("127.0.0.1:50051")
stub = scorer_pb2_grpc.ScorerStub(channel)
t0 = time.perf_counter()
for _ in range(N):
    stub.Predict(scorer_pb2.PredictRequest(income=6.0, years=5.0))
grpc_time = time.perf_counter() - t0

print(f"n_calls={N}")
print(f"REST (Flask, JSON)  : {rest_time:.3f}s total, {rest_time/N*1000:.3f} ms/call")
print(f"gRPC (protobuf)     : {grpc_time:.3f}s total, {grpc_time/N*1000:.3f} ms/call")
Output
n_calls=300
REST (Flask, JSON)  : 1.892s total, 6.305 ms/call
gRPC (protobuf)     : 0.354s total, 1.179 ms/call

These are wall-clock numbers from one run, on one machine, both servers running as plain development instances with no production tuning on either side. Treat the roughly 5x gap as illustrative of the direction of the difference, not a number to quote elsewhere — your own measurement, on your own workload, is the one that matters.

Line-by-line walkthrough

The .proto file is the shared contract. Both server and client generate code from it, which is what makes gRPC strict: a field that is not in the schema cannot be sent or read, unlike REST's JSON, which happily accepts an extra or missing field without complaint.

grpc.server(futures.ThreadPoolExecutor(...)) shows gRPC's Python server running requests on a thread pool directly — a similar shape to the run_in_threadpool pattern in serving a model with FastAPI, for the same reason: handling several requests without one blocking the rest.

Common mistakes

Choosing gRPC for a public API a browser needs to call directly. Browsers cannot speak gRPC natively without an extra proxy layer (gRPC-Web). For anything a browser calls directly, REST remains the simpler default.

Changing a .proto field's number or type without care. Protobuf's compactness comes from field numbers, not names — reusing a number for a different field silently corrupts old clients' data. Field numbers, once shipped, are close to permanent.

Assuming gRPC is always faster in practice, regardless of payload. For tiny payloads like this lesson's, most of gRPC's advantage comes from HTTP/2 connection reuse and binary encoding overhead being smaller in absolute terms — the gap narrows for very large payloads where both protocols spend most of their time on the network itself, not on parsing.

Try it yourself

Add a history: list[float] field of 100 numbers to both the REST JSON body and the .proto message, and re-run the timing. Confirm whether the gap between REST and gRPC grows, shrinks, or stays about the same as the payload gets larger.

What to learn next

Researcher — Mathematics and papers.

Where the gap actually comes from

Three separable factors contribute to gRPC's typical latency advantage over REST/JSON, and they do not all matter equally at every scale:

  • Serialisation format — Protocol Buffers encode as compact binary with no field names repeated per message, versus JSON's self-describing text. For small structured messages this is a meaningful constant-factor win in both size and parse time.
  • Transport — gRPC runs over HTTP/2, which multiplexes many requests over one TCP connection and supports server push and native streaming; REST over HTTP/1.1 typically opens more connections or serialises requests on a shared one without multiplexing, though HTTP/2-based REST is possible and closes part of this gap.
  • Schema-driven code generation — removes manual parsing and validation code on both ends, which is a development-velocity and correctness benefit more than a raw-latency one.

Benchmarks that report large gRPC-over-REST speedups often conflate these three factors, or compare a naive REST implementation against a well-tuned gRPC one. Isolating which factor dominates for a specific workload requires measuring serialisation and network time separately, not only end-to-end latency.

Streaming as gRPC's structural advantage

REST's request-response model has no first-class notion of a long-lived bidirectional stream; workarounds (long polling, Server-Sent Events, a WebSocket bolted on beside REST) exist but are not native to the protocol. gRPC defines four call types natively: unary, server-streaming, client-streaming, and bidirectional streaming, all sharing the same schema and code-generation machinery. For a workload like token-by-token LLM output — the subject of streaming token responses — gRPC's server-streaming mode is a more structurally direct fit than REST's usual SSE workaround, though SSE remains simpler to consume from a browser.

Protocol choice as a systems decision, not a benchmark

In practice, protocol choice at the edge of a system (what a browser or external partner calls) is usually constrained toward REST regardless of raw performance, by ecosystem support and debuggability. Protocol choice between internal services is more often driven by raw performance and streaming needs, which is why gRPC dominates internal service mesh traffic at companies operating at very large request volumes, while public APIs at the same companies remain REST almost universally.

Papers and systems

  • The official gRPC documentation's performance best-practices guide is the most current source on tuning either side of this comparison: grpc.io/docs/guides/performance
  • Google's Protocol Buffers design rationale, in the protobuf project documentation, explains the field-number-based wire format referenced above.
  • Fielding, Architectural Styles and the Design of Network-based Software Architectures, 2000 — the dissertation that defines REST, useful context for why REST's constraints (statelessness, uniform interface) were chosen for a very different problem than gRPC's.

What to learn next