Learn AI Create the Future — with Pranay
Main site

Reading and Reimplementing Papers

Finding the details the paper left out

Every paper omits something you need — here is the search order for recovering it, and the sweep that tells you whether the missing detail even matters.

Read these first

On this page 6
  1. Why the omissions happen
  2. Where to look, in order
  3. How it works
  4. A real example you have seen
  5. Remember this
  6. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Every paper leaves out something you need, and there is a reliable order for finding it — usually before you resort to guessing.

Someone gives you directions: "go past the temple, turn left at the big tree, and it is the third house." Every local finds it immediately. You stand at a junction with four trees, none of them noticeably bigger than the rest.

The directions are not wrong. They were written by a person who has walked that road a thousand times, for whom the big tree is not ambiguous.

Papers are written the same way. The authors ran their experiment two hundred times. The thing they forgot to write down is the thing they stopped noticing.

Why the omissions happen

Almost never on purpose.

Page limits are real — eight or nine pages, with a hard cut. Reviewers ask for more experiments, not more setup detail. And a detail that has been part of the group's default pipeline for three years does not feel like a decision worth documenting.

The result is that the parts most likely to be missing are the parts that felt most obvious to the authors. That is exactly backwards from what a reader needs.

Where to look, in order

Search in this sequence. Each step is cheaper than the one after it.

  1. The appendix. Most setup detail lives there, not in the body.
  2. The released code. If it exists, it settles the question completely.
  3. Later versions of the paper. Preprint servers keep every version. Authors add detail after review.
  4. The public reviews. For many conferences, reviewer questions and author replies are online. Reviewers ask precisely the questions you are asking.
  5. The authors' later papers. People reuse their own pipelines and describe them more fully the second time.
  6. Independent reproductions. Somebody has often already fought this exact battle.
  7. Email the authors. A short, specific question gets a reply more often than people expect.
  8. Measure it yourself. Try every plausible reading and see how much the answer moves.

Step 8 is not defeat. It is often the most useful step, and it produces information the paper never contained.

How it works

   missing detail
        │
        ├─→ found in appendix / code / reviews  →  use it, note the source
        │
        └─→ nowhere to be found
                 │
                 └─→ try every plausible value
                          │
                          ├─→ results barely move   →  stop worrying, say so
                          └─→ results move a lot    →  this detail IS the paper

That last branch matters. If a missing detail changes the result by three points, then the paper's contribution partly lives in a sentence nobody wrote.

A real example you have seen

Assembly instructions for furniture do this constantly. Step 7 shows a screw going in, without saying which way the panel faces. Half the internet's reviews of that product are people who found out the hard way. The manufacturer's own assembly video — the released code, in our analogy — answers it in four seconds.

Remember this

  • Omissions are accidental, and cluster in what felt obvious to the authors.
  • Search in order: appendix, code, later versions, reviews, then ask.
  • If you cannot find it, sweep the plausible values — and report how much it mattered.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install numpy

Verified with numpy 1.26.4 on CPU. Runs instantly.

Measuring how much a detail matters

When the search fails, the question stops being "what did they do?" and becomes "does it matter?" That second question is answerable in minutes.

Adam's bias correction is a good test case, because the paper devotes an experiment to it (Section 6.4) and plenty of reimplementations drop it as an optimisation. Here is what dropping it actually costs, under two settings of the second decay rate.

does_it_matter.py
import numpy as np

def steps_taken(correct, beta2=0.999, alpha=0.01, n=6):
    """Size of the step Adam actually takes, for a constant gradient of 1."""
    m = v = 0.0
    out = []
    for t in range(1, n + 1):
        g = 1.0                                   # a constant gradient, for clarity
        m = 0.9 * m + 0.1 * g
        v = beta2 * v + (1 - beta2) * g * g
        if correct:
            mh, vh = m / (1 - 0.9**t), v / (1 - beta2**t)
        else:
            mh, vh = m, v
        out.append(alpha * mh / (np.sqrt(vh) + 1e-8))
    return np.array(out)

print("step size actually taken, first six iterations (alpha = 0.01)")
print("  with bias correction   ", np.round(steps_taken(True), 5))
print("  without                ", np.round(steps_taken(False), 5))
print()
print("beta2 = 0.9 instead of 0.999:")
print("  with bias correction   ", np.round(steps_taken(True, beta2=0.9), 5))
print("  without                ", np.round(steps_taken(False, beta2=0.9), 5))
Output
step size actually taken, first six iterations (alpha = 0.01)
  with bias correction    [0.01 0.01 0.01 0.01 0.01 0.01]
  without                 [0.03162 0.0425  0.0495  0.05442 0.05797 0.06057]

beta2 = 0.9 instead of 0.999:
  with bias correction    [0.01 0.01 0.01 0.01 0.01 0.01]
  without                 [0.00316 0.00436 0.00521 0.00586 0.0064  0.00685]

With correction, every step has the size you asked for: 0.01. Without it, at the default decay of 0.999, the first step is roughly three times too large and grows from there. Change the decay to 0.9 and the sign of the error flips — the first step is now roughly three times too small.

That is the finding. The cost of the missing detail depends on another setting entirely. Anyone who tested "does bias correction matter?" at one decay rate and generalised would be wrong half the time.

The walkthrough

A constant gradient isolates the effect. With g = 1 at every step, any variation in step size comes from the correction terms alone, not from the problem. Strip away everything that is not the detail under test.

Correction makes the step size equal alpha exactly, for a constant gradient. That is what the correction is for: the two running averages start at zero and are biased toward zero early on, and dividing by 1 - beta^t undoes exactly that bias.

Six steps is enough. The bias decays geometrically, so the effect is concentrated in the opening iterations — the same iterations where a bad step size can push a network somewhere it never recovers from.

The effect is on step size, not on the loss. A sweep like this tells you the mechanism. Whether a three-times-too-large opening step costs accuracy is a separate experiment on a real model. Do not skip that second experiment because the first one looked convincing.

Reading the sources you already have

Three practical moves that recover most missing details.

Diff the arXiv versions. Papers gain appendices between submission and camera-ready. Downloading v1 and v3 and comparing section headings takes two minutes and often surfaces a hyperparameter table that did not exist at first.

Read the reviews. For venues using OpenReview, every reviewer question and author response is public. Reviewers routinely ask "what learning rate schedule was used?" and get an answer that never reached the paper.

Grep the released code for the thing you are missing. Configuration values, not prose. grep -rn "weight_decay" . answers in one second what an hour of reading the text will not. That skill is the next lesson.

Common mistakes

Guessing silently. Every guess must be recorded as a guess, in the code and in the write-up. An undocumented guess becomes a fact within a week.

Assuming the code matches the paper. Released code is often a later, evolved version. Where they disagree, the code is what produces reproducible numbers, and the disagreement itself is worth reporting.

Sweeping one detail at a time forever. If four details are unknown, a full sweep is expensive. Rank them by expected impact first, using cheap mechanism tests like the one above, and only sweep the top two properly.

Emailing a vague question. "How did you train this?" gets no reply. "In Table 3, was the learning rate constant or cosine-decayed over the 90 epochs?" gets one, often within a day.

Try it yourself

Add a third row to the script: bias correction applied to the first moment only, which is a common partial implementation. Predict whether its step sizes sit above or below the correct line, and by how much at step 1. Then change g = 1.0 to a gradient that decays as 1 / t and see whether the conclusion about bias correction survives.

What to learn next

Researcher — Mathematics and papers.

What gets omitted, ranked

Reproduction reports converge on a stable list of what is missing most often. Roughly in order:

  1. Learning-rate schedule details — warmup length, decay shape, and whether the schedule is defined over steps or epochs.
  2. Weight-decay scope — which parameter groups are excluded.
  3. Initialisation — scheme and gain, especially for custom modules.
  4. Preprocessing constants — normalisation statistics, resize interpolation, tokeniser version.
  5. Stopping and model selection — which checkpoint produced the reported number.
  6. Evaluation protocol — augmentation at test time, averaging convention for multi-class metrics.
  7. Number of runs and aggregation — whether the reported figure is a mean, a median or a maximum.

Items 5 to 7 are properties of the measurement rather than the method, which is why they belong at the top of the checklist in why your reimplementation is three points worse and near the bottom of most authors' attention.

Designing the sweep

When a detail cannot be recovered, treat it as a nuisance parameter and report a range rather than a point.

  • Enumerate the plausible set, not the continuous range. "Cosine or step decay", "excluded or included", "0.1 or 0.01" — discrete alternatives that a competent author might have chosen.
  • Screen cheaply first. A mechanism-level test, like the step-size computation above, costs seconds and eliminates alternatives whose effect is numerically negligible before any training run.
  • Run the survivors at reduced scale, with shared seeds across arms so the comparison is paired.
  • Report the envelope. "Accuracy ranged from 92.1 to 94.8 across the four plausible readings of the schedule; the paper reports 94.6" is a stronger and more honest statement than any single reproduction number.

The envelope also quantifies something the paper cannot: the sensitivity of the result to a choice the authors treated as free. A method whose reported gain is smaller than its sensitivity to an unstated detail has not been shown to work.

The structural fix, and its limits

Venue-level interventions have measurably improved this. Pineau et al. (2021), Improving Reproducibility in Machine Learning Research (JMLR), report on the NeurIPS code-submission policy and reproducibility checklist, which require authors to state hyperparameters, compute, and how results were aggregated. Code release rates rose substantially after these were introduced.

The limits are worth naming. A checklist captures what authors know they chose. It does not capture defaults inherited from a lab codebase, framework version behaviour, or the twenty configurations tried before the reported one. Those remain recoverable only from the code and from correspondence.

Contacting authors, effectively

A specific, low-cost question about a specific table cell has a high reply rate. Include the paper title, the exact table or equation number, the reading you have assumed, and the result you obtained. Authors respond to a message that shows the work has already been done; they rarely respond to one that asks them to do it.

Record the answer, cite it as personal correspondence, and — if it is generally useful — publish it. A reproduction report that documents which details were unrecoverable, which were obtained by correspondence, and how much each mattered is a more durable contribution than the matched number itself.

What to learn next