Calibrating a camera with a chessboard

Zhang's method#

Zhang (2000), A flexible new technique for camera calibration, TPAMI 22(11), is what cv2.calibrateCamera implements. The insight is that a planar target makes the problem tractable in closed form before any nonlinear refinement.

Set the board's plane at $Z = 0$. Then

$$ s\,\mathbf{x} = K \begin{bmatrix} \mathbf{r}_1 & \mathbf{r}_2 & \mathbf{r}_3 & \mathbf{t} \end{bmatrix} \begin{bmatrix} X \ Y \ 0 \ 1 \end{bmatrix} = \underbrace{K \begin{bmatrix} \mathbf{r}_1 & \mathbf{r}2 & \mathbf{t} \end{bmatrix}}{\textstyle H} \begin{bmatrix} X \ Y \ 1 \end{bmatrix} $$

So each view yields a homography $H = [\mathbf{h}_1\ \mathbf{h}_2\ \mathbf{h}_3] = \lambda K [\mathbf{r}_1\ \mathbf{r}_2\ \mathbf{t}]$, estimable by DLT from four or more correspondences.

Because $\mathbf{r}_1$ and $\mathbf{r}_2$ are orthonormal columns of a rotation matrix, writing $B = K^{-\top}K^{-1}$ gives two linear constraints per view:

$$ \mathbf{h}_1^\top B \, \mathbf{h}_2 = 0, \qquad \mathbf{h}_1^\top B \, \mathbf{h}_1 = \mathbf{h}_2^\top B \, \mathbf{h}_2 $$

$B$ is symmetric with 6 unknowns, defined up to scale, so 5 degrees of freedom. Three views give six constraints, enough to solve for $B$ linearly, after which $K$ follows by Cholesky decomposition of $B^{-1}$.

This is why one view fails and it is not a numerical accident. One view supplies two constraints on a 5-parameter object. With zero skew and known aspect ratio you can scrape by with two views; three is the true minimum, and the recommendation of 15 to 25 is about noise, not rank.

$B$ is the image of the absolute conic, $\omega$, introduced in camera models and intrinsics. The two constraints above say that $\mathbf{h}_1$ and $\mathbf{h}_2$ are the vanishing points of two orthogonal board directions.

The refinement stage#

The linear solution is not the answer, because it minimises an algebraic quantity with no geometric meaning. Zhang follows it with maximum-likelihood refinement over all parameters jointly:

$$ \min_{K, \mathbf{k}, {R_i, \mathbf{t}i}} \sum{i=1}^{n}\sum_{j=1}^{m} \bigl\lVert \mathbf{x}_{ij} - \hat{\mathbf{x}}(K, \mathbf{k}, R_i, \mathbf{t}_i, \mathbf{X}_j) \bigr\rVert^2 $$

Where $n$ is the number of views, $m$ the corners per view, and $\mathbf{k}$ the distortion coefficients. This is a sparse nonlinear least-squares problem solved by Levenberg-Marquardt, and it is a small instance of bundle adjustment. Under isotropic Gaussian corner noise the objective is the negative log-likelihood, which is why reprojection error is the correct cost rather than a convenient one.

The reported RMS is

$$ \mathrm{RMS} = \sqrt{\frac{1}{2nm}\sum_{i,j} \lVert \mathbf{x}{ij} - \hat{\mathbf{x}}{ij} \rVert^2} $$

It is a training error over the same points that determined the parameters. It has no cross-validation and is therefore not an estimate of accuracy on new data. This is the precise sense in which the one-view result above is overfitting: 108 residuals fitted with 15 free parameters plus a near-degenerate direction in parameter space.

A better diagnostic is a held-out view: calibrate on $n-1$ views, then compute the reprojection error of the remaining one with its pose solved by solvePnP under the fixed intrinsics. That number behaves like a test error and tracks true accuracy far better.

Degeneracies and conditioning#

Three configurations that produce a good RMS and wrong parameters:

• All views parallel. The plane normal is constant, so all homographies impose the same constraint on $B$. Rank-deficient.
• All views at the same distance and orientation, differing only by translation in the image plane. Adds nothing beyond the first view.
• Board too small in frame. The radii covered are a small subset of the image, so distortion is extrapolated everywhere else. This is the reason calibration guidance insists on covering the image corners.

The parameter covariance is available: cv2.calibrateCameraExtended returns per-parameter standard deviations. Reporting them alongside the values is good practice and instantly reveals a degenerate session.

Alternative targets#

ChArUco boards combine a chessboard with ArUco markers in the white squares. Each marker is individually identifiable, so partial occlusion and boards running off the edge of the frame are still usable — a large practical gain over plain chessboards. cv2.aruco.CharucoDetector handles them, and it is the recommended target for new work.

Circle grids (cv2.findCirclesGrid, symmetric or asymmetric) locate centroids rather than corners. Centroid estimation is more robust to blur; the drawback is that a circle's projected centroid is not the projection of the circle's centre under perspective, introducing a small systematic bias that matters at high precision.

Self-calibration from natural scenes avoids a target entirely, using the Kruppa equations or a stratified projective-to-metric upgrade. It is far less accurate and is the right choice only when you cannot get a target in front of the camera — archive footage, for instance.

What to learn next#

• Homographies and perspective warp — the plane-to-plane map this method is built on.
• Stereo depth from two cameras — the first thing you do with a calibrated camera.
• RANSAC for robust fitting — surviving the bad correspondences a real board produces.