Introduction to nonparametric camera distortion model

Recently, I was reading papers related to camera distortion model , One of them is called nonparametric model, which is interesting ：

Schöps, Thomas, Viktor Larsson, Marc Pollefeys, and Torsten Sattler. “Why Having 10,000 Parameters in Your Camera Model Is Better Than Twelve.” ArXiv:1912.02908 [Cs], June 23, 2020. http://arxiv.org/abs/1912.02908.

The paper also open source related code ：https://github.com/puzzlepaint/camera_calibration.

Parametric distortion model

At present, the most common distortion models are ：

• Brown: Radial + Tangential distortion , Parameters have k1, k2, k3, p1, p2
• Rational: Rational model , Can be seen as Brown Extension of the model , Parameters have k1, k2, k3, k4, k5, k6, p1, p2
• KB: The so-called fisheye model , Parameters have k1, k2, k3, k4

OpenCV in calibrateCamera adopt 14 Two parameters support Brown、Rational Wait for the model ,fisheye::calibrate Is used to calibrate KB Model .

There are other FoV、UCM、Double Sphere Wait for the model , Visible in some SLAM System , See paper ：

Usenko, Vladyslav, Nikolaus Demmel, and Daniel Cremers. “The Double Sphere Camera Model.” In 2018 International Conference on 3D Vision (3DV), 552–60. Verona: IEEE, 2018. https://doi.org/10.1109/3DV.2018.00069.

These models can be classified as “ Parametric model ”, Because they all use models with a small number of parameters to fit the distortion of all pixels on the whole image . such as KB The projection function of the model is ：
$$\begin{array}{rl}\pi (\mathbf{x})=& \left[\begin{array}{c}{f}_{x}d(\theta )\frac{x}{r}\\ f_{y}d(\theta )\frac{y}{r}\end{array}\right]+\left[\begin{array}{c}{c}_x\\ c_y\end{array}\right],\\ r=& \sqrt{x^2+y^2},\\ \theta =& atan2(r,z),\\ d(\theta )=& \theta +k_1{\theta }^3+k_2{\theta }^5+k_3{\theta }^7+k_4{\theta }^9\end{array}$$
Only four parameters (k1~k4) The distortion of any point on the image can be calculated .

Nonparametric distortion model

The so-called nonparametric model does not mean that there are no parameters , But relative to the parametric model , They use a large number of parameters to represent the distortion of the whole picture . The most extreme case is that each pixel has a set of parameters to represent its distortion , That is, directly record the mapping of each point from distorted image to undistorted image （ Or vice versa ）. therefore ,OpenCV Medium initUndistortRectifyMap The mapping table generated by function can be understood as the parameter table of nonparametric distortion model .

obviously , Nonparametric model is an extension of parametric model , It has the greatest flexibility , Theoretically, it can be used to fit any camera distortion . But at the same time, the more General The easier the model is to over fit , In order to achieve better distortion removal effect , Nonparametric models require much more calibration data than parametric models .

In order to reduce the difficulty of calibration , The paper mentioned at the beginning of this paper proposes a sparse nonparametric model ：

As shown in the figure above , The whole image is divided into small distortion bands according to their size Grid, Every Grid There's only one control point , Corresponding to a set of distortion parameters , That is, the mapping from the coordinates of the control point to the coordinates of the undistorted image . When removing distortion at any point , First find its surroundings 4×4 Control points , And then use be based on B Bicubic interpolation of splines Obtain the distortion parameters of the current point , Thus mapping to distortion free coordinates .

Because the model parameters save the mapping from distorted coordinates to undistorted coordinates , This method can easily realize point distortion removal （ Corresponding unproject The process ）, But the plus distortion of points （ Corresponding project The process ） But it takes iteration to achieve .

Key code

The author's open source code is a very huge project , Even the nonlinear optimization solver has been implemented once . If you just want to understand and use the nonparametric distortion model , Just focus on these documents ：

• applications/camera_calibration/src/camera_calibration/b_spline.h
  Clear interpolation calculation process
• applications/camera_calibration/src/camera_calibration/models/central_generic.cc
  Various functions related to calibration , contain project、unporject、fitToModel etc.
• applications/camera_calibration/generic_models/src/central_generic.h
  Application related functions , contain project、unproject、save、load etc.