Camera calibration: key point method vs direct method

The most common method in camera calibration is the key point method , such as OpenCV and MatLab Use Checkerboard in 、 Circle array and other two-dimensional patterns for calibration , The corners and center of the chessboard are the so-called key points . Although the key point method has many advantages , However, in some cases, it is easy to encounter insufficient calibration data , Therefore, there is a direct camera calibration method based on photometric error . This paper introduces the general principle, advantages and disadvantages of key point method and direct method .

Principle of key point method

The principle of calculating camera parameters by key point method is simply to minimize the re projection error .

The camera imaging process can be expressed as ：
$$\mathbf{u}=\pi (T(\mathbf{x}))$$
among $\mathbf{x}=[xyz{]}^T$ Is the three-dimensional point coordinate in the world coordinate system ,$\mathbf{u}=[uv{]}^T$ Is the coordinates of two-dimensional points on the image .

$T()$ Transform for 3D rigid body , Rotation translation , contain 6 A degree of freedom , In the optimization process, the rotation vector is generally used / Four yuan number + Translation vector to represent , Its function is to convert the three-dimensional coordinates from the world coordinate system to the camera coordinate system . In the key point method , The world coordinate system is established on the plane of the calibration plate , Therefore, the rotation and translation here can also be understood as the pose of the calibration plate in the camera coordinate system , External reference .

$\pi ()$ Is the camera projection function , Can be split into pinhole projections $P()$ And distortion $D()$ Two processes ：$\pi ({\mathbf{x}}^{\prime })=D(P({\mathbf{x}}^{\prime }))$, Parameters include focal length ($f_x,f_y$)、 Principal point ($c_x,c_y$) And distortion parameters ( Related to distortion model ), Collectively referred to as camera internal reference .

The internal and external parameters can obtain the initial estimation through some algorithms or a priori information , The three-dimensional coordinates of the key points on the calibration plate are known , The actual image coordinates of the key points projected on the image and extracted ($u^{\prime }$) Make a difference , You can build optimization problems ：
$$\mathrm{argmin}\{\frac{1}{2}\sum [\pi (T(\mathbf{x}))-u^{\prime }{]}^2\}$$
Accurate internal and external parameters can be obtained through nonlinear optimization .

advantage ： The key point method model is relatively simple , The calculation is also relatively simple 、 Fast . The special calibration board pattern can easily extract and decode key points ( Identify the unique of each key point ID). The extraction of key points is usually not easy to be illuminated 、 Noise and other factors .

shortcoming ： The key itself is a sparse feature , Wasted some useful image information . For example, in the chessboard calibration board image , It's not just corners that are useful , Each edge of the grid has strong gradient information, which can be used for calibration , But they are ignored in the key point method . Therefore, the utilization rate of image information of key point method is low , It often needs to be compensated by a large number of drawings .

Principle of direct method

The word direct method comes from SLAM ( Front end tracking based on direct method ), The principle is similar to optical flow , That is to minimize photometric error .

Record the camera imaging process as $\mathbf{u}=L(\mathbf{x})=\pi (T(\mathbf{x}))$. Obviously $\mathbf{x}$ There is $z=0$ The constraints of ,$L()$ It's reversible , Write it down as $\mathbf{x}=L^{-1}(\mathbf{u})$. in other words , When internal and external parameters are known , Any point on the image can find its coordinates on the calibration plate . The calibration plate pattern is known , It can be expressed as ：$v_x=I_b(\mathbf{x})$, The specific coordinates of a given pixel can be obtained . So we can get every point on the image “ should ” Pixel value ：
$$v_u=I_b({\mathbf{L}}^{-1}(\mathbf{u}))$$
Based on the above formula, the calibration image can be simulated , In the direct calibration method, it can be compared with the actual pixel value ($v_u^{\prime }$) Make a difference ( Photometric error ), Build optimization problems ：
$$\mathrm{argmin}\{\frac{1}{2}\sum [v_u-v_u^{\prime }{]}^2\}$$
Theoretically, every point on the image can provide photometric error information ( The area not covered by the calibration plate is not considered ), But in fact, the points with small gray gradient have little contribution to the solution of the optimization problem , On the contrary, it increases the amount of calculation . Therefore, we can do gradient analysis on the calibration image first (Sobel), Only the points with large gradient and the points near them are added to the solution of the optimization problem .

advantage ： Direct method can make full use of image pixel information , Improve image utilization efficiency , So as to reduce the requirements of calibration image acquisition . in addition , Through reasonable modeling , The direct method can also solve the relevant parameters of camera exposure ( time of exposure 、 Line delay of roller shutter exposure, etc ).

shortcoming ： Due to the common distortion model (Brown、KB) There is no analytical solution to the inverse operation of , De distortion process ($L^{-1}()$ Part of the function ) It requires iterative computation , And the number of residuals in each graph is much larger than that of the feature point method , As a result, the calibration speed of direct method is very slow . On the other hand , If the exposure parameters are not considered in the simplified direct method model , Photometric error will be affected by brightness value deviation and uneven illumination , However, this problem can be solved by image brightness matching and brightness homogenization algorithm .

Reference material ：

1. Rehder, Joern, Janosch Nikolic, Thomas Schneider, and Roland Siegwart. “A Direct Formulation for Camera Calibration,” n.d., 8.
2. Hannemose, Morten, Jakob Wilm, and Jeppe R. Frisvad. “Superaccurate Camera Calibration via Inverse Rendering.” In Modeling Aspects in Optical Metrology VII, edited by Bernd Bodermann, Karsten Frenner, and Richard M. Silver, 40. Munich, Germany: SPIE, 2019. https://doi.org/10.1117/12.2531769.
3. Usenko, Vladyslav, Nikolaus Demmel, and Daniel Cremers. “The Double Sphere Camera Model.” In 2018 International Conference on 3D Vision (3DV), 552–60. Verona: IEEE, 2018. https://doi.org/10.1109/3DV.2018.00069.