BabelCalib: A Universal Approach to Calibrating Central Cameras. In ICCV (2021)

Overview

BabelCalib: A Universal Approach to Calibrating Central Cameras

Paper Datasets Conference Poster Youtube

This repository contains the MATLAB implementation of the BabelCalib calibration framework.

Method overview and result. (left) BabelCalib pipeline: the camera model proposal step ensures a good initialization (right) example result showing residuals of reprojected corners of test images.


Projection of calibration target from estimated calibration. Detected corners are red crosses, target projected using initial calibration are blue squares and using the final calibration are cyan circles.

Description

BabelCalib is a calibration framework that can estimate camera models for all types of central projection cameras. Calibration is robust and fully automatic. BabelCalib provides models for pinhole cameras with additive distortion as well as omni-directional cameras and catadioptric rigs. The supported camera models are listed under the solvers directory. BabelCalib supports calibration targets made of a collection of calibration boards, i.e., multiple planar targets. The method is agnostic to the pattern type on the calibration boards. It is robust to inaccurately localized corners, outlying detections and occluded targets.

Table of Contents


Installation

You need to clone the repository. The required library Visual Geometry Toolkit is added as a submodule. Please clone the repository with submodules:

git clone --recurse-submodules https://github.com/ylochman/babelcalib

If you already cloned the project without submodules, you can run

git submodule update --init --recursive 

Calibration

Calibration is performed by the function calibrate.m. The user provides the 2D<->3D correspondence of the corner detections in the captured images as well as the coordinates of the calibration board fiducials and the absolute poses of the calibration boards. Any calibration board of the target may be partially or fully occluded in a calibration image. The camera model is returned as well as diagnostics about the calibration.

function [model, res, corners, boards] = calibrate(corners, boards, imgsize, varargin)

Parameters:

  • corners : type corners
  • boards : type boards
  • imgsize : 1x2 array specifying the height and width of the images; all images in a capture are assumed to have the same dimensions.
  • varargin : optional arguments

Returns

Evaluation

BabelCalib adopts the train-test set methodology for fitting and evaluation. The training set contains the images used for calibration, and the test set contains held-out images for evaluation. Evaluating a model on test-set images demonstrates how well a calibration generalizes to unseen imagery. During testing, the intriniscs are kept fixed and only the poses of the camera are regressed. The RMS re-projection error is used to assess calibration quality. The poses are estimated by get_poses.m:

function [model, res, corners, boards] = get_poses(intrinsics, corners, boards, imgsize, varargin)

Parameters:

  • intrinsics : type model
  • corners : type corners
  • boards : type boards
  • imgsize : 1x2 array specifies the height and width of the images; all the images are assumed to have the same dimensions
  • varargin : optional arguments

Returns

Type Defintions

corners : 1xN struct array

Contains the set of 2D<->3D correspondences of the calibration board fiducials to the detected corners in each image. Here, we let N be the number of images; Kn be the number of detected corners in the n-th image, where (n=1,...,N); and B be the number of planar calibration boards.

field data type description
x 2xKn array 2D coordinates specifying the detected corners
cspond 2xKn array correspondences, where each column is a correspondence and the first row contains the indices to points and the second row contains indices to calibration board fiducials

boards : 1xB struct array

Contains the set of absolute poses for each of the B calibration boards of the target, where (b=1,...,B) indexes the calibration boards. Also specifies the coordinates of the fiducials on each of the calibration boards.

field data type description
Rt 3x4 array absolute orientation of each pose is encoded in the 3x4 pose matrix
X 2xKb array 2D coordinates of the fiducials on board b of the target. The coordinates are specified with respect to the 2D coordinate system attached to each board

model : struct

Contains the intrinsics and extrinsics of the regressed camera model. The number of parameters of the back-projection or projection model, denoted C, depends on the chosen camera model and model complexity.

field data type description
proj_model str name of the target projection model
proj_params 1xC array parameters of the projection/back-projection function
K 3x3 array camera calibration matrix (relating to A in the paper: K = inv(A))
Rt 3x4xN array camera poses stacked along the array depth

res : struct

Contains the information about the residuals, loss and initialization (minimal solution). Here, we let K be the total number of corners in all the images.

field data type description
loss double loss value
ir double inlier ratio
reprojerrs 1xK array reprojection errors
rms double root mean square reprojection error
wrms double root mean square weighted reprojection error (Huber weights)
info type info

info : struct

Contains additional information about the residuals, loss and initialization (minimal solution).

field data type description
dx 2xK array re-projection difference vectors: dx = x - x_hat
w 1xK array Huber weights on the norms of dx
residual 2xK array residuals: residual = w .* dx
cs 1xK array (boolean) consensus set indicators (1 if inlier, 0 otherwise)
min_model type model model corresponding to the minimal solution
min_res type res residual info corresponding to the minimal solution

cfg

cfg contains the optional configurations. Default values for the optional parameters are loaded from parse_cfg.m. These values can be changed by using the varargin parameter. Parameters values passed in by varargin take precedence. The varargin format is 'param_1', value_1, 'param_2', value_2, .... The parameter descriptions are grouped by which component of BabelCalib they change.

Solver configurations:

  • final_model - the selected camera model (default: 'kb')
  • final_complexity - a degree of the polynomial if the final model is polynomial, otherwise ignored (default: 4)

Sampler configurations:

  • min_trial_count - minimum number of iterations (default: 20)
  • max_trial_count - maximum number of iterations (default: 50)
  • max_num_retries - maximum number of sampling tries in the case of a solver failure (default: 50)
  • confidence - confidence rate (default: 0.995)
  • sample_size - the number of 3D<->2D correspondences that are sampled for each RANSAC iteration (default: 14)

RANSAC configurations:

  • display - toggles the display of verbose output of intermediate steps (default: true)
  • display_freq - frequency of output during the iterations of robust sampling. (default: 1)
  • irT - minimum inlier ratio to perform refinement (default: 0)

Refinement configurations:

  • reprojT - reprojection error threshold (default: 1.5)
  • max_iter - maximum number of iterations on the refinement (default: 50)

Examples and wrappers

2D<->3D correspondences

BabelCalib provides a convenience wrapper calib_run_opt1.m for running the calibration calibrate.m with a training set and evaluating get_poses.m with a test set.

Deltille

The Deltille detector is a robust deltille and checkerboard detector. It comes with detector library, example detector code, and MATLAB bindings. BabelCalib provides functions for calibration and evaluation using the Deltille software's outputs. Calibration from Deltille detections requires format conversion which is peformed by import_ODT.m. A complete example of using calibrate and get_poses with import_ODT is provided in calib_run_opt2.m.

Citation

If you find this work useful in your research, please consider citing:

@InProceedings{Lochman-ICCV21,
    title     = {BabelCalib: A Universal Approach to Calibrating Central Cameras},
    author    = {Lochman, Yaroslava and Liepieshov, Kostiantyn and Chen, Jianhui and Perdoch, Michal and Zach, Christopher and Pritts, James},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    year      = {2021},
}

License

The software is licensed under the MIT license. Please see LICENSE for details.

Visualize Camera's Pose Using Extrinsic Parameter by Plotting Pyramid Model on 3D Space

extrinsic2pyramid Visualize Camera's Pose Using Extrinsic Parameter by Plotting Pyramid Model on 3D Space Intro A very simple and straightforward modu

JEONG HYEONJIN 106 Dec 28, 2022
Learning Visual Words for Weakly-Supervised Semantic Segmentation

[IJCAI 2021] Learning Visual Words for Weakly-Supervised Semantic Segmentation Implementation of IJCAI 2021 paper Learning Visual Words for Weakly-Sup

Lixiang Ru 24 Oct 05, 2022
PlenOctree Extraction algorithm

PlenOctrees_NeRF-SH This is an implementation of the Paper PlenOctrees for Real-time Rendering of Neural Radiance Fields. Not only the code provides t

49 Nov 05, 2022
[CVPR'21] Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild

IVOS-W Paper Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild Zhaoyun Yin, Jia Zheng, Weixin Luo, Shenhan Qian, Hanli

SVIP Lab 38 Dec 12, 2022
Vrcwatch - Supply the local time to VRChat as Avatar Parameters through OSC

English: README-EN.md VRCWatch VRCWatch は、VRChat 内のアバター向けに現在時刻を送信するためのプログラムです。 使

Kosaki Mezumona 17 Nov 30, 2022
Code for paper [ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot] (ICCV 2021, oral))

ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot This repository is the official PyTorch implementation of ICCV-21 pape

Jiarui 21 May 09, 2022
Train emoji embeddings based on emoji descriptions.

emoji2vec This is my attempt to train, visualize and evaluate emoji embeddings as presented by Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko

Miruna Pislar 17 Sep 03, 2022
Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On [Project website] [Dataset] [Video] Abstract We propose a new g

71 Dec 24, 2022
验证码识别 深度学习 tensorflow 神经网络

captcha_tf2 验证码识别 深度学习 tensorflow 神经网络 使用卷积神经网络,对字符,数字类型验证码进行识别,tensorflow使用2.0以上 目前项目还在更新中,诸多bug,欢迎提出issue和PR, 希望和你一起共同完善项目。 实例demo 训练过程 优化器选择: Adam

5 Apr 28, 2022
Monocular 3D pose estimation. OpenVINO. CPU inference or iGPU (OpenCL) inference.

human-pose-estimation-3d-python-cpp RealSenseD435 (RGB) 480x640 + CPU Corei9 45 FPS (Depth is not used) 1. Run 1-1. RealSenseD435 (RGB) 480x640 + CPU

Katsuya Hyodo 8 Oct 03, 2022
MolRep: A Deep Representation Learning Library for Molecular Property Prediction

MolRep: A Deep Representation Learning Library for Molecular Property Prediction Summary MolRep is a Python package for fairly measuring algorithmic p

AI-Health @NSCC-gz 83 Dec 24, 2022
RAMA: Rapid algorithm for multicut problem

RAMA: Rapid algorithm for multicut problem Solves multicut (correlation clustering) problems orders of magnitude faster than CPU based solvers without

Paul Swoboda 60 Dec 13, 2022
AutoVideo: An Automated Video Action Recognition System

AutoVideo is a system for automated video analysis. It is developed based on D3M infrastructure, which describes machine learning with generic pipeline languages. Currently, it focuses on video actio

Data Analytics Lab at Texas A&M University 267 Dec 17, 2022
Winners of DrivenData's Overhead Geopose Challenge

Winners of DrivenData's Overhead Geopose Challenge

DrivenData 22 Aug 04, 2022
[MedIA2021]MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning

MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning [MedIA or Arxiv] and [Demo] This repository pr

Healthcare Intelligence Laboratory 92 Dec 08, 2022
PyTorch code for the paper "Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval".

Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval (M2HSE) PyTorch code fo

Xinlei-Pei 6 Dec 23, 2022
Interactive dimensionality reduction for large datasets

BlosSOM 🌼 BlosSOM is a graphical environment for running semi-supervised dimensionality reduction with EmbedSOM. You can use it to explore multidimen

19 Dec 14, 2022
Code for "NeRS: Neural Reflectance Surfaces for Sparse-View 3D Reconstruction in the Wild," in NeurIPS 2021

Code for Neural Reflectance Surfaces (NeRS) [arXiv] [Project Page] [Colab Demo] [Bibtex] This repo contains the code for NeRS: Neural Reflectance Surf

Jason Y. Zhang 234 Dec 30, 2022
Author's PyTorch implementation of Randomized Ensembled Double Q-Learning (REDQ) algorithm.

REDQ source code Author's PyTorch implementation of Randomized Ensembled Double Q-Learning (REDQ) algorithm. Paper link: https://arxiv.org/abs/2101.05

109 Dec 16, 2022
PyTorch META-DATASET (Few-shot classification benchmark)

PyTorch META-DATASET (Few-shot classification benchmark) This repo contains a PyTorch implementation of meta-dataset and a unified implementation of s

Malik Boudiaf 39 Oct 31, 2022