Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Overview

Sensor-Guided Optical Flow

Demo code for "Sensor-Guided Optical Flow", ICCV 2021

This code is provided to replicate results with flow hints obtained from LiDAR data.

At the moment, we do not plan to release training code.

[Project page] - [Paper] - [Supplementary]

Alt text

Reference

If you find this code useful, please cite our work:

@inproceedings{Poggi_ICCV_2021,
  title     = {Sensor-Guided Optical Flow},
  author    = {Poggi, Matteo and
               Aleotti, Filippo and
               Mattoccia, Stefano},
  booktitle = {IEEE/CVF International Conference on Computer Vision (ICCV)},
  year = {2021}
}

Contents

  1. Introduction
  2. Installation
  3. Data
  4. Weights
  5. Usage
  6. Contacts
  7. Acknowledgments

Introduction

This paper proposes a framework to guide an optical flow network with external cues to achieve superior accuracy either on known or unseen domains. Given the availability of sparse yet accurate optical flow hints from an external source, these are injected to modulate the correlation scores computed by a state-of-the-art optical flow network and guide it towards more accurate predictions. Although no real sensor can provide sparse flow hints, we show how these can be obtained by combining depth measurements from active sensors with geometry and hand-crafted optical flow algorithms, leading to accurate enough hints for our purpose. Experimental results with a state-of-the-art flow network on standard benchmarks support the effectiveness of our framework, both in simulated and real conditions.

Installation

Install the project requirements in a new python 3 environment:

virtualenv -p python3 guided_flow_env
source guided_flow_env/bin/activate
pip install -r requirements.txt

Compile the guided_flow module, written in C (required for guided flow modulation):

cd external/guided_flow
bash compile.sh
cd ../..

Data

Download KITTI 2015 optical flow training set and precomputed flow hints. Place them under the data folder as follows:

data
├──training
    ├──image_2
        ├── 000000_10.png
        ├── 000000_11.png
        ├── 000001_10.png
        ├── 000001_11.png
        ...
    ├──flow_occ
        ├── 000000_10.png
        ├── 000000_11.png
        ├── 000001_10.png
        ├── 000001_11.png
        ...
    ├──hints
        ├── 000002_10.png
        ├── 000002_11.png
        ├── 000003_10.png
        ├── 000003_11.png
        ...

Weights

We provide QRAFT models tested in Tab. 4. Download the weights and unzip them under weights as follows:

weights
├──raw
    ├── C.pth
    ├── CT.pth
    ...
├──guided
    ├── C.pth
    ├── CT.pth
    ...    

Usage

You are now ready to run the demo_kitti142.py script:

python demo_kitti142.py --model CTK --guided --out_dir results_CTK_guided/

Use --model to specify the weights you want to load among C, CT, CTS and CTK. By default, raw models are loaded, specify --guided to load guided weights and enable sensor-guided optical flow.

Note: Occasionally, the demo may run out of memory on ~12GB GPUs. The script saves intermediate results are saved in --out_dir. You can run again the script and it will skip all images for which intermediate results have been already saved in --out_dir, loading them from the folder. Remember to select a brand new --out_dir when you start an experiment from scratch.

In the end, the aforementioned command should print:

Validation KITTI: 2.08, 5.97

Numbers in Tab. 4 are obtained by running this code on a Titan Xp GPU, with PyTorch 1.7.0. We observed slight fluctuations in the numbers when running on different hardware (e.g., 3090 GPUs), mostly on raw models.

Contacts

m [dot] poggi [at] unibo [dot] it

Acknowledgments

Thanks to Zachary Teed for sharing RAFT code, used as codebase in our project.

The final project of "Applying AI to 2D Medical Imaging Data" of "AI for Healthcare" nanodegree - Udacity.

Pneumonia Detection from X-Rays Project Overview In this project, you will apply the skills that you have acquired in this 2D medical imaging course t

Omar Laham 1 Jan 14, 2022
Unsupervised Foreground Extraction via Deep Region Competition

Unsupervised Foreground Extraction via Deep Region Competition [Paper] [Code] The official code repository for NeurIPS 2021 paper "Unsupervised Foregr

28 Nov 06, 2022
Generative Adversarial Networks(GANs)

Generative Adversarial Networks(GANs) Vanilla GAN ClusterGAN Vanilla GAN Model Structure Final Generator Structure A MLP with 2 hidden layers of hidde

Zhenbang Feng 2 Nov 05, 2021
An investigation project for SISR.

SISR-Survey An investigation project for SISR. This repository is an official project of the paper "From Beginner to Master: A Survey for Deep Learnin

Juncheng Li 79 Oct 20, 2022
A fuzzing framework for SMT solvers

yinyang A fuzzing framework for SMT solvers. Given a set of seed SMT formulas, yinyang generates mutant formulas to stress-test SMT solvers. yinyang c

Project Yin-Yang for SMT Solver Testing 145 Jan 04, 2023
Numba-accelerated Pythonic implementation of MPDATA with examples in Python, Julia and Matlab

PyMPDATA PyMPDATA is a high-performance Numba-accelerated Pythonic implementation of the MPDATA algorithm of Smolarkiewicz et al. used in geophysical

Atmospheric Cloud Simulation Group @ Jagiellonian University 15 Nov 23, 2022
StrongSORT: Make DeepSORT Great Again

StrongSORT StrongSORT: Make DeepSORT Great Again StrongSORT: Make DeepSORT Great Again Yunhao Du, Yang Song, Bo Yang, Yanyun Zhao arxiv 2202.13514 Abs

369 Jan 04, 2023
Official Code for VideoLT: Large-scale Long-tailed Video Recognition (ICCV 2021)

Pytorch Code for VideoLT [Website][Paper] Updates [10/29/2021] Features uploaded to Google Drive, for access please send us an e-mail: zhangxing18 at

Skye 26 Sep 18, 2022
"Segmenter: Transformer for Semantic Segmentation" reproduced via mmsegmentation

Segmenter-based-on-OpenMMLab "Segmenter: Transformer for Semantic Segmentation, arxiv 2105.05633." reproduced via mmsegmentation. We reproduce Segment

EricKani 22 Feb 24, 2022
Code for approximate graph reduction techniques for cardinality-based DSFM, from paper

SparseCard Code for approximate graph reduction techniques for cardinality-based DSFM, from paper "Approximate Decomposable Submodular Function Minimi

Nate Veldt 1 Nov 25, 2022
Gradient-free global optimization algorithm for multidimensional functions based on the low rank tensor train format

ttopt Description Gradient-free global optimization algorithm for multidimensional functions based on the low rank tensor train (TT) format and maximu

5 May 23, 2022
Awesome-google-colab - Google Colaboratory Notebooks and Repositories

Unofficial Google Colaboratory Notebook and Repository Gallery Please contact me to take over and revamp this repo (it gets around 30k views and 200k

Derek Snow 1.2k Jan 03, 2023
TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification

TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification [NeurIPS 2021] Abstract Multiple instance learn

132 Dec 30, 2022
SelfAugment extends MoCo to include automatic unsupervised augmentation selection.

SelfAugment extends MoCo to include automatic unsupervised augmentation selection. In addition, we've included the ability to pretrain on several new datasets and included a wandb integration.

Colorado Reed 24 Oct 26, 2022
An example of time series augmentation methods with Keras

Time Series Augmentation This is a collection of time series data augmentation methods and an example use using Keras. News 2020/04/16: Repository Cre

九州大学 ヒューマンインタフェース研究室 229 Jan 02, 2023
Pytorch implementation of PCT: Point Cloud Transformer

PCT: Point Cloud Transformer This is a Pytorch implementation of PCT: Point Cloud Transformer.

Yi_Zhang 265 Dec 22, 2022
Colossal-AI: A Unified Deep Learning System for Large-Scale Parallel Training

ColossalAI An integrated large-scale model training system with efficient parallelization techniques Installation PyPI pip install colossalai Install

HPC-AI Tech 7.1k Jan 03, 2023
This codebase proposes modular light python and pytorch implementations of several LiDAR Odometry methods

pyLiDAR-SLAM This codebase proposes modular light python and pytorch implementations of several LiDAR Odometry methods, which can easily be evaluated

Kitware, Inc. 208 Dec 16, 2022
Matthew Colbrook 1 Apr 08, 2022
Permute Me Softly: Learning Soft Permutations for Graph Representations

Permute Me Softly: Learning Soft Permutations for Graph Representations

Giannis Nikolentzos 7 Jul 10, 2022