Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Last update: Sep 08, 2022

Overview

Visual Interestingness

Refer to the project description for more details.
This code based on the following paper.

Chen Wang, Yuheng Qiu, Wenshan Wang, Yafei Hu, Seungchan Kim and Sebastian Scherer, Unsupervised Online Learning for Robotic Interestingness with Visual Memory, IEEE Transactions on Robotics (T-RO), 2021.
It is an extended version of the conference paper:

Chen Wang, Wenshan Wang, Yuheng Qiu, Yafei Hu, and Sebastian Scherer, Visual Memorability for Robotic Interestingness via Unsupervised Online Learning, European Conference on Computer Vision (ECCV), 2020.
If you want the original version, go to the ECCV Branch instead.
We also provide ROS wrapper for this project, you may go to interestingness_ros.

Install Dependencies

This version is tested in PyTorch 1.7

  pip3 install -r requirements.txt

Long-term Learning

You may skip this step, if you download the pre-trained vgg16.pt into folder "saves".

Download coco dataset into folder [data-root]:

bash download_coco.sh [data-root] # replace [data-root] by your desired location

The dataset will be look like:

data-root
├──coco
   ├── annotations
   │   ├── annotations_trainval2017
   │   └── image_info_test2017
   └── images
       ├── test2017
       ├── train2017
       └── val2017

Run

python3 longterm.py --data-root [data-root] --model-save saves/vgg16.pt

# This requires a long time for training on single GPU.
# Create a folder "saves" manually and a model named "ae.pt" will be saved.

Short-term Learning

Dowload the SubT front camera data (SubTF) and put into folder "data-root", so that it looks like:

data-root
├──SubTF
   ├── 0817-ugv0-tunnel0
   ├── 0817-ugv1-tunnel0
   ├── 0818-ugv0-tunnel1
   ├── 0818-ugv1-tunnel1
   ├── 0820-ugv0-tunnel1
   ├── 0821-ugv0-tunnel0
   ├── 0821-ugv1-tunnel0
   ├── ground-truth
   └── train

Run

python3 shortterm.py --data-root [data-root] --model-save saves/vgg16.pt --dataset SubTF --memory-size 100 --save-flag n100usage

# This will read the previous model "ae.pt".
# A new model "ae.pt.SubTF.n1000.mse" will be generated.

You may skip this step, if you download the pre-trained vgg16.pt.SubTF.n100usage.mse into folder "saves".

On-line Learning

Run

  python3 online.py --data-root [data-root] --model-save saves/vgg16.pt.SubTF.n100usage.mse --dataset SubTF --test-data 0 --save-flag n100usage

  # --test-data The sequence ID in the dataset SubTF, [0-6] is avaiable
  # This will read the trained model "vgg16.pt.SubTF.n100usage.mse" from short-term learning.

Alternatively, you may test all sequences by running
```
  bash test.sh
```
This will generate results files in folder "results".
You may skip this step, if you download our generated results.

Evaluation

We follow the SubT tutorial for evaluation, simply run

python performance.py --data-root [data-root] --save-flag n100usage --category normal --delta 1 2 3
# mean accuracy: [0.64455275 0.8368784  0.92165116 0.95906876]

python performance.py --data-root [data-root] --save-flag n100usage --category difficult --delta 1 2 4
# mean accuracy: [0.42088688 0.57836163 0.67878168 0.75491805]

This will generate performance figures and create data curves for two categories in folder "performance".

Citation

      @inproceedings{wang2020visual,
        title={Visual memorability for robotic interestingness via unsupervised online learning},
        author={Wang, Chen and Wang, Wenshan and Qiu, Yuheng and Hu, Yafei and Scherer, Sebastian},
        booktitle={European Conference on Computer Vision (ECCV)},
        year={2020},
        organization={Springer}
      }
      
      @article{wang2021unsupervised,
        title={Unsupervised Online Learning for Robotic Interestingness with Visual Memory},
        author={Wang, Chen and  Qiu, Yuheng and Wang, Wenshan and Hu, Yafei anad Kim, Seungchan and Scherer, Sebastian},
        journal={IEEE Transactions on Robotics (T-RO)},
        year={2021},
        publisher={IEEE}
      }

Download conferencec version paper.
Download journal version paper.

You may watch the following video to catch the idea of this work.

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh

44 Dec 14, 2022

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

207 Jan 5, 2023

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Causality In Traffic Accident (Under Construction) Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020) Overview Data Prepa

21 Nov 20, 2022

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

54 Dec 15, 2022

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

98 Dec 7, 2022

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

32 Dec 13, 2022

SNE-RoadSeg in PyTorch, ECCV 2020

SNE-RoadSeg Introduction This is the official PyTorch implementation of SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentati

242 Dec 20, 2022

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Gradient-Induced Co-Saliency Detection Zhao Zhang*, Wenda Jin*, Jun Xu, Ming-Ming Cheng ⭐ Project Home » The official repo of the ECCV 2020 paper Grad

35 Nov 25, 2022

Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

85 Dec 22, 2022

Comments

Variable

https://github.com/wang-chen/interestingness/blob/6994d50bd47d14b617f34f5c36c1beaba03acfdc/test_interest.py#L94

I think using Variable() will just return a tensor object in the new pytorch version.

opened by haleqiu 2

Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Related tags

Overview

Visual Interestingness

Install Dependencies

Long-term Learning

Short-term Learning

On-line Learning

Evaluation

Citation

You might also like...

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

SNE-RoadSeg in PyTorch, ECCV 2020

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Code for Towards Streaming Perception (ECCV 2020) :car:

Comments

Variable

Releases(v2.0)

v2.0(Apr 12, 2021)

v1.0(Jun 19, 2020)

Owner

Chen Wang

Vehicle speed detection with python

Generative Art Using Neural Visual Grammars and Dual Encoders

Code and data form the paper BERT Got a Date: Introducing Transformers to Temporal Tagging

CTF challenges and write-ups for MicroCTF 2021.

we propose a novel deep network, named feature aggregation and refinement network (FARNet), for the automatic detection of anatomical landmarks.

Invert and perturb GAN images for test-time ensembling

Do you like Quick, Draw? Well what if you could train/predict doodles drawn inside Streamlit? Also draws lines, circles and boxes over background images for annotation.

Layer 7 DDoS Panel with Cloudflare Bypass ( UAM, CAPTCHA, BFM, etc.. )

SEAN: Image Synthesis with Semantic Region-Adaptive Normalization (CVPR 2020, Oral)

SSD-based Object Detection in PyTorch

Learning Saliency Propagation for Semi-supervised Instance Segmentation

Code for our paper Aspect Sentiment Quad Prediction as Paraphrase Generation in EMNLP 2021.

Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization

Example how to deploy deep learning model with aiohttp.

RIFE - Real-Time Intermediate Flow Estimation for Video Frame Interpolation

Implementation of light baking system for ray tracing based on Activision's UberBake

(SIGIR2020) “Asymmetric Tri-training for Debiasing Missing-Not-At-Random Explicit Feedback’’

DeepOBS: A Deep Learning Optimizer Benchmark Suite

Like ThreeJS but for Python and based on wgpu

SNIPS: Solving Noisy Inverse Problems Stochastically