SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Last update: Aug 30, 2022

Related tags

Deep Learning SAAVN

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

These codes are based on the SoundSpaces code base.

Usage

This repo supports AudioGoal Task on Replica and Matterport3D datasets.

Below we show the commands for training and evaluating AudioGoal with Depth sensor on Replica, but it applies to Matterport dataset as well.

Training

python main.py --default av_nav --run-type train --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Validation (evaluate each checkpoint and generate a validation curve)

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Test the best validation checkpoint based on validation curve

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Generate demo video with audio

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Note: [exp_config_file] is the main parameter configuration file of the experiment, while [tag_config_file] is special parameter configuration file for abalation experiments.

Citation

If you use this model in your research, please cite the following paper:

@inproceedings{YinfengICLR2022saavn,
	title = {Sound Adversarial Audio-Visual Navigation},
	author = {Yinfeng Yu, Wenbing Huang, Fuchun Sun, Changan Chen, Yikai Wang, Xiaohong Liu},
	year = {2022},
        booktitle={ICLR},
}

SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Related tags

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

Usage

Citation

Owner

YinfengYu

[SDM 2022] Towards Similarity-Aware Time-Series Classification

A list of all named GANs!

Layered Neural Atlases for Consistent Video Editing

Implement the Pareto Optimizer and pcgrad to make a self-adaptive loss for multi-task

Repositório para arquivos sobre o Módulo 1 do curso Top Coders da Let's Code + Safra

Interactive Image Segmentation via Backpropagating Refinement Scheme

Custom IMDB Dataset is extracted between 2020-2021 and custom distilBERT model is trained for movie success probability prediction

A Transformer-Based Siamese Network for Change Detection

The materials used in the SaxonJS tutorial presented at Declarative Amsterdam, 2021

Python package for multiple object tracking research with focus on laboratory animals tracking.

Synthetic Scene Text from 3D Engines

Code for the paper A Theoretical Analysis of the Repetition Problem in Text Generation

[ICLR2021] Unlearnable Examples: Making Personal Data Unexploitable

This is the repo of the manuscript "Dual-branch Attention-In-Attention Transformer for speech enhancement"

A stock generator that assess a list of stocks and returns the best stocks for investing and money allocations based on users choices of volatility, duration and number of stocks

Single-Stage 6D Object Pose Estimation, CVPR 2020

This python-based package offers a way of creating a parametric OpenMC plasma source from plasma parameters.

Multi-Target Adversarial Frameworks for Domain Adaptation in Semantic Segmentation

AI assistant built in python.the features are it can display time,say weather,open-google,youtube,instagram.

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss