SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Last update: Aug 30, 2022

Related tags

Deep Learning SAAVN

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

These codes are based on the SoundSpaces code base.

Usage

This repo supports AudioGoal Task on Replica and Matterport3D datasets.

Below we show the commands for training and evaluating AudioGoal with Depth sensor on Replica, but it applies to Matterport dataset as well.

Training

python main.py --default av_nav --run-type train --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Validation (evaluate each checkpoint and generate a validation curve)

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Test the best validation checkpoint based on validation curve

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Generate demo video with audio

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Note: [exp_config_file] is the main parameter configuration file of the experiment, while [tag_config_file] is special parameter configuration file for abalation experiments.

Citation

If you use this model in your research, please cite the following paper:

@inproceedings{YinfengICLR2022saavn,
	title = {Sound Adversarial Audio-Visual Navigation},
	author = {Yinfeng Yu, Wenbing Huang, Fuchun Sun, Changan Chen, Yikai Wang, Xiaohong Liu},
	year = {2022},
        booktitle={ICLR},
}

SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Related tags

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

Usage

Citation

Owner

YinfengYu

Byzantine-robust decentralized learning via self-centered clipping

NovelD: A Simple yet Effective Exploration Criterion

basic tutorial on pytorch

Code for "NeRS: Neural Reflectance Surfaces for Sparse-View 3D Reconstruction in the Wild," in NeurIPS 2021

AttGAN: Facial Attribute Editing by Only Changing What You Want (IEEE TIP 2019)

This repository contains the code for designing risk bounded motion plans for car-like robot using Carla Simulator.

Jremesh-tools - Blender addon for quad remeshing

This repository compare a selfie with images from identity documents and response if the selfie match.

PyTorch code for DriveGAN: Towards a Controllable High-Quality Neural Simulation

ObsPy: A Python Toolbox for seismology/seismological observatories.

Optimus: the first large-scale pre-trained VAE language model

Code for EMNLP 2021 paper Contrastive Out-of-Distribution Detection for Pretrained Transformers.

A Simple Key-Value Data-store written in Python

Flexible-Modal Face Anti-Spoofing: A Benchmark

Miscellaneous and lightweight network tools

This is the code repository for the paper "Identification of the Generalized Condorcet Winner in Multi-dueling Bandits" (NeurIPS 2021).

An experiment on the performance of homemade Q-learning AIs in Agar.io depending on their state representation and available actions

Code for: Gradient-based Hierarchical Clustering using Continuous Representations of Trees in Hyperbolic Space. Nicholas Monath, Manzil Zaheer, Daniel Silva, Andrew McCallum, Amr Ahmed. KDD 2019.

Code for paper: Towards Tokenized Human Dynamics Representation