Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

Adversarial Reweighting for Partial Domain Adaptation

PyTorch implementation of SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

Pytorch reimplementation of the Mixer (MLP-Mixer: An all-MLP Architecture for Vision)

PN-Net a neural field-based framework for depth estimation from single-view RGB images.

FLAVR is a fast, flow-free frame interpolation method capable of single shot multi-frame prediction

A Peer-to-peer Platform for Secure, Privacy-preserving, Decentralized Data Science

Reference PyTorch implementation of "End-to-end optimized image compression with competition of prior distributions"

Enigma-Plus - Python based Enigma machine simulator with some extra features

This code is part of the reproducibility package for the SANER 2022 paper "Generating Clarifying Questions for Query Refinement in Source Code Search".

School of Artificial Intelligence at the Nanjing University (NJU)School of Artificial Intelligence at the Nanjing University (NJU)

CVAT is free, online, interactive video and image annotation tool for computer vision

Machine Learning University: Accelerated Computer Vision Class

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Lightwood is Legos for Machine Learning.

This repository contains the scripts for downloading and validating scripts for the documents

Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time

The easiest tool for extracting radiomics features and training ML models on them.

A Model for Natural Language Attack on Text Classification and Inference

MANO hand model porting for the GraspIt simulator

Implementation for "Conditional entropy minimization principle for learning domain invariant representation features"