AudioCLIP
Extending CLIP to Image, Text and Audio
This repository contains implementation of the models described in the paper arXiv:2106.13043. This work based on our previous works:
- ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio (2021).
- ESResNet: Environmental Sound Classification Based on Visual Domain Models (2020).
Abstract
In the past, the rapidly evolving field of sound classification greatly benefited from the application of methods from other domains. Today, we observe the trend to fuse domain-specific tasks and approaches together, which provides the community with new outstanding models.
In this work, we present an extension of the CLIP model that handles audio in addition to text and images. Our proposed model incorporates the ESResNeXt audio-model into the CLIP framework using the AudioSet dataset. Such a combination enables the proposed model to perform bimodal and unimodal classification and querying, while keeping CLIP's ability to generalize to unseen datasets in a zero-shot inference fashion.
AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task, out-performing other approaches by reaching accuracies of 90.07% on the UrbanSound8K and 97.15% on the ESC-50 datasets. Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78% and 69.40%, respectively).
Finally, we also assess the cross-modal querying performance of the proposed model as well as the influence of full and partial training on the results. For the sake of reproducibility, our code is published.
How to Run the Model
The required Python version is >= 3.7.
AudioCLIP
On the ESC-50 dataset
python main.py --config protocols/audioclip-esc50.json --Dataset.args.root /path/to/ESC50
On the UrbanSound8K dataset
python main.py --config protocols/audioclip-us8k.json --Dataset.args.root /path/to/UrbanSound8K
Cite Us
@misc{guzhov2021audioclip,
title={AudioCLIP: Extending CLIP to Image, Text and Audio},
author={Andrey Guzhov and Federico Raue and Jörn Hees and Andreas Dengel},
year={2021},
eprint={2106.13043},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
1 Aug 19, 2021
612 Jan 4, 2023
2 Mar 4, 2022
82 Dec 26, 2022
2.3k Dec 29, 2022
23 Nov 30, 2022
39 Oct 27, 2022
4.8k Dec 30, 2022
46 Dec 15, 2022
1 Jan 13, 2022
7 Nov 29, 2022
8 Dec 25, 2022
13 Dec 21, 2022
10 Jul 19, 2022
7 Nov 07, 2022
7 Apr 28, 2022
592 Dec 18, 2022
197 Dec 11, 2022
1 Jun 05, 2022
26 Oct 17, 2022
248 Dec 14, 2022
66 Dec 29, 2022
1 Jan 29, 2022
283 Nov 21, 2022
298 Nov 21, 2022
9 Sep 13, 2021
4 Jan 22, 2022
8 Oct 13, 2022