"Video Moment Retrieval from Text Queries via Single Frame Annotation" in SIGIR 2022.

Overview

ViGA: Video moment retrieval via Glance Annotation

This is the official repository of the paper "Video Moment Retrieval from Text Queries via Single Frame Annotation" published in SIGIR 2022.

https://arxiv.org/abs/2204.09409

Dependencies

This project has been tested on the following conda environment.

$ conda create --name viga python=3.7
$ source activate viga
(viga)$ conda install pytorch=1.10.0 cudatoolkit=11.3.1
(viga)$ pip install numpy scipy pyyaml tqdm 

Data preparation

This repository contains our glance annotations already. To replicate our work, one should prepare extra data and finally get the following structure.

ckpt/                                 our pre-trained model, available at https://drive.google.com/file/d/1S4e8XmIpiVFJKSSJ4Tig4qN0yaCwiVLs/view?usp=sharing
data/
+-- activitynetcaptions/
|   +-- c3d/                    
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- train.json                
|   |   |   +-- val_1.json                
|   |   |   +-- val_2.json   
|   |   +-- train.json                downloaded
|   |   +-- val_1.json                downloaded
|   |   +-- val_2.json                downloaded
+-- charadessta/
|   +-- i3d/                     
|   +-- c3d/ 
|   +-- vgg/
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- charades_sta_train.txt
|   |   |   +-- charades_sta_test.txt
|   |   +-- charades_sta_train.txt    downloaded
|   |   +-- charades_sta_test.txt     downloaded
|   |   +-- Charades_v1_train.csv     downloaded
|   |   +-- Charades_v1_test.csv      downloaded
+-- tacos/
|   +-- c3d/ 
|   +-- annotations/
|   |   +-- glance/
|   |   |   +-- train.json                
|   |   |   +-- test.json                 
|   |   |   +-- val.json
|   |   +-- train.json                downloaded
|   |   +-- test.json                 downloaded
|   |   +-- val.json                  downloaded
glove.840B.300d.txt                   downloaded from https://nlp.stanford.edu/data/glove.840B.300d.zip

1. ActivityNet Captions

c3d feature

Downloaded from http://activity-net.org/challenges/2016/download.html. We extracted the features from sub_activitynet_v1-3.c3d.hdf5 as individual files.

Folder contains 19994 vid.npys, each of shape (T, 500).

annotation

Downloaded from https://cs.stanford.edu/people/ranjaykrishna/densevid/

2. Charades-STA

c3d feature

We extracted this by ourselves, due to the lack of storage resource we are currently not able to make this feature publicly available.

Folder contains 9848 vid.npys, each of shape (T, 4096).

i3d feature

Downloaded from https://github.com/JonghwanMun/LGI4temporalgrounding. This is the features extracted from I3D (finetuned on Charades). We processed them by trimming off unnecessary dimensions.

Folder contains 9848 vid.npys, each of shape (T, 1024).

vgg feature

Downloaded from https://github.com/microsoft/2D-TAN. We processed the data by converting the downloaded version vgg_rgb_features.hdf5 into numpy arrays.

Folder contains 6672 vid.npys, each of shape (T, 4096).

annotation

Downloaded from https://github.com/jiyanggao/TALL

3. TACoS

c3dfeature

Downloaded from https://github.com/microsoft/2D-TAN. We extracted the features from tall_c3d_features.hdf5 as individual files.

Folder contains 127 vid.npys, each of shape (T, 4096).

annotation

Downloaded from https://github.com/microsoft/2D-TAN

2. Run

Our models were trained using the following commands.

(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task activitynetcaptions
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task charadessta
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.train --task tacos

Our trained models were evaluated using the following commands.

(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/activitynetcaptions
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_c3d
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_i3d
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/charadessta_vgg
(viga)$ CUDA_VISIBLE_DEVICES=0 python -m src.experiment.eval --exp ckpt/tacos
Owner
Ran Cui
Ran Cui
Scans all drives for log4j jar files and gets their version from the manifest

log4shell_scanner Scans all drives for log4j jar files and gets their version from the manifest. Windows and Windows Server only.

Zdeněk Loučka 1 Dec 29, 2021
nuclei scanner for proxyshell ( CVE-2021-34473 )

Proxyshell-Scanner nuclei scanner for Proxyshell RCE (CVE-2021-34423,CVE-2021-34473,CVE-2021-31207) discovered by orange tsai in Pwn2Own, which affect

PikaChu 29 Dec 16, 2022
BurpSuite Extension: Log4j RCE Scanner

BurpSuite Extension: Log4j RCE Scanner

1 Dec 16, 2021
This repo is about steps to create a effective custom wordlist in a few clicks/

Custom Wordlist This repo is about steps to take in order to create a effective custom wordlist in a few clicks. this comes handing in pentesting enga

2 Oct 08, 2022
MITMSDR for INDIAN ARMY cybersecurity hackthon

There mainly three things here: MITMSDR spectrum Manual reverse shell MITMSDR Installation Clone the project and run the setup file: ./setup One of th

2 Jul 26, 2022
Spring4Shell - Spring Core RCE - CVE-2022-22965

Spring Core RCE - CVE-2022-22965 After Spring Cloud, on March 29, another heavyweight vulnerability of Spring broke out on the Internet: Spring Core R

Malte Gejr 118 Dec 31, 2022
A cross-platform Python module that displays **** for password input. Works on Windows, unlike getpass. Formerly called stdiomask.

PWInput A cross-platform Python module that displays **** for password input. Works on Windows, unlike getpass. Formerly called stdiomask. Installatio

Al Sweigart 26 Sep 04, 2022
Exploiting CVE-2021-42278 and CVE-2021-42287 to impersonate DA from standard domain user

Exploiting CVE-2021-42278 and CVE-2021-42287 to impersonate DA from standard domain user Known issues it will not work outside kali , i will update it

Hossam 867 Dec 22, 2022
Tools ini digunakan untuk krekk pacebuk:v

E-Crack By Aang-XD Fitur Login • Login via token facebook • Login via cookie facebook Install On Termux $ pkg update && pkg upgrade $ pkg install pyth

Aang Ardiansyah-XD 2 Dec 24, 2021
Yesitsme - Simple OSINT script to find Instagram profiles by name and e-mail/phone

Simple OSINT script to find Instagram profiles by name and e-mail/phone

108 Jan 07, 2023
A simple Log4Shell Scan with python

🐞 Log4Scan 🔧 Log4Shell 简单的主动和被动扫描脚本 Log4scan 针对header头和fuzz参数的主动批量扫描,用于大批量黑盒检测

nul1 6 Aug 04, 2022
SSH Tool For OSINT and then Cracking.

sshmap SSH Tool For OSINT and then Cracking. Linux Systems Only Usage: Scanner Syntax: scanner start/stop/status - Sarts/stops/sho

Miss Bliss 5 Apr 04, 2022
This repo created for bypassing Widevine L3 DRM and obtaining keys.

First run: Copy headers (with cookies) of POST license request from browser to headers.py like dictionary. pip install -r requirements.txt # if doesn'

Mikhail 263 Jan 07, 2023
A Simple File Encryptor/Decryptor

Ec: A Simple File Encryptor/Decryptor This has been made for educational reasons only, any constructive criticism/advice/comments are welcome! Also, p

1 Dec 10, 2021
Caretaker 2 Jun 06, 2022
Discord Token Stealer Malware Protection

TokenGuard TokenGuard, protect your account, prevent token steal. Totally free and open source Discord Server: https://discord.gg/EmwfaGuBE8 Source Co

10 Nov 23, 2022
Big-Papa Integrates Javascript and python for remote cookie stealing which then can be used for session hijacking

Big-Papa is a remote cookie stealer which can then be used for session hijacking and Bypassing 2 Factor Authentication

77 Jan 03, 2023
SecurAID securely connects aid organizations directly with individuals in dangerous situations to allow them to discreetly and effectively get the assistance they need.

SecurAID securely connects aid organizations directly with individuals in dangerous situations to allow them to discreetly and effec

Ty K 2 Mar 23, 2022
evtx-hunter helps to quickly spot interesting security-related activity in Windows Event Viewer (EVTX) files.

Introduction evtx-hunter helps to quickly spot interesting security-related activity in Windows Event Viewer (EVTX) files. It can process a high numbe

NVISO 116 Dec 29, 2022
A (completely native) python3 wifi brute-force attack using the 100k most common passwords (2021)

wifi-bf [LINUX ONLY] A (completely native) python3 wifi brute-force attack using the 100k most common passwords (2021) This script is purely for educa

Finn Lancaster 20 Nov 12, 2022