Search and filter videos based on objects that appear in them using convolutional neural networks

Overview

Thingscoop: Utility for searching and filtering videos based on their content

Description

Thingscoop is a command-line utility for analyzing videos semantically - that means searching, filtering, and describing videos based on objects, places, and other things that appear in them.

When you first run thingscoop on a video file, it uses a convolutional neural network to create an "index" of what's contained in the every second of the input by repeatedly performing image classification on a frame-by-frame basis. Once an index for a video file has been created, you can search (i.e. get the start and end times of the regions in the video matching the query) and filter (i.e. create a supercut of the matching regions) the input using arbitrary queries. Thingscoop uses a very basic query language that lets you to compose queries that test for the presence or absence of labels with the logical operators ! (not), || (or) and && (and). For example, to search a video the presence of the sky and the absence of the ocean: thingscoop search 'sky && !ocean' <file>.

Right now two models are supported by thingscoop: vgg_imagenet uses the architecture described in "Very Deep Convolutional Networks for Large-Scale Image Recognition" to recognize objects from the ImageNet database, and googlenet_places uses the architecture described in "Going Deeper with Convolutions" to recognize settings and places from the MIT Places database. You can specify which model you'd like to use by running thingscoop models use <model>, where <model> is either vgg_imagenet or googlenet_places. More models will be added soon.

Thingscoop is based on Caffe, an open-source deep learning framework.

Installation

  1. Install ffmpeg, imagemagick, and ghostscript: brew install ffmpeg imagemagick ghostscript (Mac OS X) or apt-get install ffmpeg imagemagick ghostscript (Ubuntu).
  2. Follow the installation instructions on the Caffe Installation page.
  3. Make sure you build the Python bindings by running make pycaffe (on Caffe's directory).
  4. Set the environment variable CAFFE_ROOT to point to Caffe's directory: export CAFFE_ROOT=[Caffe's directory].
  5. Install thingscoop: easy_install thingscoop or pip install thingscoop.

Usage

thingscoop search <query> <files...>

Print the start and end times (in seconds) of the regions in <files> that match <query>. Creates an index for <file> using the current model if it does not exist.

Example output:

$ thingscoop search violin waking_life.mp4
/Users/anastasis/Downloads/waking_life.mp4 148.000000 162.000000
/Users/anastasis/Downloads/waking_life.mp4 176.000000 179.000000
/Users/anastasis/Downloads/waking_life.mp4 180.000000 186.000000
/Users/anastasis/Downloads/waking_life.mp4 189.000000 190.000000
/Users/anastasis/Downloads/waking_life.mp4 192.000000 200.000000
/Users/anastasis/Downloads/waking_life.mp4 211.000000 212.000000
/Users/anastasis/Downloads/waking_life.mp4 222.000000 223.000000
/Users/anastasis/Downloads/waking_life.mp4 235.000000 243.000000
/Users/anastasis/Downloads/waking_life.mp4 247.000000 249.000000
/Users/anastasis/Downloads/waking_life.mp4 251.000000 253.000000
/Users/anastasis/Downloads/waking_life.mp4 254.000000 258.000000

####thingscoop filter <query> <files...>

Generate a video compilation of the regions in the <files> that match <query>. Creates index for <file> using the current model if it does not exist.

Example output:

thingscoop sort <file>

Create a compilation video showing examples for every label recognized in the video (in alphabetic order). Creates an index for <file> using the current model if it does not exist.

Example output:

thingscoop describe <file>

Print every label that appears in <file> along with the number of times it appears. Creates an index for <file> using the current model if it does not exist.

thingscoop preview <file>

Create a window that plays the input video <file> while also displaying the labels the model recognizes on every frame.

$ thingscoop describe koyaanisqatsi.mp4 -m googlenet_places
sky 405
skyscraper 363
canyon 141
office_building 130
highway 78
lighthouse 66
hospital 64
desert 59
shower 49
volcano 45
underwater 44
airport_terminal 43
fountain 39
runway 36
assembly_line 35
aquarium 34
fire_escape 34
music_studio 32
bar 28
amusement_park 28
stage 26
wheat_field 25
butchers_shop 25
engine_room 24
slum 20
butte 20
igloo 20
...etc

thingscoop index <file>

Create an index for <file> using the current model if it does not exist.

thingscoop models list

List all models currently available in Thingscoop.

$ thingscoop models list
googlenet_imagenet            Model described in the paper "Going Deeper with Convolutions" trained on the ImageNet database
googlenet_places              Model described in the paper "Going Deeper with Convolutions" trained on the MIT Places database
vgg_imagenet                  16-layer model described in the paper "Return of the Devil in the Details: Delving Deep into Convolutional Nets" trained on the ImageNet database

thingscoop models info <model>

Print more detailed information about <model>.

$ thingscoop models info googlenet_places
Name: googlenet_places
Description: Model described in the paper "Going Deeper with Convolutions" trained on the MIT Places database
Dataset: MIT Places

thingscoop models freeze

List all models that have already been downloaded.

$ thingscoop models freeze
googlenet_places
vgg_imagenet

thingscoop models current

Print the model that is currently in use.

$ thingscoop models current
googlenet_places

thingscoop models use <model>

Set the current model to <model>. Downloads that model locally if it hasn't been downloaded already.

thingscoop models download <model>

Download the model <model> locally.

thingscoop models remove <model>

Remove the model <model> locally.

thingscoop models clear

Remove all models stored locally.

thingscoop labels list

Print all the labels used by the current model.

$ thingscoop labels list
abacus
abaya
abstraction
academic gown
accessory
accordion
acorn
acorn squash
acoustic guitar
act
actinic radiation
action
activity
adhesive bandage
adjudicator
administrative district
admiral
adornment
adventurer
advocate
...

thingscoop labels search <regexp>

Print all the labels supported by the current model that match the regular expression <regexp>.

$ thingscoop labels search instrument$
beating-reed instrument
bowed stringed instrument
double-reed instrument
free-reed instrument
instrument
keyboard instrument
measuring instrument
medical instrument
musical instrument
navigational instrument
negotiable instrument
optical instrument
percussion instrument
scientific instrument
stringed instrument
surveying instrument
wind instrument
...

Full usage options

thingscoop - Command-line utility for searching and filtering videos based on their content

Usage:
  thingscoop filter <query> <files>... [-o <output_path>] [-m <model>] [-s <sr>] [-c <mc>] [--recreate-index] [--gpu-mode] [--open]
  thingscoop search <query> <files>... [-o <output_path>] [-m <model>] [-s <sr>] [-c <mc>] [--recreate-index] [--gpu-mode] 
  thingscoop describe <file> [-n <words>] [-m <model>] [--recreate-index] [--gpu-mode] [-c <mc>]
  thingscoop index <files> [-m <model>] [-s <sr>] [-c <mc>] [-r <ocr>] [--recreate-index] [--gpu-mode] 
  thingscoop sort <file> [-m <model>] [--gpu-mode] [--min-confidence <ct>] [--max-section-length <ms>] [-i <ignore>] [--open]
  thingscoop preview <file> [-m <model>] [--gpu-mode] [--min-confidence <ct>]
  thingscoop labels list [-m <model>]
  thingscoop labels search <regexp> [-m <model>]
  thingscoop models list
  thingscoop models info <model>
  thingscoop models freeze
  thingscoop models current
  thingscoop models use <model>
  thingscoop models download <model>
  thingscoop models remove <model>
  thingscoop models clear

Options:
  --version                       Show version.
  -h --help                       Show this screen.
  -o --output <dst>               Output file for supercut
  -s --sample-rate <sr>           How many frames to classify per second (default = 1)
  -c --min-confidence <mc>        Minimum prediction confidence required to consider a label (default depends on model)
  -m --model <model>              Model to use (use 'thingscoop models list' to see all available models)
  -n --number-of-words <words>    Number of words to describe the video with (default = 5)
  -t --max-section-length <ms>    Max number of seconds to show examples of a label in the sorted video (default = 5)
  -r --min-occurrences <ocr>      Minimum number of occurrences of a label in video required for it to be shown in the sorted video (default = 2)
  -i --ignore-labels <labels>     Labels to ignore when creating the sorted video video
  --title <title>                 Title to show at the beginning of the video (sort mode only)
  --gpu-mode                      Enable GPU mode
  --recreate-index                Recreate object index for file if it already exists
  --open                          Open filtered video after creating it (OS X only)

CHANGELOG

0.2 (8/16/2015)

  • Added sort option for creating a video compilation of all labels appearing in a video
  • Now using JSON for the index files

0.1 (8/5/2015)

  • Conception

License

MIT

Owner
Anastasis Germanidis
🎭
Anastasis Germanidis
PyTorch implementation for ComboGAN

ComboGAN This is our ongoing PyTorch implementation for ComboGAN. Code was written by Asha Anoosheh (built upon CycleGAN) [ComboGAN Paper] If you use

Asha Anoosheh 139 Dec 20, 2022
Snscrape-jsonl-urls-extractor - Extracts urls from jsonl produced by snscrape

snscrape-jsonl-urls-extractor extracts urls from jsonl produced by snscrape Usag

1 Feb 26, 2022
Tensorflow implementation of DeepLabv2

TF-deeplab This is a Tensorflow implementation of DeepLab, compatible with Tensorflow 1.2.1. Currently it supports both training and testing the ResNe

Chenxi Liu 21 Sep 27, 2022
PyTorch Implementation of Temporal Output Discrepancy for Active Learning, ICCV 2021

Temporal Output Discrepancy for Active Learning PyTorch implementation of Semi-Supervised Active Learning with Temporal Output Discrepancy, ICCV 2021.

Siyu Huang 33 Dec 06, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 01, 2023
Official Pytorch implementation of 6DRepNet: 6D Rotation representation for unconstrained head pose estimation.

6D Rotation Representation for Unconstrained Head Pose Estimation (Pytorch) Paper Thorsten Hempel and Ahmed A. Abdelrahman and Ayoub Al-Hamadi, "6D Ro

Thorsten Hempel 284 Dec 23, 2022
Recursive Bayesian Networks

Recursive Bayesian Networks This repository contains the code to reproduce the results from the NeurIPS 2021 paper Lieck R, Rohrmeier M (2021) Recursi

Robert Lieck 11 Oct 18, 2022
DIVeR: Deterministic Integration for Volume Rendering

DIVeR: Deterministic Integration for Volume Rendering This repo contains the training and evaluation code for DIVeR. Setup python 3.8 pytorch 1.9.0 py

64 Dec 27, 2022
ToFFi - Toolbox for Frequency-based Fingerprinting of Brain Signals

ToFFi Toolbox This repository contains "before peer review" version of the software related to the preprint of the publication ToFFi - Toolbox for Fre

4 Aug 31, 2022
Codebase for ECCV18 "The Sound of Pixels"

Sound-of-Pixels Codebase for ECCV18 "The Sound of Pixels". *This repository is under construction, but the core parts are already there. Environment T

Hang Zhao 318 Dec 20, 2022
A way to store images in YAML.

YAMLImg A way to store images in YAML. I made this after seeing Roadcrosser's JSON-G because it was too inspiring to ignore this opportunity. Installa

5 Mar 14, 2022
The code from the paper Character Transformations for Non-Autoregressive GEC Tagging

Character Transformations for Non-Autoregressive GEC Tagging Milan Straka, Jakub Náplava, Jana Straková Charles University Faculty of Mathematics and

ÚFAL 5 Dec 10, 2022
Repository for XLM-T, a framework for evaluating multilingual language models on Twitter data

This is the XLM-T repository, which includes data, code and pre-trained multilingual language models for Twitter. XLM-T - A Multilingual Language Mode

Cardiff NLP 112 Dec 27, 2022
Code release for the paper “Worldsheet Wrapping the World in a 3D Sheet for View Synthesis from a Single Image”, ICCV 2021.

Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image This repository contains the code for the following paper: R. Hu,

Meta Research 37 Jan 04, 2023
OpenMMLab Model Deployment Toolset

Introduction English | 简体中文 MMDeploy is an open-source deep learning model deployment toolset. It is a part of the OpenMMLab project. Major features F

OpenMMLab 1.5k Dec 30, 2022
RINDNet: Edge Detection for Discontinuity in Reflectance, Illumination, Normal and Depth, in ICCV 2021 (oral)

RINDNet RINDNet: Edge Detection for Discontinuity in Reflectance, Illumination, Normal and Depth Mengyang Pu, Yaping Huang, Qingji Guan and Haibin Lin

Mengyang Pu 75 Dec 15, 2022
An auto discord account and token generator. Automatically verifies the phone number. Works without proxy. Bypasses captcha.

JOIN DISCORD SERVER https://discord.gg/uAc3agBY FREE HCAPTCHA SOLVING API Discord-Token-Gen An auto discord token generator. Auto verifies phone numbe

3kp 271 Jan 01, 2023
Bu repo SAHI uygulamasını mantığını öğreniyoruz.

SAHI-Learn: SAHI'den Beraber Kodlamak İster Misiniz Herkese merhabalar ben Kadir Nar. SAHI kütüphanesine gönüllü geliştiriciyim. Bu repo SAHI kütüphan

Kadir Nar 11 Aug 22, 2022
[NeurIPS 2021] COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

COCO-LM This repository contains the scripts for fine-tuning COCO-LM pretrained models on GLUE and SQuAD 2.0 benchmarks. Paper: COCO-LM: Correcting an

Microsoft 106 Dec 12, 2022
DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks

DECAF (DEbiasing CAusal Fairness) Code Author: Trent Kyono This repository contains the code used for the "DECAF: Generating Fair Synthetic Data Using

van_der_Schaar \LAB 7 Nov 24, 2022