Command-line tool for downloading and extending the RedCaps dataset.

Overview

RedCaps Downloader

This repository provides the official command-line tool for downloading and extending the RedCaps dataset. Users can seamlessly download images of officially released annotations as well as download more image-text data from any subreddit over an arbitrary time-span.

Installation

This tool requires Python 3.8 or higher. We recommend using conda for setup. Download Anaconda or Miniconda first. Then follow these steps:

# Clone the repository.
git clone https://github.com/redcaps-dataset/redcaps-downloader
cd redcaps-downloader

# Create a new conda environment.
conda create -n redcaps python=3.8
conda activate redcaps

# Install dependencies along with this code.
pip install -r requirements.txt
python setup.py develop

Basic usage: Download official RedCaps dataset

We expect most users will only require this functionality. Follow these steps to download the official RedCaps annotations and images and arrange all the data in recommended directory structure:

/path/to/redcaps/
├── annotations/
│   ├── abandoned_2017.json
│   ├── abandoned_2017.json
│   ├── ...
│   ├── itookapicture_2019.json
│   ├── itookapicture_2020.json
│   ├── 
   
    _
    
     .json
│   └── ...
│
└── images/
    ├── abandoned/
    │   ├── guli1.jpg
    |   └── ...
    │
    ├── itookapicture/
    │   ├── 1bd79.jpg
    |   └── ...
    │
    ├── 
     
      /
    │   ├── 
      
       .jpg
    │   ├── ...
    └── ...

      
     
    
   
  1. Create an empty directory and symlink it relative to this code directory:

    cd redcaps-downloader
    
    # Edit path here:
    mkdir -p /path/to/redcaps
    ln -s /path/to/redcaps ./datasets/redcaps
  2. Download official RedCaps annotations from Dropbox and unzip them.

    cd datasets/redcaps
    wget https://www.dropbox.com/s/twvv541sbg0qqux/redcaps_v1.0_annotations.zip?dl=1
    unzip redcaps_v1.0_annotations.zip
  3. Download images by using redcaps download-imgs command (for a single annotation file).

    for ann_file in ./datasets/redcaps/annotations/*.json; do
        redcaps download-imgs -a $ann_file --save-to path/to/images --resize 512 -j 4
        # Set --resize -1 to turn off resizing shorter edge (saves disk space).
    done

    Parallelize download by changing -j. RedCaps images are sourced from Reddit, Imgur and Flickr, each have their own request limits. This code contains approximate sleep intervals to manage them. Use multiple machines (= different IP addresses) or a cluster to massively parallelize downloading.

That's it, you are all set to use RedCaps!

Advanced usage: Create your own RedCaps-like dataset

Apart from downloading the officially released dataset, this tool supports downloading image-text data from any subreddit – you can reproduce the entire collection pipeline as well as create your own variant of RedCaps! Here, we show how to collect annotations from r/roses (2020) as an example. Follow these steps for any subreddit and years.

Additional one-time setup instructions

RedCaps annotations are extracted from image post metadata, which are served by the Pushshift API and official Reddit API. These APIs are authentication-based, and one must sign up for developer access to obtain API keys (one-time setup):

  1. Copy ./credentials.template.json to ./credentials.json. Its contents are as follows:

    : " }, "imgur": { "client_id": "Your client ID here", "client_secret": "Your client secret here" } } ">
    {
        "reddit": {
            "client_id": "Your client ID here",
            "client_secret": "Your client secret here",
            "username": "Your Reddit username here",
            "password": "Your Reddit password here",
            "user_agent": "
          
           : 
           "
          
        },
        "imgur": {
            "client_id": "Your client ID here",
            "client_secret": "Your client secret here"
        }
    }
  2. Register a new Reddit app here. Reddit will provide a Client ID and Client Secret tokens - fill them in ./credentials.json. For more details, refer to the Reddit OAuth2 wiki. Enter your Reddit account name and password in ./credentials.json. Set User Agent to anything and keep it unchanged (e.g. your name).

  3. Register a new Imgur App by following instructions here. Fill the provided Client ID and Client Secret in ./credentials.json.

  4. Download pre-trained weights of an NSFW detection model.

    wget https://s3.amazonaws.com/nsfwdetector/nsfw.299x299.h5 -P ./datasets/redcaps/models

Data collection from r/roses (2020)

  1. download-anns: Dowload annotations of image posts made in a single month (e.g. January).

    redcaps download-anns --subreddit roses --month 2020-01 -o ./datasets/redcaps/annotations
    
    # Similarly, download annotations for all months of 2020:
    for ((month = 1; month <= 12; month += 1)); do
        redcaps download-anns --subreddit roses --month 2020-$month -o ./datasets/redcaps/annotations
    done
    • NOTE: You may not get all the annotations present in official release as some of them may have disappeared (deleted) over time. After this step, the dataset directory would contain 12 annotation files:
        ./datasets/redcaps/
        └── annotations/
            ├── roses_2020-01.json
            ├── roses_2020-02.json
            ├── ...
            └── roses_2020-12.json
    
  2. merge: Merge all the monthly annotation files into a single file.

    redcaps merge ./datasets/redcaps/annotations/roses_2020-* \
        -o ./datasets/redcaps/annotations/roses_2020.json --delete-old
    • --delete-old will remove individual files after merging. After this step, the merged file will replace individual monthly files:
        ./datasets/redcaps/
        └── annotations/
            └── roses_2020.json
    
  3. download-imgs: Download all images for this annotation file. This step is same as (3) in basic usage.

    redcaps download-imgs --annotations ./datasets/redcaps/annotations/roses_2020.json \
        --resize 512 -j 4 -o ./datasets/redcaps/images --update-annotations
    • --update-annotations removes annotations whose images were not downloaded.
  4. filter-words: Filter all instances whose captions contain potentially harmful language. Any caption containing one of the 400 blocklisted words will be removed. This command modifies the annotation file in-place and deletes the corresponding images from disk.

    redcaps filter-words --annotations ./datasets/redcaps/annotations/roses_2020.json \
        --images ./datasets/redcaps/images
  5. filter-nsfw: Remove all instances having images that are flagged by an off-the-shelf NSFW detector. This command also modifies the annotation file in-place and deletes the corresponding images from disk.

    redcaps filter-nsfw --annotations ./datasets/redcaps/annotations/roses_2020.json \
        --images ./datasets/redcaps/images \
        --model ./datasets/redcaps/models/nsfw.299x299.h5
  6. filter-faces: Remove all instances having images with faces detected by an off-the-shelf face detector. This command also modifies the annotation file in-place and deletes the corresponding images from disk.

    redcaps filter-faces --annotations ./datasets/redcaps/annotations/roses_2020.json \
        --images ./datasets/redcaps/images  # Model weights auto-downloaded
  7. validate: All the above steps create a single annotation file (and downloads images) similar to official RedCaps annotations. To double-check this, run the following command and expect no errors to be printed.

    redcaps validate --annotations ./datasets/redcaps/annotations/roses_2020.json

Citation

If you find this code useful, please consider citing:

@inproceedings{desai2021redcaps,
    title={{RedCaps: Web-curated image-text data created by the people, for the people}},
    author={Karan Desai and Gaurav Kaul and Zubin Aysola and Justin Johnson},
    booktitle={NeurIPS Datasets and Benchmarks},
    year={2021}
}
Owner
RedCaps dataset
RedCaps dataset
A CLI for streaming, downloading anime shows. The shows data is indexed through GogoAnime.

Anime-cli A CLI for streaming, downloading anime shows. The shows data is indexed through GogoAnime. Please install mpv video-player for better experi

Chirag Singla 31 Oct 23, 2022
A Hikari command handler for people who love ducks.

A Hikari command handler for people who love ducks.

Jeremiah 2 Oct 09, 2022
Easily handle day to day CLI operation via Python instead of regular Bash programs.

pz Ever wished to use Python in Bash? Would you choose the Python syntax over sed, awk, ...? Should you exactly know what command would you use in Pyt

CZ.NIC 697 Jan 03, 2023
A Neat Application To Manage Your To-Do Lists.

WTD - What To Do? A Neat Application To Manage Your To-Do Lists. One folder can only have one to-do file. Running wth without any subcommands executes

Adam Vajda 1 Oct 24, 2021
A Bot Which Send Automatically Commands To Karuta Hub to Gain it's Currency

A Bot Which Send Automatically Commands To Karuta Hub to Gain it's Currency

HarshalWaykole 1 Feb 09, 2022
Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Phil Wang 4.4k Jan 09, 2023
A library for creating text-based graphs in the terminal

tplot is a Python package for creating text-based graphs. Useful for visualizing data to the terminal or log files.

Jeroen Delcour 164 Dec 14, 2022
CLI to show end-of-life dates for tools and technologies.

Python 3.9+ interface to endoflife.date to show end-of-life dates for tools and technologies.

Hugo van Kemenade 32 Jan 06, 2023
Bringing emacs' greatest feature to neovim - Tetris!

nvim-tetris Bringing emacs' greatest feature to neovim - Tetris! This plugin is written in Fennel using Olical's project Aniseed for creating the proj

129 Dec 26, 2022
A minimal and ridiculously good looking command-line-interface toolkit.

Pyceo Pyceo is a Python package for creating beautiful, composable, and ridiculously good looking command-line-user-interfaces without having to write

Juan-Pablo Scaletti 21 Mar 25, 2022
A simple note taker CLI program written in python

note-taker A simple note taker program written in python This allows you to snip your todo's, notes, and your tasks easily without extra charges Requi

marcusz 4 Nov 02, 2021
A **CLI** folder organizer written in Python.

Fsorter Introduction A CLI folder organizer written in Python. Dependencies Before installing, install the following dependencies: Ubuntu/Debain Based

1 Nov 17, 2021
pyGinit is a command line tools that help you to initialize your current project a local git repo and remote repo

pyGinit pyGinit is a command line tools that help you to initialize your current project a local git repo and remote repo Requirements Requirements be

AlphaBeta 15 Feb 26, 2022
Arithmos cipher on CLI based

Arithmos Cipher CLI This is the CLI version of Arithmos Cipher. Install pip inst

LyQuid :3 1 Jan 16, 2022
A command line interface to interact with the Hypixel api allowing the user to get stats, leaderboards, etc

HyConsole is a way to get data on players and leaderboards from the Hypixel Minecraft server from the command line. Keep in mind I have no a

1 Feb 14, 2022
Gitfetch is a simple tool to get github user details

Gitfetch Just a (cli?) tool to get github user details 🙂 Installation 📂 Install Gitfetch via pypi pip install gitfetch or pip install git+https://g

I'm Not A Bot #Left_TG 7 Jan 23, 2022
A python program to detect your emotion and suggest a playlist.

mood_music A python program to detect your emotion and suggest a playlist. This program is written in python using opencv,FER() and tensorflow This pr

Aditya_Sai 1 Jan 02, 2022
Tidier - a simple command line tool that helps you make your files tidy up

Tidier - a simple command line tool that helps you make your files tidy up

AmirMohammad Hosseini Nasab 8 Aug 16, 2022
ForX - get forex quotes from the terminal

A command line tool for checking exchange rates between currencies, both crypto and fiat.

Gabe Banks 52 Dec 10, 2022
Double Pendulum visualised with fetching system information in Python.

Show off your terminal, in style. A nice relaxing double pendulum simulation using ASCII, able to simulate multiple pendulums at once, and provide tra

Nekurone 62 Dec 14, 2022