Codebase for BMVC 2021 paper "Text Based Person Search with Limited Data"

Related tags

Deep LearningTextReID
Overview

Text Based Person Search with Limited Data

PWC

This is the codebase for our BMVC 2021 paper.

Please bear with me refactoring this codebase after CVPR deadline πŸ˜…

Abstract

Text-based person search (TBPS) aims at retrieving a target person from an image gallery with a descriptive text query. Solving such a fine-grained cross-modal retrieval task is challenging, which is further hampered by the lack of large-scale datasets. In this paper, we present a framework with two novel components to handle the problems brought by limited data. Firstly, to fully utilize the existing small-scale benchmarking datasets for more discriminative feature learning, we introduce a cross-modal momentum contrastive learning framework to enrich the training data for a given mini-batch. Secondly, we propose to transfer knowledge learned from existing coarse-grained large-scale datasets containing image-text pairs from drastically different problem domains to compensate for the lack of TBPS training data. A transfer learning method is designed so that useful information can be transferred despite the large domain gap. Armed with these components, our method achieves new state of the art on the CUHK-PEDES dataset with significant improvements over the prior art in terms of Rank-1 and mAP.

Comments
  • Research prepared to obtain a diploma degree in computer and Automation Engineering.

    Research prepared to obtain a diploma degree in computer and Automation Engineering.

    Hello!

    My research focuses on Person search using Visual-Textual Attributes. Having said that, I would like to use your model to assist me in my project, but I have some issues when I finish train and test the model. My problem is trying to write code to run the model to get the same response as the photo. so Can you help me please!

    photo_2022-08-07_18-44-28

    opened by ram7772 6
  • Cannot find test_query and train_query folders

    Cannot find test_query and train_query folders

    Hi @BrandonHanx

    In the ReadMe file, it is mentioned to setup the datasets dir as follows:

    └── cuhkpedes
    β”œβ”€β”€ annotations
    β”‚   β”œβ”€β”€ test.json
    β”‚   β”œβ”€β”€ train.json
    β”‚   └── val.json
    β”œβ”€β”€ clip_vocab_vit.npy
    └── imgs
        β”œβ”€β”€ cam_a
        β”œβ”€β”€ cam_b
        β”œβ”€β”€ CUHK01
        β”œβ”€β”€ CUHK03
        β”œβ”€β”€ Market
        β”œβ”€β”€ test_query
        └── train_query

    After downloading the cuhkpedes data set, we get only the imgs folder, containing cam_a, cam_b and CUHK01 folders. there is no test_query and train_query folders. Also, these folders are not in the repository. Could you provide more information regarding on these folders, more exactly, what kind of information they contain and how they must be set up?

    Also, there are few more folders that are not part of the cuhkpedes, such as CUHK03 and Market. Do we need these data sets to reproduce the results?

    Best regards, liviust

    opened by liviust 5
  • some problem in training and testing

    some problem in training and testing

    Hello

    I have some problem. first: I don't find test_query and train_query file when I get images from [Dr. Shuang Li] second: I have this problem for testing and training.

    image

    opened by ram7772 4
  • Problem about the clip_vocab_vit.npy

    Problem about the clip_vocab_vit.npy

    Hi :) I have a question about the pre-processing document clip_vocab_vit.npy. My understanding is that it contains the tensor of the CLIP-Text-Encoder output corresponding to each word (total 9408). My question is, the output dimension of CLIP-TEXT-ENCODER is 1024, but the tensor dimension of each word in clip_vocab_vit.npy is 512. Is there some other operation in it? Thanks

    opened by Frost-Yang-99 2
  • There is only caption_all.json in the dataset CUHK-PEDES, what are the train.json and test.json in the dataset part

    There is only caption_all.json in the dataset CUHK-PEDES, what are the train.json and test.json in the dataset part

    Describe the bug A clear and concise description of what the bug is.

    To Reproduce Steps to reproduce the behavior:

    1. Go to '...'
    2. Click on '....'
    3. Scroll down to '....'
    4. See error

    Expected behavior A clear and concise description of what you expected to happen.

    Screenshots If applicable, add screenshots to help explain your problem.

    Desktop (please complete the following information):

    • OS: [e.g. iOS]
    • Browser [e.g. chrome, safari]
    • Version [e.g. 22]

    Smartphone (please complete the following information):

    • Device: [e.g. iPhone6]
    • OS: [e.g. iOS8.1]
    • Browser [e.g. stock browser, safari]
    • Version [e.g. 22]

    Additional context Add any other context about the problem here.

    opened by SwimKY 1
Releases(v0.1.1)
Owner
Xiao Han
Ph.D. student @ UoSurrey CVSSP, B.Eng. @ ZJU ISEE
Xiao Han
GitHub Repository
Visualizer using audio and semantic analysis to explore BigGAN (Brock et al., 2018) latent space.

BigGAN Audio Visualizer Description This visualizer explores BigGAN (Brock et al., 2018) latent space by using pitch/tempo of an audio file to generat

Rush Kapoor 2 Nov 21, 2022
object recognition with machine learning on Respberry pi

Respberrypi_object-recognition object recognition with machine learning on Respberry pi line.py ε»Ίη«‹δΈ€ζ”―θˆ‡ζ¨Ήζ’…ζ΄Ύι€£η·šηš„ linebot 使用歀 linebot ι η«―ζŽ§εˆΆζ¨Ήζ’…ζ΄Ύζ‹η…§ config.ini l

1 Dec 11, 2021
Self-Supervised Image Denoising via Iterative Data Refinement

Self-Supervised Image Denoising via Iterative Data Refinement Yi Zhang1, Dasong Li1, Ka Lung Law2, Xiaogang Wang1, Hongwei Qin2, Hongsheng Li1 1CUHK-S

Zhang Yi 72 Jan 01, 2023
Unified file system operation experience for different backend

megfile - Megvii FILE library Docs: http://megvii-research.github.io/megfile megfile provides a silky operation experience with different backends (cu

MEGVII Research 76 Dec 14, 2022
Official Python implementation of the FuzionCoin protocol

PyFuzc Official Python implementation of the FuzionCoin protocol WARNING: Under construction. Use at your own risk. Some functions may not work. Setup

FuzionCoin 3 Jul 07, 2022
Efficient Speech Processing Tookit for Automatic Speaker Recognition

Sugar Efficient Speech Processing Tookit for Automatic Speaker Recognition | HuggingFace | What's New EfficientTDNN: Efficient Architecture Search for

WangRui 14 Sep 14, 2022
JstDoS - HTTP Protocol Stack Remote Code Execution Vulnerability

jstDoS If you are going to skid that, please give credits ! ^^ ΒΏHow works? This

apolo 4 Feb 11, 2022
A PoC Corporation Relationship Knowledge Graph System on top of Nebula Graph.

Corp-Rel is a PoC of Corpartion Relationship Knowledge Graph System. It's built on top of the Open Source Graph Database: Nebula Graph with a dataset

Wey Gu 20 Dec 11, 2022
DeOldify - A Deep Learning based project for colorizing and restoring old images (and video!)

DeOldify - A Deep Learning based project for colorizing and restoring old images (and video!)

Jason Antic 15.8k Jan 04, 2023
[CVPR 2021] Involution: Inverting the Inherence of Convolution for Visual Recognition, a brand new neural operator

involution Official implementation of a neural operator as described in Involution: Inverting the Inherence of Convolution for Visual Recognition (CVP

Duo Li 1.3k Dec 28, 2022
An implementation of EWC with PyTorch

EWC.pytorch An implementation of Elastic Weight Consolidation (EWC), proposed in James Kirkpatrick et al. Overcoming catastrophic forgetting in neural

Ryuichiro Hataya 166 Dec 22, 2022
A demonstration of using a live Tensorflow session to create an interactive face-GAN explorer.

Streamlit Demo: The Controllable GAN Face Generator This project highlights Streamlit's new hash_func feature with an app that calls on TensorFlow to

Streamlit 257 Dec 31, 2022
For auto aligning, cropping, and scaling HR and LR images for training image based neural networks

ImgAlign For auto aligning, cropping, and scaling HR and LR images for training image based neural networks Usage Make sure OpenCV is installed, 'pip

15 Dec 04, 2022
The implementation for paper Joint t-SNE for Comparable Projections of Multiple High-Dimensional Datasets.

Joint t-sne This is the implementation for paper Joint t-SNE for Comparable Projections of Multiple High-Dimensional Datasets. abstract: We present Jo

IDEAS Lab 7 Dec 18, 2022
Official implementation of "Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform", ICCV 2021

Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform This repository is the implementation of "Variable-Rate Deep Image C

Myungseo Song 47 Dec 13, 2022
The official code for paper "R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling".

R2D2 This is the official code for paper titled "R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Mode

Alipay 49 Dec 17, 2022
Feed forward VQGAN-CLIP model, where the goal is to eliminate the need for optimizing the latent space of VQGAN for each input prompt

Feed forward VQGAN-CLIP model, where the goal is to eliminate the need for optimizing the latent space of VQGAN for each input prompt. This is done by

Mehdi Cherti 135 Dec 30, 2022
Making Structure-from-Motion (COLMAP) more robust to symmetries and duplicated structures

SfM disambiguation with COLMAP About Structure-from-Motion generally fails when the scene exhibits symmetries and duplicated structures. In this repos

Computer Vision and Geometry Lab 193 Dec 26, 2022
[ICCV21] Official implementation of the "Social NCE: Contrastive Learning of Socially-aware Motion Representations" in PyTorch.

Social-NCE + CrowdNav Website | Paper | Video | Social NCE + Trajectron | Social NCE + STGCNN This is an official implementation for Social NCE: Contr

VITA lab at EPFL 125 Dec 23, 2022
TorchGeo is a PyTorch domain library, similar to torchvision, that provides datasets, transforms, samplers, and pre-trained models specific to geospatial data.

TorchGeo is a PyTorch domain library, similar to torchvision, that provides datasets, transforms, samplers, and pre-trained models specific to geospatial data.

Microsoft 1.3k Dec 30, 2022