D-VQA
We provide the PyTorch implementation for Debiased Visual Question Answering from Feature and Sample Perspectives (NeurIPS 2021).
Dependencies
- Python 3.6
- PyTorch 1.1.0
- dependencies in requirements.txt
- We train and evaluate all of the models based on one TITAN Xp GPU
Getting Started
Installation
-
Clone this repository:
git clone https://github.com/Zhiquan-Wen/D-VQA.git cd D-VQA -
Install PyTorch and other dependencies:
pip install -r requirements.txt
Download and preprocess the data
cd data
bash download.sh
python preprocess_features.py --input_tsv_folder xxx.tsv --output_h5 xxx.h5
python feature_preprocess.py --input_h5 xxx.h5 --output_path trainval
python create_dictionary.py --dataroot vqacp2/
python preprocess_text.py --dataroot vqacp2/ --version v2
cd ..
Training
- Train our model
CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7
- Train the model with 80% of the original training set
CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7 --ratio 0.8
Evaluation
- A json file of results from the test set can be produced with:
CUDA_VISIBLE_DEVICES=0 python test.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --checkpoint_path saved_models_cp2/best_model.pth --output saved_models_cp2/result/
- Compute detailed accuracy for each answer type:
python comput_score.py --input saved_models_cp2/result/XX.json --dataroot data/vqacp2/
Pretrained model
A well-trained model can be found here. The test results file produced by it can be found here and its performance is as follows:
Overall score: 61.91
Yes/No: 88.93 Num: 52.32 other: 50.39
Reference
If you found this code is useful, please cite the following paper:
@inproceedings{D-VQA,
title = {Debiased Visual Question Answering from Feature and Sample Perspectives},
author = {Zhiquan Wen,
Guanghui Xu,
Mingkui Tan,
Qingyao Wu,
Qi Wu},
booktitle = {NeurIPS},
year = {2021}
}
Acknowledgements
This repository contains code modified from SSL-VQA, thank you very much!
Besides, we thank Yaofo Chen for providing MIO library to accelerate the data loading.
下面是我的代码
def plot_rect(image, boxes):
img = Image.fromarray(np.uint8(image))
draw = ImageDraw.Draw(img)
for k in range(2):
box = boxes[k,:]
print(box)
drawrect(draw, box, outline='green', width=3)
img = np.asarray(img)
return img
def drawrect(drawcontext, xy, outline=None, width=0):
x1, y1, x2, y2 = xy
points = (x1, y1), (x2, y1), (x2, y2), (x1, y2), (x1, y1)
drawcontext.line(points, fill=outline, width=width)
44 Oct 26, 2022
139 Nov 30, 2022
29 Sep 02, 2022
68 Dec 07, 2022
2.5k Jan 04, 2023
29 Nov 28, 2022
1 Dec 14, 2022
5 Oct 20, 2022
9 Nov 24, 2022
2 Feb 18, 2022
23 Dec 09, 2022
51 Sep 15, 2022
251 Dec 30, 2022
30 Jan 01, 2023
44 Dec 21, 2022
22 Jul 20, 2022
39 Oct 12, 2022
919 Dec 25, 2022
2 Dec 21, 2021
18 Oct 20, 2022