Answer a series of contextually-dependent questions like they may occur in natural human-to-human conversations.

Overview

SCAI-QReCC-21

[leaderboards] [registration] [forum] [contact] [SCAI]

Answer a series of contextually-dependent questions like they may occur in natural human-to-human conversations.

  • Submission deadline: September 8, 2021 Extended: September 15, 2021
  • Results announcement: September 30, 2021
  • Workshop presentations: October 8, 2021

Data

[Zenodo] [original]

File names here refer to the respective files hosted on [Zenodo].

The passage collection (passages.zip) is 27.5GB with 54M passages!

The input format for the task (scai-qrecc21-[toy,training,test]-questions[,-rewritten].json) is a JSON file:

, "Turn_no": X, "Question": " " }, ... ]">
[
  {
    "Conversation_no": 
    
     ,
    "Turn_no": X,
    "Question": "
     
      "
  }, ...
]

     
    

With X being the number of the question in the conversation. Questions with the same Conversation_no are from the same conversation.

The questions-rewritten.json-files contain human rewritten questions that can be used by systems that do not want to participate in question rewriting.

Submission

Register for the task using this form. We will then send you your TIRA login once it is ready.

The challenge is hosted on TIRA. Participants are encouraged to upload their code and run the evaluation on the VMs provided by the platform to ensure reproducibility of the results. It is also possible to upload the submission as a single JSON file.

The submission format for the task is a JSON file similar to the input (all Model_xxx-fields are optional and you can omit them from the submission, e.g. provide only Conversation_no, Turn_no and Model_answer to get the EM and F1 scores for the generated answers):

, "Turn_no": X, "Model_rewrite": " ", "Model_passages": { " ": , ... }, "Model_answer": " " }, ... ]">
[
  {
    "Conversation_no": 
       
        ,
    "Turn_no": X,
    "Model_rewrite": "
        
         ",
    "Model_passages": { 
      "
         
          ": 
          
           , ...
    },
    "Model_answer": "
           
            " }, ... ] 
           
          
         
        
       

Example: scai-qrecc21-naacl-baseline.zip

You can use the code of our simple baseline to get started.

Software Submission

We recommend participants to upload (through SSH or RDP) their software/system to their dedicated TIRA virtual machine (assigned after registration), so that their runs can be reproduced and so that they can be easily applied to different data (of same format) in the future. The mail send to you after registration gives you the credentials to access the TIRA web interface and your VM. If you cannot connect to your VM, ensure it is powered on in the TIRA web interface.

Your software is expected to accept two arguments:

  • An input directory (named $inputDataset in TIRA) that contains the questions.json input file and passages-index-anserini directory. The latter contains a full Anserini index of the passage collection. Note that you need to install openjdk-11-jdk-headless to use it. We may be able to add more of such indices on request.
  • An output directory (named $outputDir in TIRA) into which your software needs to place the submission as run.json.

Install your software to your VM. Then go to the TIRA web interface and click "Add software". Specify the command to run your software (see the image for the simple baseline).

IMPORTANT: To ensure reproducibility, create a "Software" in the TIRA web interface for each parameter setting that you consider a submission to the challenge.

Click on "Run" to execute your software for the selected input dataset. Your VM will not be accessible while your system is running, be detached from the internet (to ensure your software is fully installed in your virtual machine), and afterwards restored to the state before the run. Since the test set is rather large (the simple baseline takes nearly 11 hours to complete), we highly recommend you first test your software on the scai-qrecc21-toy-dataset-2021-07-20 input dataset. This dataset contains the first conversation (6 turns/questions) only. For the test-dataset, send us a mail at [email protected] so that we unblind your results.

TIRA Interface: VM status and submission

Then go to the "Runs" section below and click on the blue (i)-icon of the software run to check the software output. You can also download the run from there.

NOTE: By submitting your software you retain full copyrights. You agree to grant us usage rights for evaluation of the corresponding data generated by your software. We agree not to share your software with a third party or use it for any purpose other than research.

Run Submission

You can upload a JSON file as a submission at https://www.tira.io/run-upload-scai-qrecc21.

TIRA Interface: VM status and submission

Please specify the name and a description of your run in the form. After a successful upload, the page will redirect you to the overview of all your submissions where you should evaluate your run to verify that your run is valid. At the "Runs" section, you can click on the blue (i)-icon to double-check your upload. You can also download the run from there.

Evaluation

[script]

Once you run your software or uploaded your run, "Run" the evaluator on that run through the TIRA web interface (below the software; works out-of-the-box).

TIRA Interface: Evaluation

Then go to the "Runs" section below and click on the blue (i)-icon of the evaluator run to see your scores.

Ground truth

We use the QReCC paper annotations in the initial phase, and will update them with alternative answer spans and passages by pooling and crowdsourcing the relevance judgements over the results submitted by the challenge participants (similar to the TREC evaluation setup).

Metrics

We use the same metrics as the QReCC paper, but may add more for the final evaluation: ROUGE1-R for question rewriting, Mean Reciprocal Rank (MRR) for passage retrieval, and F1 and Exact Match for question answering.

Baselines

We provide the following baselines for comparison:

  • scai-qrecc21-simple-baseline: BM25 baseline for passage retrieval using original conversational questions without rewriting. We recommend to use this code as a boilerplate to kickstart your own submission using the VM.
  • scai-qrecc21-naacl-baseline: results for the end-to-end approach using supervised question rewriting and QA models reported in the QReCC paper (accepted at NAACL'21). This sample run is available on Zenodo as scai-qrecc21-naacl-baseline.zip.

Note that the baseline results differ from the ones reported in the paper since we made several corrections to the evaluation script and the ground truth annotations:

  • We excluded the samples for which the ground truth is missing from the evaluation (i.e., no relevant passages or no answer text or no rewrite provided by the human annotators)

  • We removed 5,251 passages judgements annotated by the heuristic as relevant for the short answers with lengths <= 5 since these matches are often trivial and unrelated, e.g., the same noun phrase appearing in different contexts.

Resources

Some useful links to get you started on a new conversational open-domain QA system:

Conversational Passage Retrieval

Answer Generation

Passage Retrieval

Conversational Question Reformulation

yufan 81 Dec 08, 2022
PyTorch implementation of MSBG hearing loss model and MBSTOI intelligibility metric

PyTorch implementation of MSBG hearing loss model and MBSTOI intelligibility metric This repository contains the implementation of MSBG hearing loss m

BUT <a href=[email protected]"> 9 Nov 08, 2022
How to Predict Stock Prices Easily Demo

How-to-Predict-Stock-Prices-Easily-Demo How to Predict Stock Prices Easily - Intro to Deep Learning #7 by Siraj Raval on Youtube ##Overview This is th

Siraj Raval 752 Nov 16, 2022
This solves the autonomous driving issue which is supported by deep learning technology. Given a video, it splits into images and predicts the angle of turning for each frame.

Self Driving Car An autonomous car (also known as a driverless car, self-driving car, and robotic car) is a vehicle that is capable of sensing its env

Sagor Saha 4 Sep 04, 2021
Python scripts for performing object detection with the 1000 labels of the ImageNet dataset in ONNX.

Python scripts for performing object detection with the 1000 labels of the ImageNet dataset in ONNX. The repository combines a class agnostic object localizer to first detect the objects in the image

Ibai Gorordo 24 Nov 14, 2022
Language models are open knowledge graphs ( non official implementation )

language-models-are-knowledge-graphs-pytorch Language models are open knowledge graphs ( work in progress ) A non official reimplementation of Languag

theblackcat102 132 Dec 18, 2022
PyTorch implementation of U-TAE and PaPs for satellite image time series panoptic segmentation.

Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks (ICCV 2021) This repository is the official implem

71 Jan 04, 2023
Episodic-memory - Ego4D Episodic Memory Benchmark

Ego4D Episodic Memory Benchmark EGO4D is the world's largest egocentric (first p

3 Feb 18, 2022
TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation

TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentation Zhaoyun Yin, Pichao Wang, Fan Wang, Xianzhe Xu, Hanling Zhang, Hao Li

DamoCV 25 Dec 16, 2022
SSD-based Object Detection in PyTorch

SSD-based Object Detection in PyTorch 서강대학교 현대모비스 SW 프로그램에서 진행한 인공지능 프로젝트입니다. Jetson nano를 이용해 pre-trained network를 fine tuning시켜 차량 및 신호등 인식을 구현하였습니다

Haneul Kim 1 Nov 16, 2021
Trax — Deep Learning with Clear Code and Speed

Trax — Deep Learning with Clear Code and Speed Trax is an end-to-end library for deep learning that focuses on clear code and speed. It is actively us

Google 7.3k Dec 26, 2022
Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

Hehe Fan 63 Dec 09, 2022
Pytorch implementation of Zero-DCE++

Zero-DCE++ You can find more details here: https://li-chongyi.github.io/Proj_Zero-DCE++.html. You can find the details of our CVPR version: https://li

Chongyi Li 157 Dec 23, 2022
Using this codebase as a tool for my own research. Making some modifications to the original repo for my own purposes.

For SwapNet Create a list.txt file containing all the images to process. This can be done with the GNU find command: find path/to/input/folder -name '

Andrew Jong 2 Nov 10, 2021
A TensorFlow implementation of FCN-8s

FCN-8s implementation in TensorFlow Contents Overview Examples and demo video Dependencies How to use it Download pre-trained VGG-16 Overview This is

Pierluigi Ferrari 50 Aug 08, 2022
TensorFlow for Raspberry Pi

TensorFlow on Raspberry Pi It's officially supported! As of TensorFlow 1.9, Python wheels for TensorFlow are being officially supported. As such, this

Sam Abrahams 2.2k Dec 16, 2022
This repository contains the code for "Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP".

Self-Diagnosis and Self-Debiasing This repository contains the source code for Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based

Timo Schick 62 Dec 12, 2022
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement (NeurIPS 2020)

MTTS-CAN: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement Paper Xin Liu, Josh Fromm, Shwetak Patel, Daniel M

Xin Liu 106 Dec 30, 2022
Changing the Mind of Transformers for Topically-Controllable Language Generation

We will first introduce the how to run the IPython notebook demo by downloading our pretrained models. Then, we will introduce how to run our training and evaluation code.

IESL 20 Dec 06, 2022
A Pytorch implementation of "Manifold Matching via Deep Metric Learning for Generative Modeling" (ICCV 2021)

Manifold Matching via Deep Metric Learning for Generative Modeling A Pytorch implementation of "Manifold Matching via Deep Metric Learning for Generat

69 Dec 10, 2022