Deep Learning for Natural Language Processing SS 2021 (TU Darmstadt)

Overview

Deep Learning for Natural Language Processing SS 2021 (TU Darmstadt)

Task

Training huge unsupervised deep neural networks yields to strong progress in the field of Natural Language Processing (NLP). Using these extensively pre-trained networks for particular NLP applications is the current state-of-the-art approach. In this project, we approach the task of ranking possible clarifying questions for a given query. We fine-tuned a pre-trained BERT model to rank the possible clarifying questions in a classification manner. The achieved model scores a top-5 accuracy of 0.4565 on the provided benchmark dataset.

Installation

This project was originally developed with Python 3.8, PyTorch 1.7, and CUDA 11.0. The training requires one NVIDIA GeForce RTX 1080 (11GB memory).

  • Create conda environment:
conda create --name dl4nlp
source activate dl4nlp
  • Install the dependencies:
pip install -r requirements.txt

Run

We use a pretrained BERT-Base by Hugging Face and fine-tune it on the given training dataset. To run training, please use the following command:

python main.py --train

For evaluation on the test set, please use the following command:

python main.py --test

Arguments for training and/or testing:

  • --train: Run training on training dataset. Default: True
  • --val: Run evaluation during training on validation dataset. Default: True
  • --test: Run evaluation on test dataset. Default: True
  • --cuda-devices: Set GPU index Default: 0
  • --cpu: Run everything on CPU. Default: False
  • --data-parallel: Use DataParallel. Default: False
  • --data-root: Path to dataset folder. Default: data
  • --train-file-name: Name of training file name in data-root. Default: training.tsv
  • --test-file-name: Name of test file name in data-root. Default: test_set.tsv
  • --question-bank-name: Name of question bank file name in data-root. Default: question_bank.tsv
  • --checkpoints-root: Path to checkpoints folder. Default: checkpoints
  • --checkpoint-name: File name of checkpoint in checkpoints-root to start training or use for testing. Default: None
  • --runs-root: Path to output runs folder for tensorboard. Default: runs
  • --txt-root: Path to output txt folder for evaluation results. Default: txt
  • --lr: Learning rate. Default: 1e-5
  • --betas: Betas for optimization. Default: (0.9, 0.999)
  • --weight-decay: Weight decay. Default: 1e-2
  • --val-start: Set at which epoch to start validation. Default: 0
  • --val-step: Set at which epoch rate to valide. Default: 1
  • --val-split: Use subset of training dataset for validation. Default: 0.005
  • --num-epochs: Number of epochs for training. Default: 10
  • --batch-size: Samples per batch. Default: 32
  • --num-workers: Number of workers. Default: 4
  • --top-k-accuracy: Evaluation metric with flexible top-k-accuracy. Default: 50
  • --true-label: True label in dataset. Default: 1
  • --false-label: False label in dataset. Default: 0

Example output

User query:

Tell me about Computers

Propagated clarifying questions:

  1. do you like using computers
  2. do you want to know how to do computer programming
  3. do you want to see some closeup of a turbine
  4. are you looking for information on different computer programming languages
  5. are you referring to a software
Owner
Oliver Hahn
Master Thesis @ Visual Inference Lab | Grad Student @ Technical University of Darmstadt
Oliver Hahn
A numpy-based implementation of RANSAC for fundamental matrix and homography estimation. The degeneracy updating and local optimization components are included and optional.

Description A numpy-based implementation of RANSAC for fundamental matrix and homography estimation. The degeneracy updating and local optimization co

AoxiangFan 9 Nov 10, 2022
STBP is a way to train SNN with datasets by Backward propagation.

Spiking neural network (SNN), compared with depth neural network (DNN), has faster processing speed, lower energy consumption and more biological interpretability, which is expected to approach Stron

Ling Zhang 18 Dec 09, 2022
UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning This is the official PyTorch implementation for UniMoCo pape

dddzg 49 Jan 02, 2023
Deep Inertial Prediction (DIPr)

Deep Inertial Prediction For more information and context related to this repo, please refer to our website. Getting Started (non Docker) Note: you wi

Arcturus Industries 12 Nov 11, 2022
DiffWave is a fast, high-quality neural vocoder and waveform synthesizer.

DiffWave DiffWave is a fast, high-quality neural vocoder and waveform synthesizer. It starts with Gaussian noise and converts it into speech via itera

LMNT 498 Jan 03, 2023
Topic Modelling for Humans

gensim – Topic Modelling in Python Gensim is a Python library for topic modelling, document indexing and similarity retrieval with large corpora. Targ

RARE Technologies 13.8k Jan 03, 2023
TAPEX: Table Pre-training via Learning a Neural SQL Executor

TAPEX: Table Pre-training via Learning a Neural SQL Executor The official repository which contains the code and pre-trained models for our paper TAPE

Microsoft 157 Dec 28, 2022
Editing a Conditional Radiance Field

Editing Conditional Radiance Fields Project | Paper | Video | Demo Editing Conditional Radiance Fields Steven Liu, Xiuming Zhang, Zhoutong Zhang, Rich

Steven Liu 216 Dec 30, 2022
JFB: Jacobian-Free Backpropagation for Implicit Models

JFB: Jacobian-Free Backpropagation for Implicit Models

Typal Research 28 Dec 11, 2022
Replication Package for "An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets"

Replication Package for "An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Data

2 Oct 06, 2022
The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color Overview Code and dataset for The World of an Octopus: H

1 Nov 13, 2021
Python program that works as a contact list

Lista de Contatos Programa em Python que funciona como uma lista de contatos. Features Adicionar novo contato Remover contato Atualizar contato Pesqui

Victor B. Lino 3 Dec 16, 2021
Repository for the paper titled: "When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer"

When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer This repository contains code for our paper titled "When is BERT M

Princeton Natural Language Processing 9 Dec 23, 2022
paper list in the area of reinforcenment learning for recommendation systems

paper list in the area of reinforcenment learning for recommendation systems

HenryZhao 23 Jun 09, 2022
BasicNeuralNetwork - This project looks over the basic structure of a neural network and how machine learning training algorithms work

BasicNeuralNetwork - This project looks over the basic structure of a neural network and how machine learning training algorithms work. For this project, I used the sigmoid function as an activation

Manas Bommakanti 1 Jan 22, 2022
A modular, open and non-proprietary toolkit for core robotic functionalities by harnessing deep learning

A modular, open and non-proprietary toolkit for core robotic functionalities by harnessing deep learning Website • About • Installation • Using OpenDR

OpenDR 304 Dec 28, 2022
Pytorch implementation for "Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion" (NeurIPS 2021)

Density-aware Chamfer Distance This repository contains the official PyTorch implementation of our paper: Density-aware Chamfer Distance as a Comprehe

Tong WU 93 Dec 15, 2022
KITTI-360 Annotation Tool is a framework that developed based on python(cherrypy + jinja2 + sqlite3) as the server end and javascript + WebGL as the front end.

KITTI-360 Annotation Tool is a framework that developed based on python(cherrypy + jinja2 + sqlite3) as the server end and javascript + WebGL as the front end.

86 Dec 12, 2022
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation This project attempted to implement the paper Putting NeRF on a

254 Dec 27, 2022
Pytorch implementation of “Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement”

Graph-to-Graph Transformers Self-attention models, such as Transformer, have been hugely successful in a wide range of natural language processing (NL

Idiap Research Institute 40 Aug 14, 2022