KE-Dialogue: Injecting knowledge graph into a fully end-to-end dialogue system.

Overview

Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems

License: MIT

This is the implementation of the paper:

Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems. Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata, Yan Xu, Zihan Liu, Zhaojiang Lin, Pascale Fung Findings of EMNLP 2020 [PDF]

If you use any source codes or datasets included in this toolkit in your work, please cite the following paper. The bibtex is listed below:

@article{madotto2020learning,
  title={Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems},
  author={Madotto, Andrea and Cahyawijaya, Samuel and Winata, Genta Indra and Xu, Yan and Liu, Zihan and Lin, Zhaojiang and Fung, Pascale},
  journal={arXiv preprint arXiv:2009.13656},
  year={2020}
}

Abstract

Task-oriented dialogue systems are either modularized with separate dialogue state tracking (DST) and management steps or end-to-end trainable. In either case, the knowledge base (KB) plays an essential role in fulfilling user requests. Modularized systems rely on DST to interact with the KB, which is expensive in terms of annotation and inference time. End-to-end systems use the KB directly as input, but they cannot scale when the KB is larger than a few hundred entries. In this paper, we propose a method to embed the KB, of any size, directly into the model parameters. The resulting model does not require any DST or template responses, nor the KB as input, and it can dynamically update its KB via finetuning. We evaluate our solution in five taskoriented dialogue datasets with small, medium, and large KB size. Our experiments show that end-to-end models can effectively embed knowledge bases in their parameters and achieve competitive performance in all evaluated datasets.

Knowledge-embedded Dialogue:

During training, the KE dialogues are generated by fulfilling the *TEMPLATE* with the *user goal query* results, and they are used to embed the KB into the model parameter theta. At testing time, the model does not use any external knowledge to generate the correct responses.

Dependencies

We listed our dependencies on requirements.txt, you can install the dependencies by running

❱❱❱ pip install -r requirements.txt

In addition, our code also includes fp16 support with apex. You can find the package from https://github.com/NVIDIA/apex.

Experiments

bAbI-5

Dataset Download the preprocessed dataset and put the zip file inside the ./knowledge_embed/babi5 folder. Extract the zip file by executing

❱❱❱ cd ./knowledge_embed/babi5
❱❱❱ unzip dialog-bAbI-tasks.zip

Generate the delexicalized dialogues from bAbI-5 dataset via

❱❱❱ python3 generate_delexicalization_babi.py

Generate the lexicalized data from bAbI-5 dataset via

❱❱❱ python generate_dialogues_babi5.py --dialogue_path ./dialog-bAbI-tasks/dialog-babi-task5trn_record-delex.txt --knowledge_path ./dialog-bAbI-tasks/dialog-babi-kb-all.txt --output_folder ./dialog-bAbI-tasks --num_augmented_knowledge <num_augmented_knowledge> --num_augmented_dialogue <num_augmented_dialogues> --random_seed 0

Where the maximum <num_augmented_knowledge> is 558 (recommended) and <num_augmented_dialogues> is 264 as it is corresponds to the number of knowledge and number of dialogues in bAbI-5 dataset.

Fine-tune GPT-2

We provide the checkpoint of GPT-2 model fine-tuned on bAbI training set. You can also choose to train the model by yourself using the following command.

❱❱❱ cd ./modeling/babi5
❱❱❱ python main.py --model_checkpoint gpt2 --dataset BABI --dataset_path ../../knowledge_embed/babi5/dialog-bAbI-tasks --n_epochs <num_epoch> --kbpercentage <num_augmented_dialogues>

Notes that the value of --kbpercentage is equal to <num_augmented_dialogues> the one that comes from the lexicalization. This parameter is used for selecting the augmentation file to embed into the train dataset.

You can evaluate the model by executing the following script

❱❱❱ python evaluate.py --model_checkpoint <model_checkpoint_folder> --dataset BABI --dataset_path ../../knowledge_embed/babi5/dialog-bAbI-tasks

Scoring bAbI-5 To run the scorer for bAbI-5 task model, you can run the following command. Scorer will read all of the result.json under runs folder generated from evaluate.py

python scorer_BABI5.py --model_checkpoint <model_checkpoint> --dataset BABI --dataset_path ../../knowledge_embed/babi5/dialog-bAbI-tasks --kbpercentage 0

CamRest

Dataset

Download the preprocessed dataset and put the zip file under ./knowledge_embed/camrest folder. Unzip the zip file by executing

❱❱❱ cd ./knowledge_embed/camrest
❱❱❱ unzip CamRest.zip

Generate the delexicalized dialogues from CamRest dataset via

❱❱❱ python3 generate_delexicalization_CAMREST.py

Generate the lexicalized data from CamRest dataset via

❱❱❱ python generate_dialogues_CAMREST.py --dialogue_path ./CamRest/train_record-delex.txt --knowledge_path ./CamRest/KB.json --output_folder ./CamRest --num_augmented_knowledge <num_augmented_knowledge> --num_augmented_dialogue <num_augmented_dialogues> --random_seed 0

Where the maximum <num_augmented_knowledge> is 201 (recommended) and <num_augmented_dialogues> is 156 quite huge as it is corresponds to the number of knowledge and number of dialogues in CamRest dataset.

Fine-tune GPT-2

We provide the checkpoint of GPT-2 model fine-tuned on CamRest training set. You can also choose to train the model by yourself using the following command.

❱❱❱ cd ./modeling/camrest/
❱❱❱ python main.py --model_checkpoint gpt2 --dataset CAMREST --dataset_path ../../knowledge_embed/camrest/CamRest --n_epochs <num_epoch> --kbpercentage <num_augmented_dialogues>

Notes that the value of --kbpercentage is equal to <num_augmented_dialogues> the one that comes from the lexicalization. This parameter is used for selecting the augmentation file to embed into the train dataset.

You can evaluate the model by executing the following script

❱❱❱ python evaluate.py --model_checkpoint <model_checkpoint_folder> --dataset CAMREST --dataset_path ../../knowledge_embed/camrest/CamRest

Scoring CamRest To run the scorer for bAbI 5 task model, you can run the following command. Scorer will read all of the result.json under runs folder generated from evaluate.py

python scorer_CAMREST.py --model_checkpoint <model_checkpoint> --dataset CAMREST --dataset_path ../../knowledge_embed/camrest/CamRest --kbpercentage 0

SMD

Dataset

Download the preprocessed dataset and put it under ./knowledge_embed/smd folder.

❱❱❱ cd ./knowledge_embed/smd
❱❱❱ unzip SMD.zip

Fine-tune GPT-2

We provide the checkpoint of GPT-2 model fine-tuned on SMD training set. Download the checkpoint and put it under ./modeling folder.

❱❱❱ cd ./knowledge_embed/smd
❱❱❱ mkdir ./runs
❱❱❱ unzip ./knowledge_embed/smd/SMD_gpt2_graph_False_adj_False_edge_False_unilm_False_flattenKB_False_historyL_1000000000_lr_6.25e-05_epoch_10_weighttie_False_kbpercentage_0_layer_12.zip -d ./runs

You can also choose to train the model by yourself using the following command.

❱❱❱ cd ./modeling/smd
❱❱❱ python main.py --dataset SMD --lr 6.25e-05 --n_epochs 10 --kbpercentage 0 --layers 12

Prepare Knowledge-embedded dialogues

Firstly, we need to build databases for SQL query.

❱❱❱ cd ./knowledge_embed/smd
❱❱❱ python generate_dialogues_SMD.py --build_db --split test

Then we generate dialogues based on pre-designed templates by domains. The following command enables you to generate dialogues in weather domain. Please replace weather with navigate or schedule in dialogue_path and domain arguments if you want to generate dialogues in the other two domains. You can also change number of templates used in relexicalization process by changing the argument num_augmented_dialogue.

❱❱❱ python generate_dialogues_SMD.py --split test --dialogue_path ./templates/weather_template.txt --domain weather --num_augmented_dialogue 100 --output_folder ./SMD/test

Adapt fine-tuned GPT-2 model to the test set

❱❱❱ python evaluate_finetune.py --dataset SMD --model_checkpoint runs/SMD_gpt2_graph_False_adj_False_edge_False_unilm_False_flattenKB_False_historyL_1000000000_lr_6.25e-05_epoch_10_weighttie_False_kbpercentage_0_layer_12 --top_k 1 --eval_indices 0,303 --filter_domain ""

You can also speed up the finetuning process by running experiments parallelly. Please modify the GPU setting in #L14 of the code.

❱❱❱ python runner_expe_SMD.py 

MWOZ (2.1)

Dataset

Download the preprocessed dataset and put it under ./knowledge_embed/mwoz folder.

❱❱❱ cd ./knowledge_embed/mwoz
❱❱❱ unzip mwoz.zip

Prepare Knowledge-Embedded dialogues (You can skip this step, if you have downloaded the zip file above)

You can prepare the datasets by running

❱❱❱ bash generate_MWOZ_all_data.sh

The shell script generates the delexicalized dialogues from MWOZ dataset by calling

❱❱❱ python generate_delex_MWOZ_ATTRACTION.py
❱❱❱ python generate_delex_MWOZ_HOTEL.py
❱❱❱ python generate_delex_MWOZ_RESTAURANT.py
❱❱❱ python generate_delex_MWOZ_TRAIN.py
❱❱❱ python generate_redelex_augmented_MWOZ.py
❱❱❱ python generate_MWOZ_dataset.py

Fine-tune GPT-2

We provide the checkpoint of GPT-2 model fine-tuned on MWOZ training set. Download the checkpoint and put it under ./modeling folder.

❱❱❱ cd ./knowledge_embed/mwoz
❱❱❱ mkdir ./runs
❱❱❱ unzip ./mwoz.zip -d ./runs

You can also choose to train the model by yourself using the following command.

❱❱❱ cd ./modeling/mwoz
❱❱❱ python main.py --model_checkpoint gpt2 --dataset MWOZ_SINGLE --max_history 50 --train_batch_size 6 --kbpercentage 100 --fp16 O2 --gradient_accumulation_steps 3 --balance_sampler --n_epochs 10

OpenDialKG

Getting Started We use neo4j community server edition and apoc library for processing graph data. apoc is used to parallelize the query in neo4j, so that we can process large scale graph faster

Before proceed to the dataset section, you need to ensure that you have neo4j (https://neo4j.com/download-center/#community) and apoc (https://neo4j.com/developer/neo4j-apoc/) installed on your system.

If you are not familiar with CYPHER and apoc syntaxes, you can follow the tutorial in https://neo4j.com/developer/cypher/ and https://neo4j.com/blog/intro-user-defined-procedures-apoc/

Dataset Download the original dataset and put the zip file inside the ./knowledge_embed/opendialkg folder. Extract the zip file by executing

❱❱❱ cd ./knowledge_embed/opendialkg
❱❱❱ unzip https://drive.google.com/file/d/1llH4-4-h39sALnkXmGR8R6090xotE0PE/view?usp=sharing.zip

Generate the delexicalized dialogues from opendialkg dataset via (WARNING: this requires around 12 hours to run)

❱❱❱ python3 generate_delexicalization_DIALKG.py

This script will produce ./opendialkg/dialogkg_train_meta.pt which will be use to generate the lexicalized dialogue. You can then generate the lexicalized dialogue from opendialkg dataset via

❱❱❱ python generate_dialogues_DIALKG.py --random_seed <random_seed> --batch_size 100 --max_iteration <max_iter> --stop_count <stop_count> --connection_string bolt://localhost:7687

This script will produce samples of dialogues at most batch_size * max_iter samples, but in every batch there is a possibility where there is no valid candidate and resulting in less samples. The number of generation is limited by another factor called stop_count which will stop the generation if the number of generated samples is more than equal the specified stop_count. The file will produce 4 files: ./opendialkg/db_count_records_{random_seed}.csv, ./opendialkg/used_count_records_{random_seed}.csv, and ./opendialkg/generation_iteration_{random_seed}.csv which are used for checking the distribution shift of the count in the DB; and ./opendialkg/generated_dialogue_bs100_rs{random_seed}.json which contains the generated samples.

Notes:

  • You might need to change the neo4j password inside generate_delexicalization_DIALKG.py and generate_dialogues_DIALKG.py manually.
  • Because there is a ton of possibility of connection in dialkg, we use sampling method to generate the data, so random seed is crucial if you want to have reproducible result

Fine-tune GPT-2

We provide the checkpoint of GPT-2 model fine-tuned on opendialkg training set. You can also choose to train the model by yourself using the following command.

❱❱❱ cd ./modeling/opendialkg
❱❱❱ python main.py --dataset_path ../../knowledge_embed/opendialkg/opendialkg --model_checkpoint gpt2 --dataset DIALKG --n_epochs 50 --kbpercentage <random_seed> --train_batch_size 8 --valid_batch_size 8

Notes that the value of --kbpercentage is equal to <random_seed> the one that comes from the lexicalization. This parameter is used for selecting the augmentation file to embed into the train dataset.

You can evaluate the model by executing the following script

❱❱❱ python evaluate.py  --model_checkpoint <model_checkpoint_folder> --dataset DIALKG --dataset_path  ../../knowledge_embed/opendialkg/opendialkg

Scoring OpenDialKG To run the scorer for bAbI-5 task model, you can run the following command. Scorer will read all of the result.json under runs folder generated from evaluate.py

python scorer_DIALKG5.py --model_checkpoint <model_checkpoint> --dataset DIALKG  ../../knowledge_embed/opendialkg/opendialkg --kbpercentage 0

Further Details

For the details regarding to the experiments, hyperparameters, and Evaluation results you can find it in the main paper of and suplementary materials of our work.

Owner
CAiRE
CAiRE
This code provides a PyTorch implementation for OTTER (Optimal Transport distillation for Efficient zero-shot Recognition), as described in the paper.

Data Efficient Language-Supervised Zero-Shot Recognition with Optimal Transport Distillation This repository contains PyTorch evaluation code, trainin

Meta Research 45 Dec 20, 2022
Official DGL implementation of "Rethinking High-order Graph Convolutional Networks"

SE Aggregation This is the implementation for Rethinking High-order Graph Convolutional Networks. Here we show the codes for citation networks as an e

Tianqi Zhang (张天启) 32 Jul 19, 2022
NeRF visualization library under construction

NeRF visualization library using PlenOctrees, under construction pip install nerfvis Docs will be at: https://nerfvis.readthedocs.org import nerfvis s

Alex Yu 196 Jan 04, 2023
[CVPRW 21] "BNN - BN = ? Training Binary Neural Networks without Batch Normalization", Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang

BNN - BN = ? Training Binary Neural Networks without Batch Normalization Codes for this paper BNN - BN = ? Training Binary Neural Networks without Bat

VITA 40 Dec 30, 2022
Vehicles Counting using YOLOv4 + DeepSORT + Flask + Ngrok

A project for counting vehicles using YOLOv4 + DeepSORT + Flask + Ngrok

Duong Tran Thanh 37 Dec 16, 2022
Code for C2-Matching (CVPR2021). Paper: Robust Reference-based Super-Resolution via C2-Matching.

C2-Matching (CVPR2021) This repository contains the implementation of the following paper: Robust Reference-based Super-Resolution via C2-Matching Yum

Yuming Jiang 151 Dec 26, 2022
StarGAN v2 - Official PyTorch Implementation (CVPR 2020)

StarGAN v2 - Official PyTorch Implementation StarGAN v2: Diverse Image Synthesis for Multiple Domains Yunjey Choi*, Youngjung Uh*, Jaejun Yoo*, Jung-W

Clova AI Research 3.1k Jan 09, 2023
Automatic Idiomatic Expression Detection

IDentifier of Idiomatic Expressions via Semantic Compatibility (DISC) An Idiomatic identifier that detects the presence and span of idiomatic expressi

5 Jun 09, 2022
Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)

Manga Filling with ScreenVAE SIGGRAPH ASIA 2020 | Project Website | BibTex This repository is for ScreenVAE introduced in the following paper "Manga F

30 Dec 24, 2022
A scanpy extension to analyse single-cell TCR and BCR data.

Scirpy: A Scanpy extension for analyzing single-cell immune-cell receptor sequencing data Scirpy is a scalable python-toolkit to analyse T cell recept

ICBI 145 Jan 03, 2023
Simple tutorials using Google's TensorFlow Framework

TensorFlow-Tutorials Introduction to deep learning based on Google's TensorFlow framework. These tutorials are direct ports of Newmu's Theano Tutorial

Nathan Lintz 6k Jan 06, 2023
Convert onnx models to pytorch.

onnx2torch onnx2torch is an ONNX to PyTorch converter. Our converter: Is easy to use – Convert the ONNX model with the function call convert; Is easy

ENOT 264 Dec 30, 2022
HugsVision is a easy to use huggingface wrapper for state-of-the-art computer vision

HugsVision is an open-source and easy to use all-in-one huggingface wrapper for computer vision. The goal is to create a fast, flexible and user-frien

Labrak Yanis 166 Nov 27, 2022
Trafffic prediction analysis using hybrid models - Machine Learning

Hybrid Machine learning Model Clone the Repository Create a new Directory as assests and download the model from the below link Model Link To Start th

1 Feb 08, 2022
Code for AutoNL on ImageNet (CVPR2020)

Neural Architecture Search for Lightweight Non-Local Networks This repository contains the code for CVPR 2020 paper Neural Architecture Search for Lig

Yingwei Li 104 Aug 31, 2022
Text mining project; Using distilBERT to predict authors in the classification task authorship attribution.

DistilBERT-Text-mining-authorship-attribution Dataset used: https://www.kaggle.com/azimulh/tweets-data-for-authorship-attribution-modelling/version/2

1 Jan 13, 2022
Beancount-mercury - Beancount importer for Mercury Startup Checking

beancount-mercury beancount-mercury provides an Importer for converting CSV expo

Michael Lynch 4 Oct 31, 2022
🔥3D-RecGAN in Tensorflow (ICCV Workshops 2017)

3D Object Reconstruction from a Single Depth View with Adversarial Learning Bo Yang, Hongkai Wen, Sen Wang, Ronald Clark, Andrew Markham, Niki Trigoni

Bo Yang 125 Nov 26, 2022
PaddleRobotics is an open-source algorithm library for robots based on Paddle, including open-source parts such as human-robot interaction, complex motion control, environment perception, SLAM positioning, and navigation.

简体中文 | English PaddleRobotics paddleRobotics是基于paddle的机器人开源算法库集,包括人机交互、复杂运动控制、环境感知、slam定位导航等开源算法部分。 人机交互 主动多模交互技术TFVT-HRI 主动多模交互技术是通过视觉、语音、触摸传感器等输入机器人

185 Dec 26, 2022
Bio-OFC gym implementation and Gym-Fly environment

Bio-OFC gym implementation and Gym-Fly environment This repository includes the gym compatible implementation of the Bio-OFC algorithm from the paper

Siavash Golkar 1 Nov 16, 2021