An open source library for deep learning end-to-end dialog systems and chatbots.

Overview

License Apache 2.0 Python 3.6, 3.7 Downloads

DeepPavlov is an open-source conversational AI library built on TensorFlow, Keras and PyTorch.

DeepPavlov is designed for

  • development of production ready chat-bots and complex conversational systems,
  • research in the area of NLP and, particularly, of dialog systems.

Quick Links

Please leave us your feedback on how we can improve the DeepPavlov framework.

Models

Named Entity Recognition | Slot filling

Intent/Sentence Classification | Question Answering over Text (SQuAD)

Knowledge Base Question Answering

Sentence Similarity/Ranking | TF-IDF Ranking

Morphological tagging | Syntactic parsing

Automatic Spelling Correction | ELMo training and fine-tuning

Speech recognition and synthesis (ASR and TTS) based on NVIDIA NeMo

Entity Linking | Multitask BERT

Skills

Goal(Task)-oriented Bot | Seq2seq Goal-Oriented bot

Open Domain Questions Answering | eCommerce Bot

Frequently Asked Questions Answering | Pattern Matching

Embeddings

BERT embeddings for the Russian, Polish, Bulgarian, Czech, and informal English

ELMo embeddings for the Russian language

FastText embeddings for the Russian language

Auto ML

Tuning Models with Evolutionary Algorithm

Integrations

REST API | Socket API | Yandex Alice

Telegram | Microsoft Bot Framework

Amazon Alexa | Amazon AWS

Installation

  1. We support Linux and Windows platforms, Python 3.6 and Python 3.7

    • Python 3.5 is not supported!
    • installation for Windows requires Git(for example, git) and Visual Studio 2015/2017 with C++ build tools installed!
  2. Create and activate a virtual environment:

    • Linux
    python -m venv env
    source ./env/bin/activate
    
    • Windows
    python -m venv env
    .\env\Scripts\activate.bat
    
  3. Install the package inside the environment:

    pip install deeppavlov
    

QuickStart

There is a bunch of great pre-trained NLP models in DeepPavlov. Each model is determined by its config file.

List of models is available on the doc page in the deeppavlov.configs (Python):

from deeppavlov import configs

When you're decided on the model (+ config file), there are two ways to train, evaluate and infer it:

GPU requirements

To run supported DeepPavlov models on GPU you should have CUDA 10.0 installed on your host machine and TensorFlow with GPU support (tensorflow-gpu) installed in your python environment. Current supported TensorFlow version is 1.15.2. Run

pip install tensorflow-gpu==1.15.2

before installing model's package requirements to install supported tensorflow-gpu version.

Before making choice of an interface, install model's package requirements (CLI):

python -m deeppavlov install <config_path>
  • where <config_path> is path to the chosen model's config file (e.g. deeppavlov/configs/ner/slotfill_dstc2.json) or just name without .json extension (e.g. slotfill_dstc2)

Command line interface (CLI)

To get predictions from a model interactively through CLI, run

python -m deeppavlov interact <config_path> [-d]
  • -d downloads required data -- pretrained model files and embeddings (optional).

You can train it in the same simple way:

python -m deeppavlov train <config_path> [-d]

Dataset will be downloaded regardless of whether there was -d flag or not.

To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.

There are even more actions you can perform with configs:

python -m deeppavlov <action> <config_path> [-d]
  • <action> can be
    • download to download model's data (same as -d),
    • train to train the model on the data specified in the config file,
    • evaluate to calculate metrics on the same dataset,
    • interact to interact via CLI,
    • riseapi to run a REST API server (see doc),
    • telegram to run as a Telegram bot (see doc),
    • msbot to run a Miscrosoft Bot Framework server (see doc),
    • predict to get prediction for samples from stdin or from <file_path> if -f <file_path> is specified.
  • <config_path> specifies path (or name) of model's config file
  • -d downloads required data

Python

To get predictions from a model interactively through Python, run

from deeppavlov import build_model

model = build_model(<config_path>, download=True)

# get predictions for 'input_text1', 'input_text2'
model(['input_text1', 'input_text2'])
  • where download=True downloads required data from web -- pretrained model files and embeddings (optional),
  • <config_path> is path to the chosen model's config file (e.g. "deeppavlov/configs/ner/ner_ontonotes_bert_mult.json") or deeppavlov.configs attribute (e.g. deeppavlov.configs.ner.ner_ontonotes_bert_mult without quotation marks).

You can train it in the same simple way:

from deeppavlov import train_model 

model = train_model(<config_path>, download=True)
  • download=True downloads pretrained model, therefore the pretrained model will be, first, loaded and then train (optional).

Dataset will be downloaded regardless of whether there was -d flag or not.

To train on your own data you need to modify dataset reader path in the train config doc. The data format is specified in the corresponding model doc page.

You can also calculate metrics on the dataset specified in your config file:

from deeppavlov import evaluate_model 

model = evaluate_model(<config_path>, download=True)

There are also available integrations with various messengers, see Telegram Bot doc page and others in the Integrations section for more info.

Breaking Changes

Breaking changes in version 0.7.0

Breaking changes in version 0.6.0

  • REST API:
    • all models default endpoints were renamed to /model
    • by default model arguments names are taken from chainer.in configuration parameter instead of pre-set names from a settings file
    • swagger api endpoint moved from /apidocs to /docs
  • when using "max_proba": true in a proba2labels component for classification, it will return single label for every batch element instead of a list. One can set "top_n": 1 to get batches of single item lists as before

Breaking changes in version 0.5.0

  • dependencies have to be reinstalled for most pipeline configurations
  • models depending on tensorflow require CUDA 10.0 to run on GPU instead of CUDA 9.0
  • scikit-learn models have to be redownloaded or retrained

Breaking changes in version 0.4.0!

  • default target variable name for neural evolution was changed from MODELS_PATH to MODEL_PATH.

Breaking changes in version 0.3.0!

  • component option fit_on_batch in configuration files was removed and replaced with adaptive usage of the fit_on parameter.

Breaking changes in version 0.2.0!

  • utils module was moved from repository root in to deeppavlov module
  • ms_bot_framework_utils,server_utils, telegram utils modules was renamed to ms_bot_framework, server and telegram correspondingly
  • rename metric functions exact_match to squad_v2_em and squad_f1 to squad_v2_f1
  • replace dashes in configs name with underscores

Breaking changes in version 0.1.0!

  • As of version 0.1.0 all models, embeddings and other downloaded data for provided configurations are by default downloaded to the .deeppavlov directory in current user's home directory. This can be changed on per-model basis by modifying a ROOT_PATH variable or related fields one by one in model's configuration file.

  • In configuration files, for all features/models, dataset readers and iterators "name" and "class" fields are combined into the "class_name" field.

  • deeppavlov.core.commands.infer.build_model_from_config() was renamed to build_model and can be imported from the deeppavlov module directly.

  • The way arguments are passed to metrics functions during training and evaluation was changed and documented.

License

DeepPavlov is Apache 2.0 - licensed.

The Team

DeepPavlov is built and maintained by Neural Networks and Deep Learning Lab at MIPT.

Comments
  • Regarding Spelling Error model

    Regarding Spelling Error model

    Thanks for amazing toolkit :) Can you please share your views on below questions

    1. How does correct_prior & incorrect_prior calculation done in Error model ?
    2. How do we incorporate "count" with incorrect-correct pair e.g. if training data is in form of (intended_word, observed_word, count).
    3. Is there any other way we can combine LM score & EM score in LM beam search method ?

    Thanks a lot !!

    opened by smilenrhyme 26
  • Error while trying to get the probablities of the predicted entities using ontonotes_bert ner model

    Error while trying to get the probablities of the predicted entities using ontonotes_bert ner model

    Deeppavlov version: 0.12.1 Python version: 3.7.7 "return_probas"set to true in ner_ontonotes_bert config json Command: python -m deeppavlov interact F:\miniconda3\envs\ute_query_params_service\Lib\site-packages\deeppavlov\configs\ner\ner_ontonotes_bert.js Command string: London is in England. Full Traceback: [nltk_data] Downloading package punkt to [nltk_data] C:\Users\User\AppData\Roaming\nltk_data... [nltk_data] Package punkt is already up-to-date! [nltk_data] Downloading package stopwords to [nltk_data] C:\Users\User\AppData\Roaming\nltk_data... [nltk_data] Package stopwords is already up-to-date! [nltk_data] Downloading package perluniprops to [nltk_data] C:\Users\User\AppData\Roaming\nltk_data... [nltk_data] Package perluniprops is already up-to-date! [nltk_data] Downloading package nonbreaking_prefixes to [nltk_data] C:\Users\User\AppData\Roaming\nltk_data... [nltk_data] Package nonbreaking_prefixes is already up-to-date! WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\bert_dp\tokenization.py:125: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

    2020-11-30 21:37:31.122 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 115: [loading vocabulary from C:\Users\User.deeppavlov\models\ner_ontonotes_bert\tag.dict] WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:37: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:222: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:222: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:193: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py:236: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py:314: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\bert_dp\modeling.py:178: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\bert_dp\modeling.py:418: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\bert_dp\modeling.py:499: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.

    WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

    • https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
    • https://github.com/tensorflow/addons
    • https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\bert_dp\modeling.py:366: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use rate instead of keep_prob. Rate should be set to rate = 1 - keep_prob. WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\bert_dp\modeling.py:680: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.Dense instead. WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\tensorflow_core\python\layers\core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\bert_dp\modeling.py:283: The name tf.erf is deprecated. Please use tf.math.erf instead.

    WARNING:tensorflow:Variable *= will be deprecated. Use var.assign(var * other) if you want assignment to the variable value or x = x * y if you want a new python Tensor object. WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py:75: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\tensorflow_core\contrib\crf\python\ops\crf.py:213: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version. Instructions for updating: Please use keras.layers.RNN(cell), which is equivalent to this API WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:234: The name tf.train.AdadeltaOptimizer is deprecated. Please use tf.compat.v1.train.AdadeltaOptimizer instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:131: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:131: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:94: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\tensorflow_core\python\training\moving_averages.py:433: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py:671: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py:244: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

    WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\models\bert\bert_sequence_tagger.py:249: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. 2020-11-30 21:40:57.982 INFO in 'deeppavlov.core.models.tf_model'['tf_model'] at line 51: [loading model from C:\Users\User.deeppavlov\models\ner_ontonotes_bert\model] WARNING:tensorflow:From f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\models\tf_model.py:54: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

    x::London is in England. Traceback (most recent call last): File "f:\miniconda3\envs\ute_query_params_service\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "f:\miniconda3\envs\ute_query_params_service\lib\runpy.py", line 85, in run_code exec(code, run_globals) File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov_main.py", line 4, in main() File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\deep.py", line 89, in main interact_model(pipeline_config_path) File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\commands\infer.py", line 89, in interact_model pred = model(*args) File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\common\chainer.py", line 207, in call return self._compute(*args, param_names=self.in_x, pipe=self.pipe, targets=self.out_params) File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\common\chainer.py", line 230, in _compute res = component.call(*x) File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\data\simple_vocab.py", line 92, in call looked_up_batch = [self(sample, is_top=False) for sample in batch] File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\data\simple_vocab.py", line 92, in looked_up_batch = [self(sample, is_top=False) for sample in batch] File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\data\simple_vocab.py", line 92, in call looked_up_batch = [self(sample, is_top=False) for sample in batch] File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\data\simple_vocab.py", line 92, in looked_up_batch = [self(sample, is_top=False) for sample in batch] File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\data\simple_vocab.py", line 92, in call looked_up_batch = [self(sample, is_top=False) for sample in batch] File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\data\simple_vocab.py", line 92, in looked_up_batch = [self(sample, is_top=False) for sample in batch] File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\data\simple_vocab.py", line 94, in call return self[batch] File "f:\miniconda3\envs\ute_query_params_service\lib\site-packages\deeppavlov\core\data\simple_vocab.py", line 161, in getitem raise NotImplementedError("not implemented for type {}".format(type(key))) NotImplementedError: not implemented for type <class 'numpy.float32'>

    I would like to get the probablities of predicted entities. Is return_probas is the way to go? Please help. Thanks in advance.

    • Ravishankar
    opened by ravishpankar 20
  • Data set creation routine for gobot DSTC 2 format

    Data set creation routine for gobot DSTC 2 format

    Hi,

    I want to create data set creation routine for gobot DSTC 2 format. I know that that there is an on going refactoring of the codebase for the Goal-oriented bot (gobot).

    Also, there is a new DSTC 8 challenge and Alexa Prize socialbot which is to be open sourced.

    So I want to ask if this feature would be needed or is it duplication of work?

    Ideally, I want to pull the routine to the deeppavlov repo, so I need some guidance/advice before jumping into the implementation.

    Things I want to clarify:

    1. Is this routine needed to be developed? Or is it already underway and it would be duplication of work?
    2. What format would be best (DSTC 2 json, DSTC 8, etc)?
    3. I want to create CLI with python, is it good?

    Anything else you think might be appropriate.

    feature request 
    opened by Eugen2525 17
  • NER fine-tuninng using pretrained using ner_ontonotes_bert_mult

    NER fine-tuninng using pretrained using ner_ontonotes_bert_mult

    I am trying to use Bert for NER recognition of standard and some custom entities. For instance, I have the following entities: 'financial_instrument', 'amount', 'percent', 'hashtag', 'exchange', 'number', 'sector', 'period', 'location', 'media_type', 'analyst', 'ticker', 'person', 'price_movement', 'rating_agency', 'product', 'amount_price_target', 'financial_topic', 'publication', 'company', 'event' and some of them I can map to the existing ones in BIO format, such as: EVENT, PERCENT, NUMBER, GPE, PERIOD, NORG, etc. However, some of my entities are new and cannot be mapped. I was wondering if I can use the pretrained ner_ontonotes_bert_mult and just add my specific entities to fine-tune the model? Is this possible and could you provide to me sample code?

    Tnx

    opened by igormis 17
  • deeppavlov.core.common.errors.ConfigError: 'Given fasttext model does NOT match fasttext model used previously to train loaded model'

    deeppavlov.core.common.errors.ConfigError: 'Given fasttext model does NOT match fasttext model used previously to train loaded model'

    I goy the following error

    Traceback (most recent call last): File "deep.py", line 63, in main() File "deep.py", line 55, in main interact_model_by_telegram(pipeline_config_path, token) File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/telegram_utils/telegram_ui.py", line 57, in interact_model_by_telegram model = build_model_from_config(config) File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/commands/infer.py", line 34, in build_model_from_config model = from_params(REGISTRY[model_name], model_config, vocabs=vocabs, mode=mode) File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/common/params.py", line 49, in from_params mode=kwargs['mode']) File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/common/params.py", line 52, in from_params model = cls(**dict(config_params, **kwargs)) File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/models/tf_backend.py", line 47, in call obj.init(*args, **kwargs) File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/core/models/tf_backend.py", line 28, in _wrapped return func(*args, **kwargs) File "/home/mdixie/.conda/envs/deeppavlov/lib/python3.6/site-packages/deeppavlov-0.0.1-py3.6.egg/deeppavlov/models/classifiers/intents/intent_model.py", line 138, in init "Given fasttext model does NOT match fasttext model used previously to train loaded model") deeppavlov.core.common.errors.ConfigError: 'Given fasttext model does NOT match fasttext model used previously to train loaded model'

    opened by dixiematt8 16
  • Bert for classification

    Bert for classification

    Здравствуйте! Я новичок в deep learning, хотела бы претренированную модель руберт использовать для своих данных. Задача обычная классификация русских текстов - 5 классов. Работаю на гугл колаб. Есть ли у вас туториалы пошагово как применять bert от deeppavlov для собаственных данных? как менять конфиг файл? Для примера взяла rusentiment_bert.json, но не могу разобраться как привести свои данные в тот формат, который требуется для MODELS_PATH. Спасибо!

    opened by Aygera 15
  • building go-bot in russian

    building go-bot in russian

    Hi! I want to build a go-bot using DeepPavlov in russian. The task of gobot is to output phone number of requested employee by his name, surname, fathers name. I plan to use tutorial03 as a reference. And the main idea is using instead of DSTC2 data set a new one, which i gonna generate in DSTC2 format. Has the described aproach a right to exist?

    help wanted 
    opened by vitalyuf 15
  • Upgrading Tensorflow to 1.15.4 in tf.txt

    Upgrading Tensorflow to 1.15.4 in tf.txt

    What problem are we trying to solve?:

    There are many security risks around TensorFlow version 1.15.2. These issues have been fixed in version 1.15.4 and newer versions. Newer versions have also been optimized better than 1.15.2.
    

    How can we solve it?:

    Upgrading the TensorFlow to 1.15.4 in the deeppavlov>>requirements>>tf.txt from tensorflow==1.15.2 to tensorflow==1.15.4.
    

    Are there other issues that block this solution?:

    Missing out on the optimization and security enhancements.
    

    Any important aspect to consider?:

    Making sure that upgrading the TensorFlow to 1.15.4 causes no issue in syntaxes and yields the same results as 1.15.2.
    
    enhancement 
    opened by Rajmehta123 12
  • Library does't see GPU

    Library does't see GPU

    Hi everyone, thanks for your library! I use several BERT models, but I can't train them using GPU. I describe all process:

    1. I install Deeppavlov package into docker container
    2. I install tensorflow-gpu: pip install tensorflow-gpu==1.14.0
    3. I install model’s package requirements and download model
    4. I move docker container to another machine with acсess to GPU. This machine has CUDA and cudnn. But when I train model, it uses CPU. I try to check access to GPU using this command: tf.test_is_gpu_avalaible. It returns me False( May be there is a mistake in this sequence of actions?
    opened by ostreech1997 12
  • feat: Imdb sentiment dataset reader

    feat: Imdb sentiment dataset reader

    This PR implements a dataset reader for the IMDb sentiment classification dataset. It also includes a json configuration for BERT (en, cased) which is mostly the same as the configuration for rusentiment except for the max seq length and batch size (which I set to values such that I don't get out-of-memory on my hardware).

    This PR also includes a fix for the sets_accuracy metric which should now correctly work for string labels (i.e. wrap them into sets instead converting them to sets). Also I added reporting of cached files in download_decompress.

    opened by sgrechanik-h 12
  • ODQA inference speed very very slow

    ODQA inference speed very very slow

    Running the default configuration and model on a EC2 p2.xlarge instance (60~GB Ram and Nvidia K80 GPU) and inference for simple questions take 40 seconds to 5 minutes.

    Sometimes, no result even after 10 minutes.

    MobaXterm_2019-05-27_16-36-13
    opened by shubhank008 12
  • add fusion in decoder

    add fusion in decoder

    Файл c кодом модели полностью перекопирован из репозитория FiD, мне показалось что ради одного файла добавлять целый репозиторий не стоит, но мне не очевдно насколько это правильное решение

    opened by LogicZMaksimka 0
  • 👩‍💻📞DeepPavlov Community Call #18

    👩‍💻📞DeepPavlov Community Call #18

    Привет, друзья!

    Мы рады вернуться в этом месяце с DeepPavlov Community Call на русском языке. На предстоящем вебинаре к нам придет приглашенный гость Борис Галицкий, ассоциированный сотрудник лаборатории Интеллектуальных систем и структурного анализа НИУ ВШЭ, основатель нескольких стартапов в области ИИ, профессор ANECA, а также бывший сотрудник Oracle, представит доклад на тему “Дискурсивный анализ текста для организации диалога”.

    Сделать диалог с чат-ботом логичным и интересным — важнейшая задача области Conversational AI. Для этого применяются самые разные подходы, и один из них — дискурсивный анализ текста. Его идея состоит в том, чтобы чат-бот помог пользователю сфокусироваться лишь на каком-либо предложении из всего текста. В дискурсивном дереве текст разбивается на части, связанные логическими отношениями, и чат-бот направляет по ним пользователя, развивая диалог. Например, это могут быть временные (temporal) отношения, когда пользователя наверняка заинтересует, что будет после описанного события или что было до него. На нашем вебинаре Борис Галицкий подробно расскажет о способе управления ходом диалога в чат-боте на основе дискурсивного анализа текста.

    DeepPavlov Community Call #11, Русская версия (27 июля, 2022) Мы проведем следующий звонок 27 июля 2022 в 19.00 по Московскому времени (19 MSK). Добавьте напоминание в календарь: http://bit.ly/MonthlyDPCommunityCall2021Ru

    Повестка DeepPavlov Community Call #18, Русская версия:

    7:00pm–7:10pm | Приветствие 7:10 –7:45pm | Борис Галицкий: Дискурсивный анализ текста для организации диалога 7:45pm–8:00pm | Вопросы и ответы с Борисом Галицким и командой инженеров DeepPavlov

    В случае, если вы пропустили Community Calls ранее, вы всегда их можете найти в плейлисте.

    Мы приглашаем вас присоединиться к нам, чтобы сообщить, что вы думаете о последних изменениях, поделиться своими ожиданиями от предстоящей версии библиотеки и рассказать, как DeepPavlov помогает вам в ваших проектах!

    Оставьте отзыв о библиотеке DeepPavlov

    Мы хотим услышать вас. Вы можете заполнить форму ниже, чтобы сообщить нам, как вы используете DeepPavlov Library, что вы хотите, чтобы мы добавили или улучшили! http://bit.ly/DPLibrary2021Survey

    Заинтересовались? Не упускайте шанс и присоединяйтесь к нам! Этот Call открыт для всех энтузиастов в области Conversational AI.

    discussion 
    opened by PolinaMrdv 0
Releases(1.0.1)
Owner
Neural Networks and Deep Learning lab, MIPT
Neural Networks and Deep Learning lab, MIPT
Train BPE with fastBPE, and load to Huggingface Tokenizer.

BPEer Train BPE with fastBPE, and load to Huggingface Tokenizer. Description The BPETrainer of Huggingface consumes a lot of memory when I am training

Lizhuo 1 Dec 23, 2021
Simple Python library, distributed via binary wheels with few direct dependencies, for easily using wav2vec 2.0 models for speech recognition

Wav2Vec2 STT Python Beta Software Simple Python library, distributed via binary wheels with few direct dependencies, for easily using wav2vec 2.0 mode

David Zurow 22 Dec 29, 2022
🤗 The largest hub of ready-to-use NLP datasets for ML models with fast, easy-to-use and efficient data manipulation tools

🤗 The largest hub of ready-to-use NLP datasets for ML models with fast, easy-to-use and efficient data manipulation tools

Hugging Face 15k Jan 02, 2023
This is the source code of RPG (Reward-Randomized Policy Gradient)

RPG (Reward-Randomized Policy Gradient) Zhenggang Tang*, Chao Yu*, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Shaolei Du, Yu Wang, Yi Wu (

40 Nov 25, 2022
nlp基础任务

NLP算法 说明 此算法仓库包括文本分类、序列标注、关系抽取、文本匹配、文本相似度匹配这五个主流NLP任务,涉及到22个相关的模型算法。 框架结构 文件结构 all_models ├── Base_line │   ├── __init__.py │   ├── base_data_process.

zuxinqi 23 Sep 22, 2022
Various capabilities for static malware analysis.

Malchive The malchive serves as a compendium for a variety of capabilities mainly pertaining to malware analysis, such as scripts supporting day to da

MITRE Cybersecurity 64 Nov 22, 2022
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

Simplemma: a simple multilingual lemmatizer for Python Purpose Lemmatization is the process of grouping together the inflected forms of a word so they

Adrien Barbaresi 70 Dec 29, 2022
[EMNLP 2021] Mirror-BERT: Converting Pretrained Language Models to universal text encoders without labels.

[EMNLP 2021] Mirror-BERT: Converting Pretrained Language Models to universal text encoders without labels.

Cambridge Language Technology Lab 61 Dec 10, 2022
This is a NLP based project to extract effective date of the contract from their text files.

Date-Extraction-from-Contracts This is a NLP based project to extract effective date of the contract from their text files. Problem statement This is

Sambhav Garg 1 Jan 26, 2022
This converter will create the exact measure for your cappuccino recipe from the grandiose Rafaella Ballerini!

About CappuccinoJs This converter will create the exact measure for your cappuccino recipe from the grandiose Rafaella Ballerini! Este conversor criar

Arthur Ottoni Ribeiro 48 Nov 15, 2022
Universal End2End Training Platform, including pre-training, classification tasks, machine translation, and etc.

背景 安装教程 快速上手 (一)预训练模型 (二)机器翻译 (三)文本分类 TenTrans 进阶 1. 多语言机器翻译 2. 跨语言预训练 背景 TrenTrans是一个统一的端到端的多语言多任务预训练平台,支持多种预训练方式,以及序列生成和自然语言理解任务。 安装教程 git clone git

Tencent Minority-Mandarin Translation Team 42 Dec 20, 2022
Anuvada: Interpretable Models for NLP using PyTorch

Anuvada: Interpretable Models for NLP using PyTorch So, you want to know why your classifier arrived at a particular decision or why your flashy new d

EDGE 102 Oct 01, 2022
Sentence boundary disambiguation tool for Japanese texts (日本語文境界判定器)

Bunkai Bunkai is a sentence boundary (SB) disambiguation tool for Japanese texts. Quick Start $ pip install bunkai $ echo -e '宿を予約しました♪!まだ2ヶ月も先だけど。早すぎ

Megagon Labs 160 Dec 23, 2022
Script to generate VAD dataset used in Asteroid recipe

About the dataset LibriVAD is an open source dataset for voice activity detection in noisy environments. It is derived from LibriSpeech signals (clean

11 Sep 15, 2022
Learning Spatio-Temporal Transformer for Visual Tracking

STARK The official implementation of the paper Learning Spatio-Temporal Transformer for Visual Tracking Highlights The strongest performances Tracker

Multimedia Research 485 Jan 04, 2023
Simple text to phones converter for multiple languages

Phonemizer -- foʊnmaɪzɚ The phonemizer allows simple phonemization of words and texts in many languages. Provides both the phonemize command-line tool

CoML 762 Dec 29, 2022
Minimal GUI for accessing the Watson Text to Speech service.

Description Minimal graphical application for accessing the Watson Text to Speech service. Requirements Python 3 plus all dependencies listed in requi

Moritz Maxeiner 1 Oct 22, 2021
Implementation of ProteinBERT in Pytorch

ProteinBERT - Pytorch (wip) Implementation of ProteinBERT in Pytorch. Original Repository Install $ pip install protein-bert-pytorch Usage import torc

Phil Wang 92 Dec 25, 2022
Built for cleaning purposes in military institutions

Ferramenta do AL Construído para fins de limpeza em instituições militares. Instalação Requer python = 3.2 pip install -r requirements.txt Usagem Exe

0 Aug 13, 2022
Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing

Token Shift GPT Implementation of Token Shift GPT - An autoregressive model that relies solely on shifting along the sequence dimension and feedforwar

Phil Wang 32 Oct 14, 2022