Adds a boolean aligned option to the FastText object's constructor.

If set to True, the FastText embeddings will be initialized with the aligned MUSE embeddings (see details here).

If not specified or set to False, the regular FastText embeddings are used. This way, the PR does not break any code written before the PR.

Example usage:

>>> from torchnlp.word_to_vector import FastText
>>> from scipy.spatial.distance import euclidean as dist

# Load aligned FastText embeddings for English and French
en_vectors = FastText(aligned=True)
fr_vectors = FastText(language='fr', aligned=True)

# Compare the euclidean distances of semantically related vs unrelated words
>>> dist(en_vectors['car'], fr_vectors['voiture'])
0.61194908618927
>>> dist(en_vectors['car'], fr_vectors['baguette'])
1.2417925596237183

Expected Behavior

encoder = MosesEncoder( ["<s> hello This ain't funny. </s>", "<s> Don't? </s>"]) print (encoder.encode("<s> hello </s>"))

--CONSOLE--- tensor([3, 5, 2])

Actual Behavior

encoder = MosesEncoder( ["<s> hello This ain't funny. </s>", "<s> Don't? </s>"]) print (encoder.encode("<s> hello </s>"))

--CONSOLE--- tensor([ 5, 6, 7, 8, 5, 14, 6, 7])

Explanation

Most if this tokenizers are not aware of this special tokens and end up splitting the special token into different tokens. For instance the '<s>' token becames '<', 's', '>'.

My solution to this problem was to create a method for masking special tokens and another one to restore them in place.

   def _mask_reserved_tokens(self, sequence):
        reserved_tokens = re.findall(r'\<pad\>|\<unk\>|\</s\>|\<s\>|\<copy\>', sequence)
        sequence = re.sub(r'\<pad\>|\<unk\>|\</s\>|\<s\>|\<copy\>', "RESERVEDTOKENMASK", sequence)
        return reserved_tokens, sequence

    def _restore_reserved_tokens(self, reserved_tokens, sequence):
        sequence = _detokenize(sequence)
        for token in reserved_tokens:
            sequence = sequence.replace('RESERVEDTOKENMASK', token, 1)
        return _tokenize(sequence)

Then the encode function becames:

def encode(self, sequence):
        """ Encodes a ``sequence``.
        Args:
            sequence (str): String ``sequence`` to encode.
        Returns:
            torch.Tensor: Encoding of the ``sequence``.
        """
        sequence = super().encode(sequence)
        reserved_tokens, sequence = self._mask_reserved_tokens(sequence)
        sequence = self.tokenize(sequence)
        sequence = self._restore_reserved_tokens(reserved_tokens, sequence)
        vector = [self.stoi.get(token, self.unknown_index) for token in sequence]
        if self.append_eos:
            vector.append(self.eos_index)
        return torch.tensor(vector)

I dont know if this is just a problem that I have but If not I believe that this should be handled natively.

There is one remaining issue that lies with the dataset itself. The code works for all the datasets except for SNLI and QQP where there are some lines in the data file that contain too much or too little data fields. See my comment in the issue discussion for more explanations. I think that someone that is used to those datasets should be able to know what to do.

lambda cannot be pickled, therefore it is better to not use it as attribute. Even though there are alternatives to pickle, some libraries use pickle internally, which is why it's better to support it.

Tests were added to all samplers and encoders for whether objects can be pickled.

Expected Behavior

Load FastText vectors

Environment: Ubuntu 16.04 Python 3.6.4 Pytorch 0.4.1

Actual Behavior

Throws the following error:

File "", line 1, in File "/home/zxi/.local/lib/python3.6/site-packages/torchnlp/word_to_vector/fast_text.py", line 83, in init super(FastText, self).init(name, url=url, **kwargs) File "/home/zxi/.local/lib/python3.6/site-packages/torchnlp/word_to_vector/pretrained_word_vectors.py", line 72, in init self.cache(name, cache, url=url) File "/home/zxi/.local/lib/python3.6/site-packages/torchnlp/word_to_vector/pretrained_word_vectors.py", line 153, in cache word, len(entries), dim)) RuntimeError: Vector for token darang has 230 dimensions, but previously read vectors have 300 dimensions. All vectors must have the same number of dimensions.

Steps to Reproduce the Problem

1. Open python console
2. Write the following code:

       from torchnlp.word_to_vector import FastText
       vectors = FastText()
   
   

3. Throws the error mentioned above.

Unable to recreate the SRU example code provided in readme. Getting a Variable object has no attribute 'new_zeros'. Running pytorch .3.0.

Expected Behavior

RETURNS: ( output [torch.FloatTensor (6x3x20)], hidden_state [torch.FloatTensor (2x3x20)] )

Actual Behavior

`--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) in () ----> 1 sru(input_)

/anaconda3/envs/nlp/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs) 355 result = self._slow_forward(*input, **kwargs) 356 else: --> 357 result = self.forward(*input, **kwargs) 358 for hook in self._forward_hooks.values(): 359 hook_result = hook(self, input, result)

/anaconda3/envs/nlp/lib/python3.6/site-packages/torchnlp/nn/sru.py in forward(self, input_, c0) 509 dir_ = 2 if self.bidirectional else 1 510 if c0 is None: --> 511 zeros = input_.new_zeros(input_.size(1), self.hidden_size * dir_) 512 c0 = [zeros for i in range(self.num_layers)] 513 else:

/anaconda3/envs/nlp/lib/python3.6/site-packages/torch/autograd/variable.py in getattr(self, name) 65 if name in self._fallthrough_methods: 66 return getattr(self.data, name) ---> 67 return object.getattribute(self, name) 68 69 def getitem(self, key):

AttributeError: 'Variable' object has no attribute 'new_zeros' `

Steps to Reproduce the Problem

`from torchnlp.nn import SRU import torch

input_ = torch.autograd.Variable(torch.randn(6, 3, 10)) sru = SRU(10, 20)

sru(input_) `

Problem description

Some classes have lambdas as default arguments (e.g. here). This prevents these objects from being pickled.

Steps to Reproduce the Problem

import pickle
from torchnlp.text_encoders import StaticTokenizerEncoder

encoder = StaticTokenizerEncoder(['hi', 'you'])
pickle.dumps(encoder) 

# raises error
PicklingError: Can't pickle <function StaticTokenizerEncoder.<lambda> at 0x7fb77c042b70>: attribute lookup StaticTokenizerEncoder.<lambda> on torchnlp.text_encoders.static_tokenizer_encoder failed

Solution

Replace lambdas by actual functions.

In the readme where loading the glove embeddings is demonstrated, the lambda function is extremely inefficient. The encoder vocab is converted in to a set every single time the function is called. Instead of:

import torch
from torchnlp.encoders.text import WhitespaceEncoder
from torchnlp.word_to_vector import GloVe

encoder = WhitespaceEncoder(["now this ain't funny", "so don't you dare laugh"])

pretrained_embedding = GloVe(name='6B', dim=100, is_include=lambda w: w in set(encoder.vocab))
embedding_weights = torch.Tensor(encoder.vocab_size, pretrained_embedding.dim)
for i, token in enumerate(encoder.vocab):
    embedding_weights[i] = pretrained_embedding[token]

it should be:

import torch
from torchnlp.encoders.text import WhitespaceEncoder
from torchnlp.word_to_vector import GloVe

encoder = WhitespaceEncoder(["now this ain't funny", "so don't you dare laugh"])

vocab_set = set(encoder.vocab)

pretrained_embedding = GloVe(name='6B', dim=100, is_include=lambda w: w in vocab_set)
embedding_weights = torch.Tensor(encoder.vocab_size, pretrained_embedding.dim)
for i, token in enumerate(encoder.vocab):
    embedding_weights[i] = pretrained_embedding[token]

For my personal code (where the vocab was much much larger than the example) this changed the time taken to load the embeddings from 2 hours to ~10 seconds.

This PR implements the common __contains__() method for the _PretrainedWordVectors class. This way, the Python in keyword can be used to easily find out if a vector representation of a token exists.

Example usage:

>>> from torchnlp.word_to_vector import FastText
>>> vectors = FastText()

>>> 'the' in vectors
True
>>> 'theqwe' in vectors
False

• The BucketBatchSampler's sort_key lambda function in examples/snli/train.py used the row variable, which was loaded due to the for loop a few dozens line above. This also meant that the sort_key lambda function did not have too much of an effect, as it always returned the same number.

• The same issue can be seen in case of the dev dataset.

• This commit fixes the situation by making use of the passed index and the loaded train and dev datasets.

Replaced .view with .reshape in Attention module in order to avoid RuntimeError: "view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.".

There is a small typo in torchnlp/encoders/text/subword_text_tokenizer.py.

Should read expressed rather than experessed.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

I wanted to install this package in anaconda with "conda install torchnlp", but it came out with the "PackagesNotFoundError" notion, how can I install it in anaconda？

Expected Behavior

from torchnlp.encoders.text import SpacyEncoder encoder = SpacyEncoder(["This ain't funny.", "Don't?"], language='en

Actual Behavior

TypeError: init() got an unexpected keyword argument 'language'

Expected Behavior

• I tried to follow example of pytorch nlp documentation with wmt14 dataset. (https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html)
• download wmt dataset successfully

Actual Behavior

• wmt_dataset [DOWNLOAD_FAILED] occurs.

Steps to Reproduce the Problem

1. install pytorch-nlp 0.5.0
2. from torchnlp.datasets import wmt_dataset
3. train=wmt_dataset(train=True)

>>> train = wmt_dataset(train=True)
tar: Error opening archive: Unrecognized archive format
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/torchnlp/datasets/wmt.py", line 63, in wmt_dataset
    download_file_maybe_extract(
  File "/usr/local/lib/python3.9/site-packages/torchnlp/download.py", line 170, in download_file_maybe_extract
    raise ValueError('[DOWNLOAD FAILED] `*check_files` not found')
ValueError: [DOWNLOAD FAILED] `*check_files` not found

Hello,

I was hoping to get some pointers related to the below query. I want to apply gating to my inputs using attention such that only important ones go forward in the network. While searching for inbuilt attention libraries in PyTorch, I came across this. But I having problems understanding what the inputs are (i.e context and query). Would really appreciate any pointers.

Thanks, Neha