You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by zh...@apache.org on 2018/03/15 17:21:41 UTC
[incubator-mxnet] branch nlp_toolkit updated (092df8f -> 9e4f9d1)
This is an automated email from the ASF dual-hosted git repository.
zhasheng pushed a change to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git.
discard 092df8f Add vocabulary and embedding (#10074)
discard 7e59d74 fix CorpusReader (#10079)
omit 453fc23 CorpusReader (#10061)
omit a61fd99 gluon language modeling dataset and text token reader (#9986)
add ccf0469 [MXNET-67] Sync master with v1.1.0 branch (#10031)
add 884408a Parallelization for ROIpooling OP (#9958)
add c153a6c comments to copy and copyto are corrected (#10040)
add a7ba921 Bug Fix and performance optimized for rtc (#10018)
add 85215b6 CI docker revamp; Add Jetson, Raspberry and CentOS 7 build [MXNET-42][MXNET-43] (#9995)
add 279ccb1 [MXNET-58]Layer Normalization in C++ (#10029)
add 17643e7 Update test_sparse_operator.py (#10050)
add bf8658b fixed calling error while preparing data in ssd (#10024)
add 13ae4d1 Cpu lstm inference (#9977)
add 1794a2a [MXNET-38]add reshape predicator function to c_predict_api (#9984)
add 94f68fc Fixes for profiler (#9932)
add 0bcf334 Adding back comments to index.md that cause nightly test to fail (#10058)
add a9c1717 [TENSOR] Fix DLTensor conversion for int64 (#10083)
add dd1f21b Support single array input for metric (#9930)
add c9ec311 [MXNET-86] Revert to pre-profile-changes copy code (#10090)
add 1dc01ef [MXNET-47] Documentation for build_version_doc scripts folder (#10010)
add faa4f9f [MXNET-83] Fix CMake build issue with MKL. (#10075)
add f202f10 [MXNET-74]Update mkldnn to the newest & Add clang build test with mkldnn. (#9918)
add 93e0ceb [MXNET-34] Onnx Module to import onnx models into mxnet (#9963)
add f7fa49e Compile MKLDNN in Mac. (#10115)
new 929f5ef gluon language modeling dataset and text token reader (#9986)
new 85f564e CorpusReader (#10061)
new 911ff3b fix CorpusReader (#10079)
new 9e4f9d1 Add vocabulary and embedding (#10074)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (092df8f)
\
N -- N -- N refs/heads/nlp_toolkit (9e4f9d1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
example/utils/__init__.py => .mxnet_root | 0
3rdparty/mkldnn | 2 +-
CMakeLists.txt | 15 +-
Jenkinsfile | 351 ++++++--------
LICENSE | 312 +++++++++++--
Makefile | 10 +-
NEWS.md | 41 ++
README.md | 1 +
ci/README.md | 50 ++
ci/build.py | 179 ++++++++
ci/docker/Dockerfile.build.amzn_linux_cpu | 44 ++
ci/docker/Dockerfile.build.android_arm64 | 66 +++
.../docker/Dockerfile.build.android_armv7 | 24 +-
.../Makefile => ci/docker/Dockerfile.build.arm64 | 27 +-
.../docker/Dockerfile.build.armv7 | 22 +-
.../docker/Dockerfile.build.centos7_cpu | 25 +-
.../docker/Dockerfile.build.centos7_gpu | 23 +-
.../docker}/Dockerfile.build.jetson | 74 ++-
ci/docker/Dockerfile.build.ubuntu_build_cuda | 57 +++
ci/docker/Dockerfile.build.ubuntu_cpu | 55 +++
ci/docker/Dockerfile.build.ubuntu_gpu | 55 +++
.../docker/install/amzn_linux_core.sh | 39 +-
.../docker/install/amzn_linux_julia.sh | 13 +-
.../docker/install/amzn_linux_library.sh | 11 +-
.../docker/install/amzn_linux_maven.sh | 13 +-
.../docker/install/amzn_linux_openblas.sh | 10 +-
.../docker/install/amzn_linux_opencv.sh | 19 +-
.../docker/install/amzn_linux_python2.sh | 15 +-
.../docker/install/amzn_linux_python3.sh | 18 +-
.../docker/install/amzn_linux_testdeps.sh | 11 +-
.../docker/install/android_arm64_ndk.sh | 31 +-
.../docker/install/android_arm64_openblas.sh | 17 +-
.../docker/install/arm64_openblas.sh | 18 +-
.../docker/install/centos7_adduser.sh | 27 +-
.../docker/install/centos7_core.sh | 35 +-
.../docker/install/centos7_python.sh | 22 +-
.../docker/install/ubuntu_adduser.sh | 27 +-
ci/docker/install/ubuntu_caffe.sh | 49 ++
.../setup.sh => ci/docker/install/ubuntu_clang.sh | 14 +-
.../docker/install/ubuntu_core.sh | 27 +-
.../docker/install/ubuntu_docs.sh | 24 +-
.../setup.sh => ci/docker/install/ubuntu_lint.sh | 8 +-
.../docker/install/ubuntu_llvm.sh | 4 +-
.../docker/install/ubuntu_mklml.sh | 10 +-
.../docker/install/ubuntu_nvidia.sh | 18 +-
.../docker/install/ubuntu_onnx.sh | 20 +-
.../docker/install/ubuntu_perl.sh | 10 +-
.../docker/install/ubuntu_python.sh | 16 +-
.../docker/install/ubuntu_r.sh | 12 +-
.../docker/install/ubuntu_scala.sh | 11 +-
.../docker/install/ubuntu_tvm.sh | 4 +-
ci/docker/runtime_functions.sh | 505 +++++++++++++++++++++
cpp-package/include/mxnet-cpp/ndarray.h | 12 +-
docker_multiarch/.gitignore | 2 -
docker_multiarch/Dockerfile.build.android.arm64 | 77 ----
docker_multiarch/Dockerfile.build.arm64 | 37 --
docker_multiarch/Dockerfile.build.armv6 | 38 --
docker_multiarch/Dockerfile.build.armv7 | 26 --
.../Dockerfile.build.cmake.ubuntu-17.04 | 37 --
.../Dockerfile.build.ubuntu-16.04-cuda_8.0_cudnn5 | 32 --
docker_multiarch/Dockerfile.build.ubuntu-17.04 | 30 --
.../Dockerfile.build.ubuntu-17.04.scala.docker | 17 -
docker_multiarch/Dockerfile.run.ubuntu-17.04.julia | 24 -
docker_multiarch/Dockerfile.run.ubuntu-17.04.perl | 23 -
.../Dockerfile.run.ubuntu-17.04.python | 17 -
docker_multiarch/Dockerfile.run.ubuntu-17.04.r | 36 --
docker_multiarch/Dockerfile.test.ubuntu-17.04 | 41 --
docker_multiarch/License.md | 20 -
docker_multiarch/README.md | 42 --
docker_multiarch/arm.crosscompile.android.mk | 161 -------
docker_multiarch/arm.crosscompile.mk | 161 -------
docker_multiarch/tool.py | 155 -------
docs/api/python/gluon/nn.md | 1 +
docs/api/python/ndarray/ndarray.md | 1 +
docs/api/python/symbol/symbol.md | 1 +
docs/build_version_doc/Dockerfile | 10 +-
docs/build_version_doc/README.md | 112 +++++
docs/build_version_doc/build_all_version.sh | 1 +
docs/install/index.md | 2 +
example/numpy-ops/custom_softmax_rtc.py | 131 +++---
example/onnx/super_resolution.py | 84 ++++
example/ssd/tools/prepare_coco.sh | 2 +-
example/ssd/tools/prepare_dataset.py | 16 +-
example/ssd/tools/prepare_pascal.sh | 2 +-
include/mxnet/c_predict_api.h | 21 +
include/mxnet/tensor_blob.h | 23 +-
prepare_mkldnn.sh | 19 +-
python/mxnet/contrib/__init__.py | 2 +-
.../mxnet/contrib/onnx/__init__.py | 6 +-
.../{ndarray.py => onnx/_import/__init__.py} | 5 +-
python/mxnet/contrib/onnx/_import/import_helper.py | 105 +++++
.../onnx/_import/import_model.py} | 34 +-
python/mxnet/contrib/onnx/_import/import_onnx.py | 168 +++++++
.../mxnet/contrib/onnx/_import/op_translations.py | 411 +++++++++++++++++
.../contrib/onnx/_import/translation_utils.py | 163 +++++++
python/mxnet/gluon/data/dataloader.py | 1 +
python/mxnet/gluon/nn/basic_layers.py | 99 +++-
python/mxnet/gluon/rnn/rnn_layer.py | 14 +-
python/mxnet/image/detection.py | 4 +-
python/mxnet/libinfo.py | 12 +
python/mxnet/metric.py | 59 ++-
src/c_api/c_predict_api.cc | 97 ++++
src/ndarray/ndarray_function.cc | 19 +-
src/operator/nn/batch_norm-inl.h | 2 +-
src/operator/nn/batch_norm.cc | 3 +-
src/operator/nn/layer_norm-inl.h | 296 ++++++++++++
src/operator/nn/layer_norm.cc | 148 ++++++
src/operator/nn/{lrn.cu => layer_norm.cu} | 18 +-
src/operator/nn/mkldnn/mkldnn_base.cc | 3 +-
src/operator/rnn-inl.h | 210 +++++++++
src/operator/rnn.cc | 1 -
src/operator/roi_pooling.cc | 48 +-
src/profiler/profiler.cc | 16 +-
src/profiler/profiler.h | 8 +-
tests/ci_build/Dockerfile.amzn_linux_cpu | 16 -
tests/ci_build/Dockerfile.build_cuda | 26 --
tests/ci_build/Dockerfile.caffe_gpu | 36 --
tests/ci_build/Dockerfile.cpu | 12 -
tests/ci_build/Dockerfile.cpu_clang | 21 -
tests/ci_build/Dockerfile.cpu_mklml | 18 -
tests/ci_build/Dockerfile.crosstool | 23 -
tests/ci_build/Dockerfile.doc | 15 -
tests/ci_build/Dockerfile.emscripten | 19 -
tests/ci_build/Dockerfile.gpu | 20 -
tests/ci_build/Dockerfile.gpu_mklml | 18 -
tests/ci_build/Dockerfile.lint | 5 -
tests/ci_build/Dockerfile.spell_checker | 6 -
tests/ci_build/Dockerfile.ubuntu1404_cuda75_cudnn5 | 40 --
tests/ci_build/README.md | 41 --
tests/ci_build/ci_build.sh | 188 --------
tests/ci_build/with_the_same_user | 47 --
tests/python-pytest/onnx/backend.py | 177 ++++++++
tests/python-pytest/onnx/backend_rep.py | 77 ++++
tests/python-pytest/onnx/onnx_backend_test.py | 116 +++++
tests/python-pytest/onnx/onnx_test.py | 136 ++++++
tests/python/cpu/test_mkldnn.py | 58 ---
tests/python/gpu/test_operator_gpu.py | 19 +-
tests/python/gpu/test_tvm_bridge.py | 21 +-
tests/python/unittest/test_gluon.py | 5 +
tests/python/unittest/test_gluon_data.py | 81 ++++
tests/python/unittest/test_gluon_rnn.py | 14 +
tests/python/unittest/test_metric.py | 21 +
tests/python/unittest/test_operator.py | 41 ++
tests/python/unittest/test_sparse_operator.py | 4 +-
tools/caffe_converter/compare_layers.py | 2 +-
tools/caffe_converter/convert_mean.py | 2 +-
tools/caffe_converter/convert_model.py | 2 +-
147 files changed, 4833 insertions(+), 2351 deletions(-)
copy example/utils/__init__.py => .mxnet_root (100%)
create mode 100644 ci/README.md
create mode 100755 ci/build.py
create mode 100755 ci/docker/Dockerfile.build.amzn_linux_cpu
create mode 100755 ci/docker/Dockerfile.build.android_arm64
rename docker_multiarch/Dockerfile.build.android.armv7 => ci/docker/Dockerfile.build.android_armv7 (74%)
mode change 100644 => 100755
copy cpp-package/example/Makefile => ci/docker/Dockerfile.build.arm64 (62%)
mode change 100644 => 100755
copy example/gluon/embedding_learning/get_cub200_data.sh => ci/docker/Dockerfile.build.armv7 (68%)
copy python/mxnet/contrib/__init__.py => ci/docker/Dockerfile.build.centos7_cpu (66%)
mode change 100644 => 100755
copy python/mxnet/gluon/data/__init__.py => ci/docker/Dockerfile.build.centos7_gpu (64%)
mode change 100644 => 100755
rename {docker_multiarch => ci/docker}/Dockerfile.build.jetson (52%)
mode change 100644 => 100755
create mode 100755 ci/docker/Dockerfile.build.ubuntu_build_cuda
create mode 100755 ci/docker/Dockerfile.build.ubuntu_cpu
create mode 100755 ci/docker/Dockerfile.build.ubuntu_gpu
copy scala-package/core/scripts/get_cifar_data.sh => ci/docker/install/amzn_linux_core.sh (60%)
rename tests/ci_build/install/install_julia.sh => ci/docker/install/amzn_linux_julia.sh (77%)
mode change 100644 => 100755
rename tests/ci_build/install/install_library.sh => ci/docker/install/amzn_linux_library.sh (81%)
mode change 100644 => 100755
rename tests/ci_build/install/install_maven.sh => ci/docker/install/amzn_linux_maven.sh (75%)
mode change 100644 => 100755
rename tests/ci_build/install/install_openblas.sh => ci/docker/install/amzn_linux_openblas.sh (86%)
mode change 100644 => 100755
rename tests/ci_build/install/install_opencv.sh => ci/docker/install/amzn_linux_opencv.sh (70%)
mode change 100644 => 100755
rename tests/ci_build/install/install_python2.sh => ci/docker/install/amzn_linux_python2.sh (83%)
mode change 100644 => 100755
rename tests/ci_build/install/install_python3.sh => ci/docker/install/amzn_linux_python3.sh (78%)
mode change 100644 => 100755
rename tests/ci_build/install/install_testdeps.sh => ci/docker/install/amzn_linux_testdeps.sh (82%)
mode change 100644 => 100755
copy scala-package/core/scripts/get_cifar_data.sh => ci/docker/install/android_arm64_ndk.sh (60%)
copy example/gluon/tree_lstm/fetch_and_preprocess.sh => ci/docker/install/android_arm64_openblas.sh (67%)
copy cpp-package/tests/travis/setup.sh => ci/docker/install/arm64_openblas.sh (58%)
copy scala-package/core/scripts/get_mnist_data.sh => ci/docker/install/centos7_adduser.sh (57%)
copy scala-package/core/scripts/get_cifar_data.sh => ci/docker/install/centos7_core.sh (59%)
copy tests/ci_build/install/ubuntu_install_python.sh => ci/docker/install/centos7_python.sh (65%)
copy scala-package/core/scripts/get_mnist_data.sh => ci/docker/install/ubuntu_adduser.sh (57%)
create mode 100755 ci/docker/install/ubuntu_caffe.sh
copy cpp-package/tests/travis/setup.sh => ci/docker/install/ubuntu_clang.sh (57%)
rename tests/ci_build/install/ubuntu_install_core.sh => ci/docker/install/ubuntu_core.sh (67%)
copy scala-package/core/scripts/get_mnist_data.sh => ci/docker/install/ubuntu_docs.sh (66%)
copy cpp-package/tests/travis/setup.sh => ci/docker/install/ubuntu_lint.sh (82%)
rename tests/ci_build/install/ubuntu_install_llvm.sh => ci/docker/install/ubuntu_llvm.sh (94%)
copy example/gluon/tree_lstm/fetch_and_preprocess.sh => ci/docker/install/ubuntu_mklml.sh (70%)
rename tests/ci_build/install/ubuntu_install_nvidia.sh => ci/docker/install/ubuntu_nvidia.sh (74%)
copy tests/ci_build/install/ubuntu_install_r.sh => ci/docker/install/ubuntu_onnx.sh (55%)
rename tests/ci_build/install/ubuntu_install_perl.sh => ci/docker/install/ubuntu_perl.sh (81%)
rename tests/ci_build/install/ubuntu_install_python.sh => ci/docker/install/ubuntu_python.sh (75%)
rename tests/ci_build/install/ubuntu_install_r.sh => ci/docker/install/ubuntu_r.sh (89%)
rename tests/ci_build/install/ubuntu_install_scala.sh => ci/docker/install/ubuntu_scala.sh (86%)
rename tests/ci_build/install/ubuntu_install_tvm.sh => ci/docker/install/ubuntu_tvm.sh (98%)
create mode 100755 ci/docker/runtime_functions.sh
delete mode 100644 docker_multiarch/.gitignore
delete mode 100644 docker_multiarch/Dockerfile.build.android.arm64
delete mode 100644 docker_multiarch/Dockerfile.build.arm64
delete mode 100644 docker_multiarch/Dockerfile.build.armv6
delete mode 100644 docker_multiarch/Dockerfile.build.armv7
delete mode 100644 docker_multiarch/Dockerfile.build.cmake.ubuntu-17.04
delete mode 100644 docker_multiarch/Dockerfile.build.ubuntu-16.04-cuda_8.0_cudnn5
delete mode 100644 docker_multiarch/Dockerfile.build.ubuntu-17.04
delete mode 100644 docker_multiarch/Dockerfile.build.ubuntu-17.04.scala.docker
delete mode 100644 docker_multiarch/Dockerfile.run.ubuntu-17.04.julia
delete mode 100644 docker_multiarch/Dockerfile.run.ubuntu-17.04.perl
delete mode 100644 docker_multiarch/Dockerfile.run.ubuntu-17.04.python
delete mode 100644 docker_multiarch/Dockerfile.run.ubuntu-17.04.r
delete mode 100644 docker_multiarch/Dockerfile.test.ubuntu-17.04
delete mode 100644 docker_multiarch/License.md
delete mode 100644 docker_multiarch/README.md
delete mode 100644 docker_multiarch/arm.crosscompile.android.mk
delete mode 100644 docker_multiarch/arm.crosscompile.mk
delete mode 100755 docker_multiarch/tool.py
create mode 100644 docs/build_version_doc/README.md
create mode 100644 example/onnx/super_resolution.py
copy perl-package/AI-MXNetCAPI/t/AI-MXNetCAPI.t => python/mxnet/contrib/onnx/__init__.py (88%)
copy python/mxnet/contrib/{ndarray.py => onnx/_import/__init__.py} (91%)
create mode 100644 python/mxnet/contrib/onnx/_import/import_helper.py
copy python/mxnet/{gluon/nn/__init__.py => contrib/onnx/_import/import_model.py} (50%)
create mode 100644 python/mxnet/contrib/onnx/_import/import_onnx.py
create mode 100644 python/mxnet/contrib/onnx/_import/op_translations.py
create mode 100644 python/mxnet/contrib/onnx/_import/translation_utils.py
create mode 100644 src/operator/nn/layer_norm-inl.h
create mode 100644 src/operator/nn/layer_norm.cc
copy src/operator/nn/{lrn.cu => layer_norm.cu} (73%)
delete mode 100644 tests/ci_build/Dockerfile.amzn_linux_cpu
delete mode 100644 tests/ci_build/Dockerfile.build_cuda
delete mode 100644 tests/ci_build/Dockerfile.caffe_gpu
delete mode 100644 tests/ci_build/Dockerfile.cpu
delete mode 100644 tests/ci_build/Dockerfile.cpu_clang
delete mode 100644 tests/ci_build/Dockerfile.cpu_mklml
delete mode 100644 tests/ci_build/Dockerfile.crosstool
delete mode 100644 tests/ci_build/Dockerfile.doc
delete mode 100644 tests/ci_build/Dockerfile.emscripten
delete mode 100644 tests/ci_build/Dockerfile.gpu
delete mode 100644 tests/ci_build/Dockerfile.gpu_mklml
delete mode 100644 tests/ci_build/Dockerfile.lint
delete mode 100644 tests/ci_build/Dockerfile.spell_checker
delete mode 100644 tests/ci_build/Dockerfile.ubuntu1404_cuda75_cudnn5
delete mode 100644 tests/ci_build/README.md
delete mode 100755 tests/ci_build/ci_build.sh
delete mode 100755 tests/ci_build/with_the_same_user
create mode 100644 tests/python-pytest/onnx/backend.py
create mode 100644 tests/python-pytest/onnx/backend_rep.py
create mode 100644 tests/python-pytest/onnx/onnx_backend_test.py
create mode 100644 tests/python-pytest/onnx/onnx_test.py
delete mode 100644 tests/python/cpu/test_mkldnn.py
--
To stop receiving notification emails like this one, please contact
zhasheng@apache.org.
[incubator-mxnet] 04/04: Add vocabulary and embedding (#10074)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
commit 9e4f9d18ec6827799f5b787f4c0697df7c017ef2
Author: Aston Zhang <22...@users.noreply.github.com>
AuthorDate: Thu Mar 15 09:39:05 2018 -0700
Add vocabulary and embedding (#10074)
* [MXNET-67] Sync master with v1.1.0 branch (#10031)
* [REVIEW REQUIRED] Revert PR #9484 & add additional dependency licenses to LICENSE file (#9701)
* Revert "[Review Required] Fixing Licenses: Cleaning up the Top Level LICENSE file (#9484)"
This reverts commit 8930d96b265560a797c5554a9617f607cea7740f.
* Some more LICENSE fixes
* Adding some more packages to the LICENSE file
* Adding dependencies of dependencies
* update v1.1.0 change log to NEWS.md
* sync README.md from v1.1.0 branch
* revert to correct jenkins url in README
* Parallelization for ROIpooling OP (#9958)
* parallelization for roipooling
* remove some useless computation
* remove useless muls
* add author and retriggering
* retrigger again
* comments to copy and copyto are corrected (#10040)
* Bug Fix and performance optimized for rtc (#10018)
* Bug Fix and performance optimized for rtc
1. "super().__init__()" bug is fixed in python 2.
2. Kernel is initialized in the stage of operator init.
* Update custom_softmax_rtc.py
fix unnessesary format
* set embedding
* Code and test revised
* api implementation done
* license and news
* readme and cpp
* pylint disable
* Add API doc
* less pylint disable
* remove contrib
* move to gluon, revise api doc
* fix import order
* re-test
* relative imports
* re-run test
* revise implementation, test case, and api doc
* re-test
---
docs/api/python/gluon/text.md | 347 +++++++++++++++
python/mxnet/gluon/text/__init__.py | 26 ++
python/mxnet/gluon/text/_constants.py | 704 +++++++++++++++++++++++++++++++
python/mxnet/gluon/text/embedding.py | 581 +++++++++++++++++++++++++
python/mxnet/gluon/text/utils.py | 77 ++++
python/mxnet/gluon/text/vocab.py | 323 ++++++++++++++
tests/python/unittest/test_gluon_text.py | 681 ++++++++++++++++++++++++++++++
7 files changed, 2739 insertions(+)
diff --git a/docs/api/python/gluon/text.md b/docs/api/python/gluon/text.md
new file mode 100644
index 0000000..f9fcf33
--- /dev/null
+++ b/docs/api/python/gluon/text.md
@@ -0,0 +1,347 @@
+# Gluon Text API
+
+## Overview
+
+The `mxnet.gluon.text` APIs refer to classes and functions related to text data processing, such
+as bulding indices and loading pre-trained embedding vectors for text tokens and storing them in the
+`mxnet.ndarray.NDArray` format.
+
+This document lists the text APIs in `mxnet.gluon`:
+
+```eval_rst
+.. autosummary::
+ :nosignatures:
+
+ mxnet.gluon.text.embedding
+ mxnet.gluon.text.vocab
+ mxnet.gluon.text.utils
+```
+
+All the code demonstrated in this document assumes that the following modules or packages are
+imported.
+
+```python
+>>> from mxnet import gluon
+>>> from mxnet import nd
+>>> from mxnet.gluon import text
+>>> import collections
+
+```
+
+### Indexing words and using pre-trained word embeddings in `gluon`
+
+As a common use case, let us index words, attach pre-trained word embeddings for them, and use
+such embeddings in `gluon` in just a few lines of code.
+
+To begin with, suppose that we have a simple text data set in the string format. We can count word
+frequency in the data set.
+
+```python
+>>> text_data = " hello world \n hello nice world \n hi world \n"
+>>> counter = text.count_tokens_from_str(text_data)
+
+```
+
+The obtained `counter` has key-value pairs whose keys are words and values are word frequencies.
+This allows us to filter out infrequent words (See details at
+[Vocabulary API specifications](#mxnet.gluon.text.vocab.Vocabulary)).
+Suppose that we want to build indices for all the keys in `counter`. We need a Vocabulary instance
+with `counter` as its argument.
+
+```python
+>>> my_vocab = text.Vocabulary(counter)
+
+```
+
+To attach word embedding to indexed words in `my_vocab`, let us go on to create a fastText word
+embedding instance by specifying the embedding name `fasttext` and the pre-trained file name
+`wiki.simple.vec`.
+
+```python
+>>> fasttext = text.embedding.create('fasttext', file_name='wiki.simple.vec')
+
+```
+
+So we can attach word embedding `fasttext` to indexed words `my_vocab`.
+
+```python
+>>> my_vocab.set_embedding(fasttext)
+
+```
+
+Now we are ready to access the fastText word embedding vectors for indexed words, such as 'hello'
+and 'world'.
+
+```python
+>>> my_vocab.embedding[['hello', 'world']]
+
+[[ 3.95669997e-01 2.14540005e-01 -3.53889987e-02 -2.42990002e-01
+ ...
+ -7.54180014e-01 -3.14429998e-01 2.40180008e-02 -7.61009976e-02]
+ [ 1.04440004e-01 -1.08580001e-01 2.72119999e-01 1.32990003e-01
+ ...
+ -3.73499990e-01 5.67310005e-02 5.60180008e-01 2.90190000e-02]]
+<NDArray 2x300 @cpu(0)>
+
+```
+
+To demonstrate how to use pre-trained word embeddings in the `gluon` package, let us first obtain
+indices of the words 'hello' and 'world'.
+
+```python
+>>> my_vocab[['hello', 'world']]
+[2, 1]
+
+```
+
+We can obtain the vector representation for the words 'hello' and 'world' by specifying their
+indices (2 and 1) and the weight matrix `my_vocab.embedding.idx_to_vec` in
+`mxnet.gluon.nn.Embedding`.
+
+```python
+>>> input_dim, output_dim = my_vocab.embedding.idx_to_vec.shape
+>>> layer = gluon.nn.Embedding(input_dim, output_dim)
+>>> layer.initialize()
+>>> layer.weight.set_data(my_vocab.embedding.idx_to_vec)
+>>> layer(nd.array([2, 1]))
+
+[[ 3.95669997e-01 2.14540005e-01 -3.53889987e-02 -2.42990002e-01
+ ...
+ -7.54180014e-01 -3.14429998e-01 2.40180008e-02 -7.61009976e-02]
+ [ 1.04440004e-01 -1.08580001e-01 2.72119999e-01 1.32990003e-01
+ ...
+ -3.73499990e-01 5.67310005e-02 5.60180008e-01 2.90190000e-02]]
+<NDArray 2x300 @cpu(0)>
+
+```
+
+## Vocabulary
+
+The vocabulary builds indices for text tokens and can be attached with token embeddings. The input
+counter whose keys are candidate indices may be obtained via
+[`count_tokens_from_str`](#mxnet.gluon.text.utils.count_tokens_from_str).
+
+
+```eval_rst
+.. currentmodule:: mxnet.gluon.text.vocab
+.. autosummary::
+ :nosignatures:
+
+ Vocabulary
+```
+
+Suppose that we have a simple text data set in the string format. We can count word frequency in the
+data set.
+
+```python
+>>> text_data = " hello world \n hello nice world \n hi world \n"
+>>> counter = text.utils.count_tokens_from_str(text_data)
+
+```
+
+The obtained `counter` has key-value pairs whose keys are words and values are word frequencies.
+This allows us to filter out infrequent words. Suppose that we want to build indices for the 2 most
+frequent keys in `counter` with the unknown token representation '(unk)' and a reserved token
+'(pad)'.
+
+```python
+>>> my_vocab = text.Vocabulary(counter, max_size=2, unknown_token='(unk)',
+... reserved_tokens=['(pad)'])
+
+```
+
+We can access properties such as `token_to_idx` (mapping tokens to indices), `idx_to_token` (mapping
+indices to tokens), `unknown_token` (representation of any unknown token) and `reserved_tokens`
+(reserved tokens).
+
+
+```python
+>>> my_vocab.token_to_idx
+{'(unk)': 0, '(pad)': 1, 'world': 2, 'hello': 3}
+>>> my_vocab.idx_to_token
+['(unk)', '(pad)', 'world', 'hello']
+>>> my_vocab.unknown_token
+'(unk)'
+>>> my_vocab.reserved_tokens
+['(pad)']
+>>> len(my_vocab)
+4
+>>> my_vocab[['hello', 'world']]
+[3, 2]
+```
+
+Besides the specified unknown token '(unk)' and reserved_token '(pad)' are indexed, the 2 most
+frequent words 'world' and 'hello' are also indexed.
+
+
+### Attach token embedding to vocabulary
+
+A vocabulary instance can be attached with token embedding.
+
+To begin with, suppose that we have a simple text data set in the string format. We can count word
+frequency in the data set.
+
+```python
+>>> text_data = " hello world \n hello nice world \n hi world \n"
+>>> counter = text.count_tokens_from_str(text_data)
+
+```
+
+The obtained `counter` has key-value pairs whose keys are words and values are word frequencies.
+This allows us to filter out infrequent words.
+Suppose that we want to build indices for the most frequent 2 keys in `counter`.
+
+```python
+>>> my_vocab = text.Vocabulary(counter, max_size=2)
+
+```
+
+Let us define the fastText word embedding instance with the pre-trained file `wiki.simple.vec`.
+
+```python
+>>> fasttext = text.embedding.create('fasttext', file_name='wiki.simple.vec')
+
+```
+
+So we can attach word embedding `fasttext` to indexed words `my_vocab`.
+
+```python
+>>> my_vocab.set_embedding(fasttext)
+
+```
+
+Now we are ready to access the fastText word embedding vectors for the indexed words.
+
+```python
+>>> my_vocab.embedding[['hello', 'world']]
+
+[[ 3.95669997e-01 2.14540005e-01 -3.53889987e-02 -2.42990002e-01
+ ...
+ -7.54180014e-01 -3.14429998e-01 2.40180008e-02 -7.61009976e-02]
+ [ 1.04440004e-01 -1.08580001e-01 2.72119999e-01 1.32990003e-01
+ ...
+ -3.73499990e-01 5.67310005e-02 5.60180008e-01 2.90190000e-02]]
+<NDArray 2x300 @cpu(0)>
+
+```
+
+Let us define the GloVe word embedding with the pre-trained file `glove.6B.50d.txt`. Then,
+we can re-attach a GloVe text embedding instance to the vocabulary.
+
+```python
+>>> glove = text.embedding.create('glove', file_name='glove.6B.50d.txt')
+>>> my_vocab.set_embedding(glove)
+
+```
+
+Now we are ready to access the GloVe word embedding vectors for the indexed words.
+
+```python
+>>> my_vocab.embedding[['hello', 'world']]
+
+[[ -0.38497001 0.80092001
+ ...
+ 0.048833 0.67203999]
+ [ -0.41486001 0.71847999
+ ...
+ -0.37639001 -0.67541999]]
+<NDArray 2x50 @cpu(0)>
+
+```
+
+If a token is unknown to `my_vocab`, its embedding vector is initialized according to the default
+specification in `glove` (all elements are 0).
+
+```python
+
+>>> my_vocab.embedding['nice']
+
+[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
+ ...
+ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
+<NDArray 50 @cpu(0)>
+
+```
+
+
+
+## Text token embedding
+
+To load token embeddings from an externally hosted pre-trained token embedding file, such as those
+of GloVe and FastText, use
+[`embedding.create(embedding_name, file_name)`](#mxnet.gluon.text.embedding.create).
+
+To get all the available `embedding_name` and `file_name`, use
+[`embedding.get_file_names()`](#mxnet.gluon.text.embedding.get_file_names).
+
+```python
+>>> text.embedding.get_file_names()
+{'glove': ['glove.42B.300d.txt', 'glove.6B.50d.txt', 'glove.6B.100d.txt', ...],
+'fasttext': ['wiki.en.vec', 'wiki.simple.vec', 'wiki.zh.vec', ...]}
+
+```
+
+Alternatively, to load embedding vectors from a custom pre-trained text token embedding file, use
+[`TokenEmbedding.from_file`](#mxnet.gluon.text.embedding.TokenEmbedding.from_file).
+
+
+```eval_rst
+.. currentmodule:: mxnet.gluon.text.embedding
+.. autosummary::
+ :nosignatures:
+
+ register
+ create
+ get_file_names
+ TokenEmbedding
+ GloVe
+ FastText
+```
+
+See [Assign token embedding to vocabulary](#assign-token-embedding-to-vocabulary) for how to attach
+token embeddings to vocabulary and use token embeddings.
+
+
+### Implement a new text token embedding
+
+For ``embedding``, create a subclass of `mxnet.gluon.text.embedding.TokenEmbedding`.
+Also add ``@mxnet.gluon.text.embedding.TokenEmbedding.register`` before this class. See
+[`embedding.py`](https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/text/embedding.py)
+for examples.
+
+
+## Text utilities
+
+The following functions provide utilities for text data processing.
+
+```eval_rst
+.. currentmodule:: mxnet.gluon.text.utils
+.. autosummary::
+ :nosignatures:
+
+ count_tokens_from_str
+```
+
+
+## API Reference
+
+<script type="text/javascript" src='../../_static/js/auto_module_index.js'></script>
+
+```eval_rst
+
+.. automodule:: mxnet.gluon.text.embedding
+ :members: register, create, get_file_names
+.. autoclass:: mxnet.gluon.text.embedding.TokenEmbedding
+ :members: from_file
+.. autoclass:: mxnet.gluon.text.embedding.GloVe
+.. autoclass:: mxnet.gluon.text.embedding.FastText
+
+.. automodule:: mxnet.gluon.text.vocab
+.. autoclass:: mxnet.gluon.text.vocab.Vocabulary
+ :members: set_embedding, to_tokens
+
+.. automodule:: mxnet.gluon.text.utils
+ :members: count_tokens_from_str
+
+```
+<script>auto_index("api-reference");</script>
\ No newline at end of file
diff --git a/python/mxnet/gluon/text/__init__.py b/python/mxnet/gluon/text/__init__.py
new file mode 100644
index 0000000..3c33272
--- /dev/null
+++ b/python/mxnet/gluon/text/__init__.py
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=wildcard-import
+"""This module includes utilities for indexing and embedding text."""
+
+from .vocab import *
+
+from . import embedding
+
+from .utils import *
diff --git a/python/mxnet/gluon/text/_constants.py b/python/mxnet/gluon/text/_constants.py
new file mode 100644
index 0000000..3457a29
--- /dev/null
+++ b/python/mxnet/gluon/text/_constants.py
@@ -0,0 +1,704 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+
+"""Constants."""
+from __future__ import absolute_import
+from __future__ import print_function
+
+UNKNOWN_IDX = 0
+
+GLOVE_PRETRAINED_FILE_SHA1 = \
+ {'glove.42B.300d.zip': 'f8e722b39578f776927465b71b231bae2ae8776a',
+ 'glove.6B.zip': 'b64e54f1877d2f735bdd000c1d7d771e25c7dfdc',
+ 'glove.840B.300d.zip': '8084fbacc2dee3b1fd1ca4cc534cbfff3519ed0d',
+ 'glove.twitter.27B.zip': 'dce69c404025a8312c323197347695e81fd529fc'}
+
+GLOVE_PRETRAINED_ARCHIVE_SHA1 = \
+ {'glove.42B.300d.txt': '876767977d6bd4d947c0f84d44510677bc94612a',
+ 'glove.6B.50d.txt': '21bf566a9d27f84d253e0cd4d4be9dcc07976a6d',
+ 'glove.6B.100d.txt': '16b1dbfaf35476790bd9df40c83e2dfbd05312f1',
+ 'glove.6B.200d.txt': '17d0355ddaa253e298ede39877d1be70f99d9148',
+ 'glove.6B.300d.txt': '646443dd885090927f8215ecf7a677e9f703858d',
+ 'glove.840B.300d.txt': '294b9f37fa64cce31f9ebb409c266fc379527708',
+ 'glove.twitter.27B.25d.txt':
+ '767d80889d8c8a22ae7cd25e09d0650a6ff0a502',
+ 'glove.twitter.27B.50d.txt':
+ '9585f4be97e286339bf0112d0d3aa7c15a3e864d',
+ 'glove.twitter.27B.100d.txt':
+ '1bbeab8323c72332bd46ada0fc3c99f2faaa8ca8',
+ 'glove.twitter.27B.200d.txt':
+ '7921c77a53aa5977b1d9ce3a7c4430cbd9d1207a'}
+
+FAST_TEXT_ARCHIVE_SHA1 = \
+ {'crawl-300d-2M.zip': 'bb40313d15837ceecc1e879bc954e9be04b17c3c',
+ 'wiki.aa.zip': '0d85feb259e17d5258f38b2b615a2b87cd628427',
+ 'wiki.ab.zip': '7a8c555b9cf3837c9b31c901e9e0142209990365',
+ 'wiki.ace.zip': '51555fccbe53b726f6c86a84d704c026a78dd02f',
+ 'wiki.ady.zip': '725d2c30c03001c941ac4084549c55c7f8e1d766',
+ 'wiki.af.zip': '1a18d34e1b60433b837f5850750a44ca3845323d',
+ 'wiki.ak.zip': 'daecc2303cfd05bc6c33b24d78c14e0d7f33e3a7',
+ 'wiki.als.zip': '38851192e0b556e566be6c3c93370abf9867e525',
+ 'wiki.am.zip': '4576e0121448564b07f448e05e287236343f17c1',
+ 'wiki.ang.zip': '9c03da3b06d4becef5d387b9a61438b9362fc36a',
+ 'wiki.an.zip': '170f60bdd161cf8e4b5e018acd7d36e8bfc457a6',
+ 'wiki.arc.zip': 'c8dad8b00865bf736b087e7b323999ab404bda29',
+ 'wiki.ar.zip': '34e9869daa463fdc5609040ff33a03e67512e9fd',
+ 'wiki.arz.zip': '2d2790e11e401d46e1bce2970ee5264d5678a32b',
+ 'wiki.ast.zip': '1136515e2de556c077324bcd42ffe7f40c8d94c6',
+ 'wiki.as.zip': 'f9efde3e4ccda4a1e93fa275a3210f74036e9e46',
+ 'wiki.av.zip': '9f8568a3e094a48de4a3b6bea3bdb6fd7e875a08',
+ 'wiki.ay.zip': 'f09a422cedc6a0f15fbf30d290febe8057de83db',
+ 'wiki.azb.zip': 'd8895581050b9fdb5a10dfec3e27910a150b6faf',
+ 'wiki.az.zip': '2a34c2db872597ba3e345ce8b7db138241f9efbf',
+ 'wiki.bar.zip': 'd6e40135a6f4ba7a07fab11633034eccb1b05d0a',
+ 'wiki.bat_smg.zip': '5d08bd04f0515a36723776c0682b3de0f11d4264',
+ 'wiki.ba.zip': '412ac2f3bf9a605e56e2b0990bb0baed41ddf3b0',
+ 'wiki.bcl.zip': 'd3717cda357e08390cb57a64e07f5c7b7768d5be',
+ 'wiki.be.zip': 'b691e63b8080af23cc37f5f2b21b3154e464c425',
+ 'wiki.bg.zip': '08509a510a95e2a8905c19d83faf40d614d2268b',
+ 'wiki.bh.zip': 'a812600c6454b779d442b7680e3867e15d895095',
+ 'wiki.bi.zip': 'd0d4a3f57419424815f77b3951ef9c7336f6adf5',
+ 'wiki.bjn.zip': '0d81879ff7611380896eac6059bb677a5b3fe308',
+ 'wiki.bm.zip': 'f3a2a1a8dbc94973a74343c059595a310a66665b',
+ 'wiki.bn.zip': 'b3bc70520edf3963c2217873ff5c2537d3545650',
+ 'wiki.bo.zip': '2be9fe7701d6a8501461df7bd98fee26859cf83a',
+ 'wiki.bpy.zip': 'd44b9267bb4f86e3e43972a6a952cc0ccf90dd3c',
+ 'wiki.br.zip': '4bfa66f1ea5aa5cad736eccaa211f6025596bcd6',
+ 'wiki.bs.zip': '40c560c5994ab50485d08eeaffd88740f30236ab',
+ 'wiki.bug.zip': 'bc7cd87bb067ac477000259cd4f95f45bfb6e4df',
+ 'wiki.bxr.zip': '8396fd67ef53f3123540766788a0db54734c4f1a',
+ 'wiki.ca.zip': '8f5d3caf0f5d223b2771ec44f7e620e396974fb2',
+ 'wiki.cbk_zam.zip': '0af3be50823b564433455d10c8753df88461458f',
+ 'wiki.cdo.zip': '19024215aa0c13872c027fc6127b5d7506198b5f',
+ 'wiki.ceb.zip': '96374428bf36a43983ba4307d7f6fb5ab52a6c6a',
+ 'wiki.ce.zip': 'b27f1a8da448bc9315e15d4261519c64f00de8eb',
+ 'wiki.cho.zip': '20944e34c2b58f14adb849dd5a6f5168c7affdea',
+ 'wiki.chr.zip': 'b7f41ee3fa76e933e0b5ad6b793c507fc19afe98',
+ 'wiki.chy.zip': '4ef66004a609c724fd7d8aab2877f7634323d43f',
+ 'wiki.ch.zip': '7f73678b685c9b5f5d6eea9bc00322cfc18d40cb',
+ 'wiki.ckb.zip': 'b7db2805526ad8bed878af257b32ca9ba814855f',
+ 'wiki.co.zip': '1b9e19b11763cb87ca00520dbdd6ada565547c9c',
+ 'wiki.crh.zip': '792003bae25c4471d25721440002c983fa5af020',
+ 'wiki.cr.zip': '875e4aa0de8a829e57f6c8e13d43cac5103210de',
+ 'wiki.csb.zip': 'fa776014c4c83487d7cb2485bd08eaf6739d9dca',
+ 'wiki.cs.zip': 'dca18cb80460522cd281ccc3c9922cf2b3c08b81',
+ 'wiki.cu.zip': 'ed23b48ba3193181a358d7a73005afa7655a4fc3',
+ 'wiki.cv.zip': '27ccd50942c9c218e00365ee293fa0c3087a7646',
+ 'wiki.cy.zip': '78940d5be2969b82c99f785bda2ac5f4e18e149c',
+ 'wiki.da.zip': 'a45077d9d73328bd6a96efdba1b31ed9a3639dcd',
+ 'wiki.de.zip': '0d9e4bf80100b46237dcb73cfefe390103e7e827',
+ 'wiki.diq.zip': '0eef7d9e2f0ce3f100a22dc8fcede9449e466528',
+ 'wiki.dsb.zip': '903cd80550931effba1d4e52a19c22592837d11c',
+ 'wiki.dv.zip': '3fa06719641ff33ac8a5439d330a8108521da1e7',
+ 'wiki.dz.zip': '8bf3937971c3c996493c30b264cb8268627d7bd6',
+ 'wiki.ee.zip': 'e66bc50013d884fe69f4f67ba44af2e34fe97927',
+ 'wiki.el.zip': '3015f358036658fb126d42fa794d67a90c5b91ad',
+ 'wiki.eml.zip': '5be541be6115af5914ac2b8118a09232b771123b',
+ 'wiki.en.zip': '7f83d578a31a8168423c77ea25ad381494a5e920',
+ 'wiki.eo.zip': 'e7612df98c37cb872f0edc3c3e21dcd2f80a4d69',
+ 'wiki.es.zip': '1b7668b23db26810ea433173ce0c11281e801f74',
+ 'wiki.et.zip': 'aa31004e7b8ebf359e166b8ea6b8e6f77fac190f',
+ 'wiki.eu.zip': '8d7699451cbac4d69750caa8d58b4740cc72e0ca',
+ 'wiki.ext.zip': '3aeb4d77c48eb503b26ceb2a76a0a7d841124a71',
+ 'wiki.fa.zip': '08b6e805c8623fba526143d46f4685549c4380a6',
+ 'wiki.ff.zip': '64f690eda733a6fb4f794e42eb6ff05f09ec1d38',
+ 'wiki.fiu_vro.zip': '35c3fdcec0f0dc1ce303212967ea59936641daee',
+ 'wiki.fi.zip': '252299a2a59cc0ac07ba25f9458afc26bbac669f',
+ 'wiki.fj.zip': '004d1279c27324d02b961341cf0d6ee06dbe8966',
+ 'wiki.fo.zip': '12f1d6360d4867cdebcc93be87c024a4709d1af5',
+ 'wiki.frp.zip': '8a0f636b5440a9aab38014efada9edfdf94150d5',
+ 'wiki.frr.zip': '7c9e7b8109b98aa39b303dd77d837b37e96d4113',
+ 'wiki.fr.zip': 'd906e68760153d771e5982009b0150e913254b2d',
+ 'wiki.fur.zip': 'd5d2ae08696ed074a581eac563a60eb85467a792',
+ 'wiki.fy.zip': '342609d29882fae0a3b402d8ea1478606be0d93b',
+ 'wiki.gag.zip': 'f2b91f89dd9b9a1301727476f7823b7260b5f129',
+ 'wiki.gan.zip': 'd3ad3c1151555266e1feb9f98b066ee31ee5f410',
+ 'wiki.ga.zip': '798b0c26783c7af05d9c4f899ca9fddafeb1e0a1',
+ 'wiki.gd.zip': '49085fa182a528bdc51f10e99bef33c88c1e3112',
+ 'wiki.glk.zip': '9e16727ffcc691483b69ecbcd331b1df2efa4bcd',
+ 'wiki.gl.zip': 'c71c7e6601b2cbdc7930982fbeea636deddd107d',
+ 'wiki.gn.zip': '493ccb583211217ccd23e0a43f42ba773bd94f78',
+ 'wiki.gom.zip': '45bbd49750ddb7df5afe01fcfd5dda2958934dfa',
+ 'wiki.got.zip': '669d018f72827fb965e5ef37e224e21f4682b2e5',
+ 'wiki.gu.zip': '4afe874f7830f693e9f83508fc3fb444b33aebdf',
+ 'wiki.gv.zip': '9411197eebc07775949d9bb6e440780a68502a5c',
+ 'wiki.hak.zip': 'cd1e14bd5d50fa764883b148bda5b821375531e0',
+ 'wiki.haw.zip': 'cacd4eb4e476bdd842e8014764b8ae380b346ed2',
+ 'wiki.ha.zip': '14acc50950b451f40fe028fd08d042af44732398',
+ 'wiki.he.zip': 'a9e2cd13bc2e55d83820c529bac1f518a7198bc0',
+ 'wiki.hif.zip': 'dcdd488239deb0ede807cff263ddc972009c21f5',
+ 'wiki.hi.zip': '15899ec17985bc0e1db1df497e1b4a51bba1982b',
+ 'wiki.ho.zip': 'fde454bb4f3841ea5dde2bbf879138305a4d0b36',
+ 'wiki.hr.zip': 'f5d33ba967f7c56538fa9f5f0093f6d634e9db44',
+ 'wiki.hsb.zip': '64dc13c7645d2b65b8ba252bd8dfb1c616e8923a',
+ 'wiki.ht.zip': 'cf50a5cadcf91aba9ab58d095d65f348e2375d12',
+ 'wiki.hu.zip': 'b27f293caedf81a2d09204b11f52a7c8d7443643',
+ 'wiki.hy.zip': '641b8666bc2168998989fae1b20a09d3428766bb',
+ 'wiki.hz.zip': '1639f9f096de6fac84336a784a391ce73e523d62',
+ 'wiki.ia.zip': '37640aaf8a25c02883190951337b5a6f0157d781',
+ 'wiki.id.zip': '56ee0c7a38a6d232706932493eaa37b2a87667ee',
+ 'wiki.ie.zip': '7c3a5d7f96c801570e2305f45a40d401fcc038b9',
+ 'wiki.ig.zip': '405ebc2e8a959163c9f2f8dd015a0bcefd440111',
+ 'wiki.ii.zip': '1ec1c7d95d61eeca2dbbd8e432caf88524aaf28e',
+ 'wiki.ik.zip': 'e9d088c0d8d0ab420d6d0469c6a0fdb668f1833c',
+ 'wiki.ilo.zip': 'cbc9754978ce55e86da2eb3db20579f4a1f19947',
+ 'wiki.io.zip': '9e5ab1fd5c4f1094d111f501129e0eecccec69a0',
+ 'wiki.is.zip': '0744e63636cf794e0a406c922827628a3dd415b7',
+ 'wiki.it.zip': '29f4eb6a5d7dcf45b02b4d08a4a70dfae4c41200',
+ 'wiki.iu.zip': 'fb2e8de825d554257768d363a3a09f711afb001b',
+ 'wiki.jam.zip': '077cfb6de9d025aee4a5b2ea9ce15ada02f10a4f',
+ 'wiki.ja.zip': '7940f6c2bc490c04902f0faf0562b92cae7136bf',
+ 'wiki.jbo.zip': '3d086b6c9a369f197516cd0dc699a94612f45c6a',
+ 'wiki.jv.zip': '2f68cb3436b27a25ddfa40fab3e2cd44574b437e',
+ 'wiki.kaa.zip': '9fd5df362b7cb615f2267084d8b3fb8608be2693',
+ 'wiki.kab.zip': '96abf1440ad21de58d7274d3a16885ef4a2efda4',
+ 'wiki.ka.zip': '72ddb2382c87184fc05a93e89ed8aa4f54a62a0a',
+ 'wiki.kbd.zip': '81dfc3c6f8581c2aa15342c84688b4ba59b81cc6',
+ 'wiki.kg.zip': '4d07cabef6f804fc6432d3f630675ed4cbbdd49e',
+ 'wiki.ki.zip': '59b5c31df227ff9454ad8b3a1d16b065620dbddf',
+ 'wiki.kj.zip': '751b80c4a4d82dd217d3d2b3905eb39b349874d7',
+ 'wiki.kk.zip': '7fb733a2405f421a7c49b756381a52965a8af205',
+ 'wiki.kl.zip': '05a9d5c9bf12d8845356f88b546418d2e40f79c6',
+ 'wiki.km.zip': 'da0a67028fa0244a2e7257ae259c2f7a7544dc66',
+ 'wiki.kn.zip': '6cead946350b31fb2f353085fd00b8ea9c9ecc77',
+ 'wiki.koi.zip': '0c61f83434404267527eaf583e89b4d8bb3a6a65',
+ 'wiki.ko.zip': 'c0825282faf1e7af6820bd8b28d06c77760dcbe4',
+ 'wiki.krc.zip': '0df3c3f0f89521299dab741be3d698b2c94c194e',
+ 'wiki.kr.zip': '71651f046cef420fb28ca15e35720bb7747c4586',
+ 'wiki.ksh.zip': '8b9ab88baa49e72e40a5a80bef98f3ea2afbdd07',
+ 'wiki.ks.zip': '02af37f12753662c9e7bcac3b8786dfd2f298710',
+ 'wiki.ku.zip': 'ca1d370b327ceca025884bf83139456024a3a978',
+ 'wiki.kv.zip': '28b3617c5566f3182f14bf11a906456b227840ba',
+ 'wiki.kw.zip': '075a02e8eaae26897c23898fb4d36f4e41e4d1d0',
+ 'wiki.ky.zip': '771601a934cd4d0a98e5059f6389d2496e8dcf7c',
+ 'wiki.lad.zip': '2788ba3f275d72299e877c96cde106bd8590f405',
+ 'wiki.la.zip': '759f6365874442ab8e04d992b047f53ad74231a6',
+ 'wiki.lbe.zip': 'c8105f1cf8a3d46ccfacff1d40a581f442b3c4a1',
+ 'wiki.lb.zip': 'dac5af52364f2c0d3a0c794411465d1254f2fb48',
+ 'wiki.lez.zip': '17331cb779dee8cb60f2734213af80d57acfcfad',
+ 'wiki.lg.zip': 'fd4e2d67d1f098474053abc9a1984dfe4a2854b7',
+ 'wiki.lij.zip': 'c29157f5e4d2b37c01cf6e389f03ddafef6acdb2',
+ 'wiki.li.zip': '10490e49a12230af2127543da69c427f92c6508f',
+ 'wiki.lmo.zip': 'cc44163572deddd78af6b006394f623cb21934fc',
+ 'wiki.ln.zip': 'bf52699c5cbf79bedb2e2856d8a720189b6864f3',
+ 'wiki.lo.zip': '3fd8a70d8e26071a365f10016875a4a4f15ffcee',
+ 'wiki.lrc.zip': 'e262b4fcc55cba48d997cd06d006b82a5abe09a9',
+ 'wiki.ltg.zip': 'df6a83f2fab35f9a2f97fd8d857cb1cfa59f331f',
+ 'wiki.lt.zip': 'a738a3f29a6a5481082a7a9a41b2040b9cf537e4',
+ 'wiki.lv.zip': '8e328d99aacaa021fcc51425caebc063e22e6cf4',
+ 'wiki.mai.zip': 'e909de86c27eced2cb5f02f550da7fc2502b5eda',
+ 'wiki.map_bms.zip': '192bf6b88f955746abb398893868482730585e3a',
+ 'wiki.mdf.zip': '3d0d5da3c85bef8ae52f0fd17e314a1960a26d36',
+ 'wiki.mg.zip': 'fe66055b63ce8771bf43f8dd543bbd967f8ea8b3',
+ 'wiki.mhr.zip': '33514c98da3bd9602851db96fa3dd8192aac0674',
+ 'wiki.mh.zip': 'dc77309103c6cfed7ff095b3f9f158e1ae437e71',
+ 'wiki.min.zip': '8b925eea6df0411ee09baef5801d807cfec8cfa4',
+ 'wiki.mi.zip': 'd57831e8d7cb2ec260fc9d83d4281f0bacfb29a5',
+ 'wiki.mk.zip': 'b1fc2d85527e99530a93e3bbc5fa9fcde89910f3',
+ 'wiki.ml.zip': 'b9d53b8e76a05f5e959afd190da3015b36793297',
+ 'wiki.mn.zip': '715bf0ee67b48ec872659380fcf63ad006ddcc7e',
+ 'wiki.mo.zip': 'fb273fe373eb61310051d94ad6911320f573d0ec',
+ 'wiki.mrj.zip': 'b0d1e43e37e1718c8e05fd81a511095636def361',
+ 'wiki.mr.zip': '67e942a7742cc957298c8cd0cd0af0531dc936d7',
+ 'wiki.ms.zip': 'e218f113702b039fc8e80a77b894cd9fa4eff77d',
+ 'wiki.mt.zip': 'd68d5b636eac07b2e1307186c2c05b9a80e39658',
+ 'wiki.multi.ar.zip': '31c7b742c63c3367e9bce5c4dca37d5ceb33f1a6',
+ 'wiki.multi.bg.zip': '8991e8123bce7fd6c8e4510c71ede5715ae36f01',
+ 'wiki.multi.ca.zip': '0786e071438150485d394a4bf2e976d3a1b313ff',
+ 'wiki.multi.cs.zip': '7237f291146e69f0fc7002a0e175c7fd003d44e8',
+ 'wiki.multi.da.zip': '5591c20015191101aee190c02738c99073a8fe76',
+ 'wiki.multi.de.zip': '986160e51a08f4a93f1573d17352e375cbaedd6d',
+ 'wiki.multi.el.zip': '570eb12811ce61f6176f263eff3e945be69e7da0',
+ 'wiki.multi.en.zip': '2c3ef35d8338d4a905e7d10645572ab7a6730d44',
+ 'wiki.multi.es.zip': 'c1db7c7175665a7230f92ed038b78de780e060e9',
+ 'wiki.multi.et.zip': '54d0515865c754331b445dd9ba0ae7ed79b770aa',
+ 'wiki.multi.fi.zip': 'c94abc803a42b89cd75b278114b1f2cf4e2f3ecd',
+ 'wiki.multi.fr.zip': 'd4904b79eaf8ae386a7011ad84afc9b4238c9928',
+ 'wiki.multi.he.zip': '370ec2a379eecc2d2e984cde3e0f6d0a027eade7',
+ 'wiki.multi.hr.zip': 'd3f25ae76b040ffa09e964f6edc55488f6086394',
+ 'wiki.multi.hu.zip': '4b64bcdf0fc1f01bbd8427bd7bf6b46319308e7a',
+ 'wiki.multi.id.zip': '3ad5f590d5c847b35a334f1bdb48b9c466f5de68',
+ 'wiki.multi.it.zip': '18746450e665e96c33f2e2026986f643a27e0945',
+ 'wiki.multi.mk.zip': '1d899f1449d8729b7dbae226f05151a656694626',
+ 'wiki.multi.nl.zip': 'ff0a04dbb07c2cdbc61d5a241175e30ed46b48d4',
+ 'wiki.multi.no.zip': 'd1af729024181e64f58ae37ab233fc53811e2601',
+ 'wiki.multi.pl.zip': '91c3984c4f3158b1cb1ff11d8cc4f9240631266e',
+ 'wiki.multi.pt.zip': 'a1782c4fa4337008f82c0e2bf78e4323d145be29',
+ 'wiki.multi.ro.zip': 'b1a0840d084009ce00c47a3c24c984648dbe8785',
+ 'wiki.multi.ru.zip': '540607ba4334dab6089de463f974861aac8a35ae',
+ 'wiki.multi.sk.zip': '2a2bb39e011cf2bf6dcb8cb6c482b8eb9764eea3',
+ 'wiki.multi.sl.zip': '99442dab442dc196c107868db9174c78e270db1e',
+ 'wiki.multi.sv.zip': 'b40be83d2d7c27633c712aea62ceec0d409cc03a',
+ 'wiki.multi.tr.zip': 'e2bffab1616f54d180ba3d8bfe5e94ec9a489184',
+ 'wiki.multi.uk.zip': 'e97f64d9ba2b58a5e80c9b896b87340aba1e0eb0',
+ 'wiki.multi.vi.zip': '532fa24d8787a8906fb04a88e74a713b00cb33ec',
+ 'wiki.mus.zip': '1bb0cad10889b8a3bfa36c36c7da1f2fb2237bb8',
+ 'wiki.mwl.zip': 'e3d1fd1fa6290521d403e84eba577e552e330844',
+ 'wiki.myv.zip': '64a6505691441778766b7941b5e7f45a624a64a5',
+ 'wiki.my.zip': '491ce8dbf174d4abff758db4950f49eda90883d9',
+ 'wiki.mzn.zip': '76abf410749fd4516ead20ced891b54245fcd4a3',
+ 'wiki.nah.zip': '0496592cdd70eaf61b257fb5345843d38f425592',
+ 'wiki.nap.zip': 'f0df66cdbef5734f0afeb806cda631722fb426d8',
+ 'wiki.na.zip': '2456e4776b5e985cfaedfac244e0b40cff4e613c',
+ 'wiki.nds_nl.zip': 'ffd10e05b749281634eb7a758102d8d6ff42760e',
+ 'wiki.nds.zip': '2455e9fa4294828b25b32bdad7307a105f9fbe1d',
+ 'wiki-news-300d-1M-subword.zip': '697f4c8f37443be3aee7b96abe28fd7ebec95ef3',
+ 'wiki-news-300d-1M.zip': '567ef9c2e207be25da23e61312e6ba620da30466',
+ 'wiki.new.zip': 'a781885678cc1079d4be221c414339eb9bee8d19',
+ 'wiki.ne.zip': '180b068343288cda40d012aaa99d29459d341eb4',
+ 'wiki.ng.zip': '6db8111ab700f7b0841af87f1f1453341048014e',
+ 'wiki.nl.zip': '582420f290947cf38503b7f4b8ea9bb21918005e',
+ 'wiki.nn.zip': '4a0e30376b361ee19800e6d897a865572e330f84',
+ 'wiki.nov.zip': 'ac98c0300302019ff855698561708abd81730db3',
+ 'wiki.no.zip': '6893a7912ab3756e31d09ef1f9023c27c0b047f8',
+ 'wiki.nrm.zip': 'bd27aadf25a165ebbac486437ea6a06b710fdda6',
+ 'wiki.nso.zip': 'c55dfebb83351c952831db34e779e0a380212f05',
+ 'wiki.nv.zip': 'cf122e5ee041287917c594a2cb6cd247978f1ec0',
+ 'wiki.ny.zip': '9086021a60babd7e87afa469dbadb004523f5fd2',
+ 'wiki.oc.zip': '15075544cf837135127d8688cd06fb8e4c8b7f3d',
+ 'wiki.olo.zip': '523628bb652e1563b4dd5a94b518addf10699f74',
+ 'wiki.om.zip': 'a29360ab3930d889c4eb5b385589f84c1ff9f06e',
+ 'wiki.or.zip': 'a782e649ae5307dece445b0c11b15ffb9ce88297',
+ 'wiki.os.zip': '0d76ca005afd48b87dea5c9784c4c48bb51d3e3e',
+ 'wiki.pag.zip': 'b046ef71badc9d7eec161e3aec2ffc3abb7bad20',
+ 'wiki.pam.zip': 'abed25ef407e05209f2653d571bba5bc7c66e7b3',
+ 'wiki.pap.zip': '5d099bfc65c85f824634a191ce33e8e42f947ded',
+ 'wiki.pa.zip': '2066ed0016720b9f8779f55f2cc2de08511025f6',
+ 'wiki.pcd.zip': '66914c99e5531c0484448b84568971362cdad0f6',
+ 'wiki.pdc.zip': '6ed181fa1f8782917ae7849490c0a5cb0b0b9b29',
+ 'wiki.pfl.zip': '8d271226af8509962b15a96c4d6e41d9aabd972c',
+ 'wiki.pih.zip': '365955dbecb17027435fe487ab92a7a267fa25bd',
+ 'wiki.pi.zip': 'eeb863545392c92cff0f3e3d9c3f61539d3fa1dd',
+ 'wiki.pl.zip': '2b0cae8af2637bc24b958e6757149d1b9f8c8fea',
+ 'wiki.pms.zip': '9eff2e96e1cb9bf02adf816c4feb5aa3cd1a384f',
+ 'wiki.pnb.zip': '23f77d1d9469f5b2c342984288cb3092d53d8dee',
+ 'wiki.pnt.zip': '84cc9532d2fd7b322bcba91e01ac36c9a719e23a',
+ 'wiki.ps.zip': '18c9ffb2a81cbc25299b26e35170a29b7de9309c',
+ 'wiki.pt.zip': '37752109a44829de5ea10b173d7c0cecc0b1a0d7',
+ 'wiki.qu.zip': '5582c07eeeaec10d9382b3ab90d2921fc97fa2e0',
+ 'wiki.rmy.zip': 'a106ab536001e92e7a9708417faee9418f4058d0',
+ 'wiki.rm.zip': '67a324941f2b895a418fbd89314a18bfda19b1de',
+ 'wiki.rn.zip': 'ce17294909c046e90bb0131632e1d795d1771816',
+ 'wiki.roa_rup.zip': 'a9a378e90cd46353283c92cfb7d34dd485a018d2',
+ 'wiki.roa_tara.zip': '953fe4cf1667cbb9b3b8e11666885bfedf74b411',
+ 'wiki.ro.zip': '6bbb0f9452398416d9183e00e6cd091a02fb351f',
+ 'wiki.rue.zip': 'e9f9b8ab63c7722b4b68e8c465b1c69436132553',
+ 'wiki.ru.zip': 'f8f68aa5792941d7750b545e56f1ff5127e88cc2',
+ 'wiki.rw.zip': '018b9fb76fca5ce7a3e1f266df33fcc1bbc50493',
+ 'wiki.sah.zip': 'f6c94dbd3b719b154217388310fab72e5a69f823',
+ 'wiki.sa.zip': '4dc78b48d651056546d14b659c6598770c6bce77',
+ 'wiki.scn.zip': '218ba35c042cb3e179988bac9acf51cccf37422b',
+ 'wiki.sco.zip': 'daa8cedbb223e87d48f720aed9ce63dd0c81c632',
+ 'wiki.sc.zip': '909cc5160cad60fda34ab89c2b87ae4229402eeb',
+ 'wiki.sd.zip': '5468ed141bf2f1d9b1f8d7b31fee926b496ea9db',
+ 'wiki.se.zip': '0eb962f8768d88ffcbde3aac833e134a263c2055',
+ 'wiki.sg.zip': '651035aa74dc2f515253444f48aa9911094f9d27',
+ 'wiki.sh.zip': 'cf3057b61bd5bca6f47640801681d451aee210cf',
+ 'wiki.simple.zip': '367737535e39defb0e713a7ff2374cb932c5a9bc',
+ 'wiki.si.zip': 'cebb2f4011b0d679fe856c5950076e3c48496ecc',
+ 'wiki.sk.zip': '6c43758d0c0f52351210c558cc33266a65709068',
+ 'wiki.sl.zip': 'd0239eefc830e5919bef8d9173a884e9e7371e7a',
+ 'wiki.sm.zip': '2e3cf33f17b449c8f81cc9ea4c84d542cfd23a14',
+ 'wiki.sn.zip': '4d3844ee350ee0065e5fe910a3f669ef863a2fc9',
+ 'wiki.so.zip': '9da45db9b21d1f27c4f73152539c1e4fc9b1c49c',
+ 'wiki.sq.zip': '0db976ec147df49e648cf8256562371d0ae6f2f0',
+ 'wiki.srn.zip': '120e229d522cc22008c50e0eb74b23d9f6eca51d',
+ 'wiki.sr.zip': '63b67391158bdd7a642f7d8412771c22e1041744',
+ 'wiki.ss.zip': '4368f7931f6730a6e8cb9b5794906f2d827582a8',
+ 'wiki.stq.zip': 'fb1ba577bf6fb7f7fcdc52bf392e63ed8492465d',
+ 'wiki.st.zip': 'b7e96392b3880c19e210fd42bc72e3f76c07a4c3',
+ 'wiki.su.zip': '4c4880cfca1ff954c88e44a32f201218eb2be146',
+ 'wiki.sv.zip': 'e2b10091585f795dd18289c4a65a1da591a78196',
+ 'wiki.sw.zip': '726631d8998ba1647d040e6b70f4bad7b8d8c367',
+ 'wiki.szl.zip': 'a70de974cff95cad0443f5faa6c8412c92998100',
+ 'wiki.ta.zip': '6bafd0bb523f654038393ba191012527745b940b',
+ 'wiki.tcy.zip': 'b4bd573eaf9fd87300a25648b38a053161d12c39',
+ 'wiki.tet.zip': '7e5608958977164e544850a5a169f5d55cd47a20',
+ 'wiki.te.zip': '948e5a6ec13ac95b595c3f52a6e7b9642a56c530',
+ 'wiki.tg.zip': '5b46429024d6819f6b511a4924b90c958615d40e',
+ 'wiki.th.zip': 'b8ee0878cec41b4ab1055a17d0ed669de1ed9afd',
+ 'wiki.ti.zip': 'd55abb74bb3ff195d2293ee9e77886111ee50e52',
+ 'wiki.tk.zip': '20263f39a31a1d55343f9dea7aecaa2860aefde8',
+ 'wiki.tl.zip': '2f2b809017249f8c4f8d5eb62979b58f16e8732b',
+ 'wiki.tn.zip': '0aa11b07b1ad6437bc1e9b6476d51ddd35dad994',
+ 'wiki.to.zip': '6b90b32ae258a56e67b42736675236b91163b3ad',
+ 'wiki.tpi.zip': 'ca9591e621ae667a1521d0bb5275435d45e974cc',
+ 'wiki.tr.zip': '3b6f86c2a115c7adec1b073b1f5624890e680148',
+ 'wiki.ts.zip': '8a00b16f2881977ad6f8c8665316c27fcab9b842',
+ 'wiki.tt.zip': '8d2f559bf1e09180d6dc4b127d61815a27670a20',
+ 'wiki.tum.zip': '5b3f6f3d8cae4d9534cd1fd3afc2f64ec8342b8d',
+ 'wiki.tw.zip': '7c189fabfcdb2973178c25d35fd10e46ee7148aa',
+ 'wiki.tyv.zip': '5e3811a19bbf961a5361ac37ff3502287c9ab022',
+ 'wiki.ty.zip': 'a7f31f8cabf4282533773aa7e63f294315cc85ea',
+ 'wiki.udm.zip': '643df5ab0914535e46e6839845d0ab585c81a119',
+ 'wiki.ug.zip': 'a5388269893ac4c7da28b2284f3536ca0f3c9341',
+ 'wiki.uk.zip': 'fdc9b0a0ab806e5845e9d89b8887ec9d555a0547',
+ 'wiki.ur.zip': '75579eb5609ea31d79bc2d1bd81d01f48e01bc7c',
+ 'wiki.uz.zip': 'aa149200f8c6e3e8bb5aa3c67112675d136900b8',
+ 'wiki.vec.zip': '58c4c9528154e256fbefeb97b8c1675356079f74',
+ 'wiki.vep.zip': '966b371afcc383058a5fbc6ee8f822620f03feac',
+ 'wiki.ve.zip': '6450e3ec2c78980c5a41d71ff159aa27918dda75',
+ 'wiki.vi.zip': 'bfa287fbb358a66b4f9576585df3e46607e1595c',
+ 'wiki.vls.zip': '7335bfda43890f42e045b8a5de25d1a8629fe012',
+ 'wiki.vo.zip': 'c2ca18bea165cb1253c1d88fa9958a25088fc84b',
+ 'wiki.war.zip': '5cda8fdd64e3acf5488ad361b68a63fb23747559',
+ 'wiki.wa.zip': '2e538c10a0e9f43ea5875c90a8ce01a07c4695a7',
+ 'wiki.wo.zip': 'f54c65ab63f98ffec7b3fb5bdd51a814034bd673',
+ 'wiki.wuu.zip': '68d9ad802836737392d62056231bf1b7a58594c9',
+ 'wiki.xal.zip': 'fb39fed41ccba2e4e58ab7714a53aae3695dbe04',
+ 'wiki.xh.zip': 'd37caa4d94e66588879231d0826798d8aa4b0a44',
+ 'wiki.xmf.zip': '956c43bca0d88e9348099cde43d58898e43d9f27',
+ 'wiki.yi.zip': '151c1670c48e976e4202272b066d7080a8c83615',
+ 'wiki.yo.zip': 'fdbd0fc6e35bb04c3aef1fa6f0262ba261b11199',
+ 'wiki.za.zip': '11f6a5dcb49c4d0571d5ac4fb3d7dda1d378fc06',
+ 'wiki.zea.zip': '22159a722c5c0390bad9206eb75e6e166efe38e9',
+ 'wiki.zh_classical.zip': 'c689d61d2254caf1ecec0909249523b09a737717',
+ 'wiki.zh_min_nan.zip': '0516a413565484d924a4c8b50c690d39344cdb64',
+ 'wiki.zh_yue.zip': '464f4c1c2039194cbae7502ed3a2eeff4df9e34f',
+ 'wiki.zh.zip': '2374ec566f6411b9bb570077636695fe9768a5ba',
+ 'wiki.zu.zip': 'a6d0325dab37cd551e6d7f6c783dd13f4c71db2f'}
+
+FAST_TEXT_FILE_SHA1 = \
+ {'crawl-300d-2M.vec': '9b556504d099a6c01f3dd76b88775d02cb2f1946',
+ 'wiki.aa.vec': '5cce30fc85471572c498f278bbe495184577363e',
+ 'wiki.ab.vec': '9d89a403a9a866d3da8dd8cfab849f59ee499343',
+ 'wiki.ace.vec': '85d00074f7a08626f39da6a0c8a5cfa250096ab9',
+ 'wiki.ady.vec': '9d17d74f0348224cdebf8a831e61af0825f8952d',
+ 'wiki.af.vec': '999e64bcd8dab8de42cb1feceeca360def35324d',
+ 'wiki.ak.vec': '6092b8af335c2dc93e8df2bbf1d715f01e637bb4',
+ 'wiki.als.vec': '96052e96870695cca50857b5fde5f9f42219139a',
+ 'wiki.am.vec': 'dff7fcdd8f5ba0638ab9e1758a89800766156d72',
+ 'wiki.ang.vec': 'a7c30e02422d97d23a0701279c5c1c03159130a5',
+ 'wiki.an.vec': '5b4c2b1de5c04e4e0be83841410ca84c47305d21',
+ 'wiki.arc.vec': 'fd3ad743103f80cde9cfc048d7ca509e50efb35a',
+ 'wiki.ar.vec': 'c46e2142f799cc385bd25f0c0a8943ca565505a4',
+ 'wiki.arz.vec': '5e904087043b91f4945dd708f4230fdf51360132',
+ 'wiki.ast.vec': '89a90357101953b7c292697fd050c00fe5c38ac5',
+ 'wiki.as.vec': 'cad5883b5147cbe6cdbf604f65cabdb675a59258',
+ 'wiki.av.vec': '99976a63ca8c4231f808fd4314f0433db35e290d',
+ 'wiki.ay.vec': 'be359dad25b2c742d3abfa94c5f5db13f86c730e',
+ 'wiki.azb.vec': 'e23af0a436b97434813c3cb14ed114cc5b352faa',
+ 'wiki.az.vec': '9581d55d9056ad398a153c37b502f3a07867d091',
+ 'wiki.bar.vec': '96130f1f2e5bffdd06c202ad4472e5234020980a',
+ 'wiki.bat_smg.vec': 'cb3aef58da2011183b39fca64cabf3d9d7a62f4b',
+ 'wiki.ba.vec': '22147ee16b2d163cc88d09a035264fd0c10dab68',
+ 'wiki.bcl.vec': 'd4117b5c443438ddfa608b10a5be2c2501817e7e',
+ 'wiki.be.vec': '6cf81322cd7b046a7f02ec4c4960ad27045383fa',
+ 'wiki.bg.vec': '7c1cc6d0c52b038e4b7173259b0c009f242cf486',
+ 'wiki.bh.vec': 'ab2d29017afa015c49566a6d9bf75393c23ac4c0',
+ 'wiki.bi.vec': '15785220cd6e6c86cc87e7d3f3322a5541a4fe5d',
+ 'wiki.bjn.vec': '5f134cf288e8042dcd048a3ee76159aab42c7288',
+ 'wiki.bm.vec': 'f36a19c95e90865f6518d4487e59f363b47bd865',
+ 'wiki.bn.vec': '6fc3bfd9af455719f55bee0bea31b11afc70cf06',
+ 'wiki.bo.vec': '2e9358e03dcfa09da23d2e1499d84b10348fd8a9',
+ 'wiki.bpy.vec': 'c2bb15487c4bdb8fa869772694300ae1fee73896',
+ 'wiki.br.vec': 'df44e16abd2017e2a1b6c6588ee02779b19907f6',
+ 'wiki.bs.vec': 'c4943a290819ceae1611dd11179b40aab0df0471',
+ 'wiki.bug.vec': '942d8f7dadde5faa33aa72862501434f48e29f60',
+ 'wiki.bxr.vec': 'eaf767690c6b194605ae778719212e3874873d4c',
+ 'wiki.ca.vec': 'f5971edee11c939f6a7accfd33a9a45caa54141a',
+ 'wiki.cbk_zam.vec': '6fef47b4559eec402ce371de20dfb018acd6347d',
+ 'wiki.cdo.vec': '95e8196bf76323dbabab1b8a49ba4d677af3ccea',
+ 'wiki.ceb.vec': 'b8516a55537b8f80c927d77d95cdf7e4ff849a05',
+ 'wiki.ce.vec': '1d94b0168a773895b23889f7f07d7cf56c11a360',
+ 'wiki.cho.vec': 'cec6778f025fa9ae4134046c6c3a6291bd9c63f9',
+ 'wiki.chr.vec': '8501bf86b41074ed6c8d15b9209ef7ce83122e70',
+ 'wiki.ch.vec': '46803f3a1734f6a7b0d8cb053bbb86a6915d02e9',
+ 'wiki.chy.vec': '26c87688551ffe3a0c7a5952e894306651e62131',
+ 'wiki.ckb.vec': 'adb2fef309f1d93f429442b9c16c1564192c58f3',
+ 'wiki.co.vec': 'af876a918594e5541207bc12f17bfc4268df7b93',
+ 'wiki.crh.vec': 'c0d2310a1207fcacc94b25b149420b33bf835015',
+ 'wiki.cr.vec': '61dd9f044b7dfa56dcf1c3c07c7504c569420528',
+ 'wiki.csb.vec': '649cb2692f08414987c875dc331022567d367497',
+ 'wiki.cs.vec': 'f3ec1502aeee6a550d8cf784273fa62f61419a4e',
+ 'wiki.cu.vec': 'ddadb14ea00ea1dda716ee33732497ec049b526f',
+ 'wiki.cv.vec': '9cdb0bee5a0fea030def85597dba7108f21b0424',
+ 'wiki.cy.vec': '32d976a9bfc4dd6e39328c906eead0f597bd9e25',
+ 'wiki.da.vec': '526947dab1ffbc1465c7a766f2bca4de50676b08',
+ 'wiki.de.vec': '2ed2696afe55f023b0040b238d9a47e5fedfe48b',
+ 'wiki.diq.vec': '77f3c370d1d77806fafe368cf788af550ff607dd',
+ 'wiki.dsb.vec': 'e49a647a441fbf011ac5411dd6005e8725b9a65d',
+ 'wiki.dv.vec': 'e135ba97c711a021bc3317db2b95db5212c17658',
+ 'wiki.dz.vec': '24888f0b2cd156360bfb5e9e905240163ba798d8',
+ 'wiki.ee.vec': 'afd1670655daa7ffba51187a415fdd0b43f1d487',
+ 'wiki.el.vec': '6f034271390feaa6f9d7d16f933ddef637755979',
+ 'wiki.eml.vec': 'de6be7a2ffdda226eec730dd54b4c614bd7f5dca',
+ 'wiki.en.vec': 'c1e418f144ceb332b4328d27addf508731fa87df',
+ 'wiki.eo.vec': 'b56998fd69f66755b722a9481a9bdaf10f62c9aa',
+ 'wiki.es.vec': '2f41401aa0925167176bcd7a6770423d891dfef5',
+ 'wiki.et.vec': '64d56b66c02d5e49b1b66a85854d67d2dd9ebd41',
+ 'wiki.eu.vec': '5e72f4ef93666971fea5d2180b354e0a0821ba91',
+ 'wiki.ext.vec': '456c5632b13a0f136cd180ebe2dda67b83f78397',
+ 'wiki.fa.vec': '09b6cc685c895c66b853af9617787d3ab0891e2c',
+ 'wiki.ff.vec': '12b09d695f5fb8de4b5da9d36a73eb178b293a04',
+ 'wiki.fiu_vro.vec': '168a71a2b1c478e6810fa5dce9612d8bf8a273dc',
+ 'wiki.fi.vec': '91d19baae994d7e556b5b5938be2dc6013f9c706',
+ 'wiki.fj.vec': '36d36dc14001a109926bfc633594f6a2f7401697',
+ 'wiki.fo.vec': 'eead8ddc7bb74b12b16784723abf802bb51f844d',
+ 'wiki.frp.vec': '0eb70a613ccf807c7308c1f62535f0606465029d',
+ 'wiki.frr.vec': 'cde62af939cb2de35e341cef2c74813802a58ed4',
+ 'wiki.fr.vec': 'b092229005a65d8683a4112852fe6eb8161a6917',
+ 'wiki.fur.vec': 'd4a595cffa1abcdcf4229ba15277179ce5d20bc6',
+ 'wiki.fy.vec': 'd4beef537b7ff142a3986513879ff51a9ec14a7b',
+ 'wiki.gag.vec': 'c82ec7a5d081f0673661824f4fc34345dee255f0',
+ 'wiki.gan.vec': '7e53a33b7bd5b0360ea4cb452145616c09445029',
+ 'wiki.ga.vec': 'caaa5b2167a499893313ac1aa38416a6a0fe9a24',
+ 'wiki.gd.vec': 'f4b513598a1bf0f0d5b6521ea8ce363e9596cb97',
+ 'wiki.glk.vec': '20a7759075916e10531f5b3577302353cef565cd',
+ 'wiki.gl.vec': '8888bb8f3d70b36729b9ae479fe3765e0c083862',
+ 'wiki.gn.vec': '98594af7897c5a1f35885ddecc77556a7e7ae981',
+ 'wiki.gom.vec': '5a1193d9e5d49d06354c14e2b7c01bea176e13f1',
+ 'wiki.got.vec': 'dfa06de83a0e3099027c57b84561d7d990ea8310',
+ 'wiki.gu.vec': 'f9e13452eb63d92bea44c7c3db8fba9945c7000e',
+ 'wiki.gv.vec': '993a7ee31bdacc91763dad656aa6c2947b873473',
+ 'wiki.hak.vec': '9e83512d34c7f81739492bf0abbb25ff1ef88573',
+ 'wiki.ha.vec': '677a24efeeb1bcb8c0a931407775f18b18e875ae',
+ 'wiki.haw.vec': '58fea5aa1b37723797d26fb3d050ce6176757240',
+ 'wiki.he.vec': '55534560247394669e3f5c169136770c93bc2708',
+ 'wiki.hif.vec': '49697cf784814d3f1a47559724028e0fc0940d36',
+ 'wiki.hi.vec': '8049bb8604bc049d48bd934e27b0e184c480a413',
+ 'wiki.ho.vec': '9c75a09e099213aa8cd1f1020b223427537cbdd8',
+ 'wiki.hr.vec': '0c96f9af092cf8a84b03aec1426cd23921671489',
+ 'wiki.hsb.vec': '3dc7830544c58535bed308c552d609e13b973502',
+ 'wiki.ht.vec': '5039dfb58a074ac046813f2dae81159be8c5213f',
+ 'wiki.hu.vec': 'cd777e9efca3d4bd97c89f01690cfa4840d9c46f',
+ 'wiki.hy.vec': '21f9259d04cfd22db446a45d3622af225f00cf20',
+ 'wiki.hz.vec': '2a94b1390d68027748a05169fbc0c11a9a183456',
+ 'wiki.ia.vec': '2a348dc924638efc20c34785852b0837364aed76',
+ 'wiki.id.vec': 'c49d5c9bec89114599427f6c12a5bda2e5523dfd',
+ 'wiki.ie.vec': '01b0d11c0e7397418e73853d220e97bdcf7a8961',
+ 'wiki.ig.vec': 'd2d1643b4fb1a18a4d002cf2969073f7f201b3b2',
+ 'wiki.ii.vec': '41c6cd68b3ebe4ece2a06c37b06dca5d07c9fb3a',
+ 'wiki.ik.vec': 'af31cbec7b839f50fa70553ec63c58f7067d3ea8',
+ 'wiki.ilo.vec': 'c0e43835a3f4e0033ea5d7c6ff189982b2f26a05',
+ 'wiki.io.vec': 'af0c480c5872bff31d82e767c1116da2a6be0c00',
+ 'wiki.is.vec': 'ae0b018f92b3e218f2dacb2045a8f0a0446788a5',
+ 'wiki.it.vec': 'ac4a985e85ffae48047034e2603d804bf126caa9',
+ 'wiki.iu.vec': '5d51b2ba215005216ae003f4a6d6ef39fb30ca2e',
+ 'wiki.jam.vec': '6d51e384c56330097c2531fdbf4e74418909e388',
+ 'wiki.ja.vec': '7a2b1af1e46d795410692a002e40fa3085135f69',
+ 'wiki.jbo.vec': 'c90481946aa4b6b304528292612ae620f6549f3e',
+ 'wiki.jv.vec': '2ff7927d3ff04b8208133497b3778ede00ea463f',
+ 'wiki.kaa.vec': 'd990d3b9bd511d2d630f923099a6b9110231b2ed',
+ 'wiki.kab.vec': 'e3b73d41267d8d4cd42f6cc5a0c05dc4e021bf74',
+ 'wiki.ka.vec': '8b92b73f27f9b77818211e053a33985589de7c62',
+ 'wiki.kbd.vec': 'f5b8dbe47a7fae702232b5680b070ef6e865539e',
+ 'wiki.kg.vec': '1550647b6059e6eb649b100e31c53bd0661117b2',
+ 'wiki.ki.vec': 'c4e373e2ea13f7fa1e95b0733365e4b3fc8b2cc8',
+ 'wiki.kj.vec': 'c27e563683f9c96ff6f680a6d6bb9e9e2f9960d0',
+ 'wiki.kk.vec': '6343b2b31bad2e13d03a110b91c38fab4adc01cd',
+ 'wiki.kl.vec': 'e5def7fb1b56c5956b6e951e912d53ba0ff089f8',
+ 'wiki.km.vec': '64f7fff1df90b1f7241b232e901f76223a3719e0',
+ 'wiki.kn.vec': '32763f4f860f0d081f3aabf3e7d17b7858e7d877',
+ 'wiki.koi.vec': '4001f0617fe0fdd3b22116b304f497b7b16c6e4c',
+ 'wiki.ko.vec': '042c85a788c2778cca538cf716b8a78f0d7fa823',
+ 'wiki.krc.vec': '0c6ef043d51e5f337a309804f1db180fa0bb2cb8',
+ 'wiki.kr.vec': '25d5b4d5911a819c48328c48fb346417d07d4070',
+ 'wiki.ksh.vec': '4c3bb4f12073532b6fb7cc6c2be5e53319ef5b65',
+ 'wiki.ks.vec': '5056a87c4ee2d8bf0792436fc6b2b61648014de9',
+ 'wiki.ku.vec': '4d3a2401527dd9ba6be2b0cd31f6cd3edebadce9',
+ 'wiki.kv.vec': '164dc44d701b9d606a45f0b0446076adc3858dca',
+ 'wiki.kw.vec': 'f9eaa35a7e4f077f6de85c7801f74582f91b52c1',
+ 'wiki.ky.vec': '13b0ae3f23822317a0243bd9182105c631c834b3',
+ 'wiki.lad.vec': 'c510e520cde97050bf1cbeb36f2b90e6348ceed4',
+ 'wiki.la.vec': '9ea6286a0581084533db8d6ee96e0b7d15166543',
+ 'wiki.lbe.vec': '283619d93255571f14fd4545bb0577979171b990',
+ 'wiki.lb.vec': 'b146f23628c84e64314a35a5b6cc65a33777e22d',
+ 'wiki.lez.vec': '8e579b984a500ad89fc66767bfd7319766bd669b',
+ 'wiki.lg.vec': 'b096f5248dfbb343dc4696c97ea253510e1c4ef9',
+ 'wiki.lij.vec': '4ff5bb405c820e4119f0636efc301da15a08c00a',
+ 'wiki.li.vec': '0fb9ec4ac93676d8ef651692062bc3d7f6ae0843',
+ 'wiki.lmo.vec': 'a89414d9ceee4823622258f18936f67faf7e06e7',
+ 'wiki.ln.vec': '70b6a286b42958e25cb80824e0d8f1aee2de6dde',
+ 'wiki.lo.vec': '7c83f82b80c49b8eab21f62ecdb3681b8bda40a6',
+ 'wiki.lrc.vec': 'c1ae4fb79a19d44bfe8f601f0a30fbec841fa612',
+ 'wiki.ltg.vec': 'ec2f13d1290bd54afcaa74569e66e43e9bfef264',
+ 'wiki.lt.vec': '58d3ebef24e5e31be1a8318b45c08ebb16ad775a',
+ 'wiki.lv.vec': 'ef6b549f96e22718f513d47a611d3d6bc001a164',
+ 'wiki.mai.vec': '7f513ff36e485b19f91f83b30c32dd82e9e497f6',
+ 'wiki.map_bms.vec': 'e7deab5fdd38fa3331b1bcb4a16432b38c512e21',
+ 'wiki.mdf.vec': 'b16099ce0283a241339716eac41cfd99fdea7f36',
+ 'wiki.mg.vec': '0808252740909d6129f672584311263e7b2adadc',
+ 'wiki.mhr.vec': '39f62e292336cabc364f0d1913540b881b406393',
+ 'wiki.mh.vec': '7d2d8bff722fe0a5d869d9da11792a406aff3dc3',
+ 'wiki.min.vec': '3bb0fa596cf27a1d165c55684bebdc8d40cb8ad7',
+ 'wiki.mi.vec': 'e8acf9c7c2ab840a192c563aa776201a88e4ca89',
+ 'wiki.mk.vec': '85a3d3f13fa88ffde023d2326c65bdded4983dff',
+ 'wiki.ml.vec': '2b70fe76e8cf199a18551de782784a21e8db0b66',
+ 'wiki.mn.vec': '7cef7ecdf9d98484d9b598b25d0e717dba6acfd9',
+ 'wiki.mo.vec': 'cc54b661aefabdf516b49d24acb51273b3acf210',
+ 'wiki.mrj.vec': 'aa1c1ecba1ffd6b42c8d9659a8a04ab328ae1650',
+ 'wiki.mr.vec': '2cd6cf88bfdfb24850d345749ce0cfea8d65829e',
+ 'wiki.ms.vec': '458e1a079799a54cdc0a7b78c7fa1729d2683a6d',
+ 'wiki.mt.vec': '81f4c1d84dd4cc4276d59cb903fcc9aba46be981',
+ 'wiki.multi.ar.vec': 'f1f12cc9d629382af574a3db74fe49c2fd615c8f',
+ 'wiki.multi.bg.vec': '22470e664e4b35761a33c64433ea2f0c12140673',
+ 'wiki.multi.ca.vec': 'bc8d98b4d86d740d1985d73d211d887d561bcdd7',
+ 'wiki.multi.cs.vec': '17358b62e63f96b0479d6a70e9235a0421493884',
+ 'wiki.multi.da.vec': 'ebc75f428714d26fb1fa31accce49ad3b31e273b',
+ 'wiki.multi.de.vec': 'b9a63406aedf4446b467b94d12674bfe4723b52d',
+ 'wiki.multi.el.vec': '03d33db85bf83f35b943ce93b18c02fa98a0bc05',
+ 'wiki.multi.en.vec': '696719afdbe470ee4a2eb668229486dba1df19cc',
+ 'wiki.multi.es.vec': '98c9e35564ec57fee5dbc6155890150452f45d3f',
+ 'wiki.multi.et.vec': 'db10189093387e853f2fd3978770e1cc7bc07820',
+ 'wiki.multi.fi.vec': '746916885a1c7d4ec3f139a32cf267f9e15f5363',
+ 'wiki.multi.fr.vec': 'fe1535827b631d934beb02f8d36ba901b2c94a46',
+ 'wiki.multi.he.vec': '6dd112f018165317da22971a2b6fdb2a15dafa91',
+ 'wiki.multi.hr.vec': 'ff9f23cf595ec8dd93cd93c6b48049730c34253b',
+ 'wiki.multi.hu.vec': '6da405c9b048f3cbb990bfb29ef149f0430aa2e7',
+ 'wiki.multi.id.vec': '34edadab182682198c37ade8538530c545635742',
+ 'wiki.multi.it.vec': 'c55802bd73d46a6fc86771097670e02a70b5d46d',
+ 'wiki.multi.mk.vec': 'cec8550503ebca0bdc7ad11f2c15085b7072a990',
+ 'wiki.multi.nl.vec': 'c3f45a5fe8a8bc213cdf35dce51651b752ca60c4',
+ 'wiki.multi.no.vec': '105236df530c8fc2ce5b1e2550a2059bbc46fc28',
+ 'wiki.multi.pl.vec': '676eb5acb22982c0c9a7d6e4c90d26730c6d120e',
+ 'wiki.multi.pt.vec': '625b0a5384873c79a5dcfff5ee3fde49a3a65013',
+ 'wiki.multi.ro.vec': '82bd59674509b69f988f9870e3a291836ba43e84',
+ 'wiki.multi.ru.vec': 'a7d9c5f2ab2abb448a5111d352caa921adabe830',
+ 'wiki.multi.sk.vec': '98d849ee77f0320472cc5afa002bfde129be7089',
+ 'wiki.multi.sl.vec': 'fb5cfb8a9c44380d74fb21ddd204e820c4e05c31',
+ 'wiki.multi.sv.vec': '95d6cc3ba23dffff9be6adb467b617dd57780cb2',
+ 'wiki.multi.tr.vec': 'ecb0e353eaccba3fcacc6994d93065934ef429e9',
+ 'wiki.multi.uk.vec': '35f4f5a1ead8bd66bcaf865021fc3aae94456ab6',
+ 'wiki.multi.vi.vec': 'b1abe06360e1d65a0db65dd41ead7b2f9d651ea0',
+ 'wiki.mus.vec': 'fa1066f7bd09df4589993ca498c19aeb6cf986fd',
+ 'wiki.mwl.vec': '3d10a218242b94fcc3981aa3beb012b701827a55',
+ 'wiki.my.vec': 'e7c7989e32b23ca1a9caf534cc65ecaf9e1b9112',
+ 'wiki.myv.vec': '7de0927fd3d65677de7f770b3bd57c73b58df85d',
+ 'wiki.mzn.vec': 'aefad49237808acab99e1ca8eeaaf531666f261d',
+ 'wiki.nah.vec': 'c52e01cf4479fb7ec91ef39f298e8f97aeb6496e',
+ 'wiki.nap.vec': '6c9bd8ce1e85ee679b25189fd6f6d36afb119b6c',
+ 'wiki.na.vec': '8a592eb3dbe5693372714dff495d01cabc3ea215',
+ 'wiki.nds_nl.vec': '1cd96d12e78e5cd3f65ca2773a17696bda387b9f',
+ 'wiki.nds.vec': '7bf293149c08226e05bcf0442ac6e601162b9ffd',
+ 'wiki.ne.vec': '1045d7876f947cd4602d9ca79f7c4323a5d3a52d',
+ 'wiki-news-300d-1M-subword.vec': '717a3058e0ba5ef3cde52c3df0d4f0f60b0a113a',
+ 'wiki-news-300d-1M.vec': '11cac9efe6f599e659be182f5766d6fbd5b1cab9',
+ 'wiki.new.vec': '51f6c0b4ef1aee9fad4ab1cb69a7479db35e39a5',
+ 'wiki.ng.vec': 'c3016cc07d40bd43bea84b7c600244ff3d2a928e',
+ 'wiki.nl.vec': 'd796ee27e37b7d1d464e03c265c31ab62b52533e',
+ 'wiki.nn.vec': '35aeab89ffeca0377accbbd3bf18b81913c75448',
+ 'wiki.no.vec': 'd52e8019d7cc48569c8c3b514d2b1bd10261b5c0',
+ 'wiki.nov.vec': '5455c6e8463b1c43dd073e3e177702fb9a1dd834',
+ 'wiki.nrm.vec': 'b4cb941b126b26fa045c5fc75a490a31a969101c',
+ 'wiki.nso.vec': 'a906271509c2b343df35d1471509492bbfa883aa',
+ 'wiki.nv.vec': 'f5a6ea213bfe95c82cb22b53b4965df8b67ffeab',
+ 'wiki.ny.vec': '3aec3dcaea6c35f8254c407621644f87df37e411',
+ 'wiki.oc.vec': 'cc1833492899d75571148c2c305591f53d63f0b1',
+ 'wiki.olo.vec': 'cbadb4cada4dc579d0becdac93dfb479d76bf6c8',
+ 'wiki.om.vec': '91789a8d9f9284f7e71e4bb8d9a60eae4af4adca',
+ 'wiki.or.vec': 'a6b120fe536b6c0133b077dca0043c3bc97eef0b',
+ 'wiki.os.vec': '791b26cc300e9a1f0a08c7b2213a264e41ce30d6',
+ 'wiki.pag.vec': '03f71faf060c4eb33802275279967349c0337553',
+ 'wiki.pam.vec': '8fbd31e70d0ca0c61eb1a152efaa8ecb29180967',
+ 'wiki.pap.vec': '8cd98267cc55a4f9de80212e29651ddf7a9e83fd',
+ 'wiki.pa.vec': '4939d0db77a5b28d7d5aab0fab4f999d93b2053e',
+ 'wiki.pcd.vec': 'd2e8e7321b6f1bce94c563cb8ef8af2b45cc3e48',
+ 'wiki.pdc.vec': '401e24d0fb9b0ae9e06a5c700684361f58727fcf',
+ 'wiki.pfl.vec': '0ad9b7f3ae13f909f12835107432fee4c4ed3031',
+ 'wiki.pih.vec': '4ae6ef2a9c6c88e9322eda900e0f58be5592a29b',
+ 'wiki.pi.vec': 'd388db284357042f4260e1a567cb489b05bb8e0b',
+ 'wiki.pl.vec': 'd031adb6f83eda0364a861dcbf5ef779b5951c0b',
+ 'wiki.pms.vec': 'e30bda8d33d61db43243c157b9ac2feeaff316c8',
+ 'wiki.pnb.vec': '35f38862d3d83012d6db7baa8a4105e3e0a416e7',
+ 'wiki.pnt.vec': '38134772012d68f247e34daf220d9d4ed3e7f489',
+ 'wiki.ps.vec': '64f1bec5d5b937289199ceae2e1da6557ce48852',
+ 'wiki.pt.vec': '7f11ebdb0cbf5929b38319f1e977d2c13bcd741b',
+ 'wiki.qu.vec': '58de8c8290e8bc8f2a6a677312e28457113437b2',
+ 'wiki.rm.vec': '5d3144b47a0dd98648a6df0636384ab2a010ad7b',
+ 'wiki.rmy.vec': '3d36d3485961900c23355a0f7c2ba656a8558c29',
+ 'wiki.rn.vec': '80b6171b78dd932f59f70dbef074abb906af4eee',
+ 'wiki.roa_rup.vec': 'e31a44353cd84b976586c8df35a2ab58318120f0',
+ 'wiki.roa_tara.vec': 'b3fcb01ff0bac53a0ba08c5c0c411f26ee83a95a',
+ 'wiki.ro.vec': 'c088ea2752d5ec8b42e32410c191a14839ae8a1f',
+ 'wiki.rue.vec': 'fe539e0ea0bbbfd3ee06bd0c5521a035c7361ec5',
+ 'wiki.ru.vec': '7514a2c60ee4118abb451ed32a0d61cb52dec384',
+ 'wiki.rw.vec': 'af2ec410da6519a86ba21004c8b4c7fde768a91c',
+ 'wiki.sah.vec': '202470467194a1cbdcd571b14ef68371a29b38d9',
+ 'wiki.sa.vec': '7fed78d1d7674453b9876ee99aeeeba85ea46699',
+ 'wiki.scn.vec': 'bde043a235551e1643506774c5d9b61ecf2fc424',
+ 'wiki.sco.vec': '4625a5ad90a57f994be9b3aa4f8f3ecda941a821',
+ 'wiki.sc.vec': 'dba8dc7754ef04b1ba0cd702d94eea9575cde91c',
+ 'wiki.sd.vec': '36852d1253496e598fbd9b9009f07f454a6bea5b',
+ 'wiki.se.vec': 'f46b35ee6b893c2f12dd1b929bbc2b8120cbcd8d',
+ 'wiki.sg.vec': '90ece136bef7ad6e4e97776a1c7238499544405d',
+ 'wiki.sh.vec': '016691ecb26ace442731d92b1265e5c6c3d8ca5f',
+ 'wiki.simple.vec': '55267c50fbdf4e4ae0fbbda5c73830a379d68795',
+ 'wiki.si.vec': 'd05ed6a0bc1ee56e5d2e5f881d47372095f6eb0c',
+ 'wiki.sk.vec': '98759aacf7352d49a51390fae02030776510ae13',
+ 'wiki.sl.vec': 'b26997c0ed1de26a47b11efdc26ac1e7f189fa54',
+ 'wiki.sm.vec': '88c2c57ca483626b052403418cb4372d72352bc9',
+ 'wiki.sn.vec': '8dbb1019dcc8f842a8c0f550295ae697f8e1b7e0',
+ 'wiki.so.vec': '294756b60b03fe57cb08abd8d677d6a717b40bc8',
+ 'wiki.sq.vec': 'd07ffed553f5eb4756d0a1548a7ba9a51a52f7c6',
+ 'wiki.srn.vec': 'faee05e550f5b08809a9ae5586ac4b08c9a1c359',
+ 'wiki.sr.vec': '3cf09f476f55a92fdd2880f7ba336656ab232736',
+ 'wiki.ss.vec': '488546a3b2f88f549c50ae9f32f1997cc441b039',
+ 'wiki.stq.vec': '1bf88af29f1d86cac16042a5bea6b1651c96a8c1',
+ 'wiki.st.vec': '963646055d12873b1c83b0eef8649ecaf473d42e',
+ 'wiki.su.vec': '25e864495acb6d280bab0e62480f68550c9ceed4',
+ 'wiki.sv.vec': 'eab83ae36701139696477b91b6e8d292ef175053',
+ 'wiki.sw.vec': '8e70d207dbbd14e60a48e260a23fbf284a8e9f06',
+ 'wiki.szl.vec': '0573cf888ec70b459b0596d34814fe60fd69f190',
+ 'wiki.ta.vec': 'b66b5358527b1f3a6a421ab26464a3c1e75e18af',
+ 'wiki.tcy.vec': '388b1d89642fcc790b688e9643b3d19e14d66f40',
+ 'wiki.tet.vec': 'f38fe0e76b9b08ff652689eeee42c4fdadd9a47e',
+ 'wiki.te.vec': 'e71dcf3cc45da1bcdae5e431324025bd2026d0c8',
+ 'wiki.tg.vec': '6a5cd5bfe571ca0359b66d21bf6950553213f42d',
+ 'wiki.th.vec': '1d6e0d525392a1042d017534f6c320c5a0afd345',
+ 'wiki.ti.vec': 'c769fbc99bbb4138a40231e573685c7948d4a4c4',
+ 'wiki.tk.vec': '33ae577f77d339ab7a0dff88855b8d5c974d0aef',
+ 'wiki.tl.vec': 'd508e229ced7201510999e76d583de3ff2339d8b',
+ 'wiki.tn.vec': '39f45f3fa86645bb25c54150204abcd51cc1048c',
+ 'wiki.to.vec': '64d512665b55e9ef9a3915e8167347be79310fa0',
+ 'wiki.tpi.vec': '407b96d235f54f3e0be9dc23a3bab89c6593a621',
+ 'wiki.tr.vec': '13234aa1bf5f99e81d933482b3b83c3e4bf6c85e',
+ 'wiki.ts.vec': '00f8229e2f230afd388221c0f823a1de9fc0e443',
+ 'wiki.tt.vec': '913bb3a11da6f8142b3bbec3ef065162d9350f1d',
+ 'wiki.tum.vec': 'bfbe43364724af882a520d2edcc2ce049c7357cd',
+ 'wiki.tw.vec': 'f329b667d70d9f0b753e55e1b1579b5a5191d3bd',
+ 'wiki.ty.vec': 'b881f60b8c75a71864d9847a17961d368f3058fc',
+ 'wiki.tyv.vec': 'e8f9a36dc58e4108c553f96e247a877a099ab5ba',
+ 'wiki.udm.vec': '336a8526f22e177faac69573661dc9c3ce36591f',
+ 'wiki.ug.vec': '586d2febafaf17c9187c599ffd7b96e559103c34',
+ 'wiki.uk.vec': '77f7737b9f88eac2b3e130ea8abb8886336fd0c6',
+ 'wiki.ur.vec': 'cb8132102152a958df72bd3e25f1a72abb4c9c76',
+ 'wiki.uz.vec': '11c3a76dae12b454f693811e33ae2e60015743e2',
+ 'wiki.vec.vec': 'ae4b055fba21974e56beecab3a95f9dc24a62fd0',
+ 'wiki.vep.vec': 'a38a781fde24f4d7b52aa8bc450b9949dd4e1808',
+ 'wiki.ve.vec': 'b7d2947501de1c30a9f8496d5efae20c051104e1',
+ 'wiki.vi.vec': 'bc84245b52b2e212e28dc6856c0693ce9845a9c5',
+ 'wiki.vls.vec': '07e8636908c057b9870ce4b98c7130d460cf882a',
+ 'wiki.vo.vec': 'c830988b6965bfce2f932b1be193f7d1f755f411',
+ 'wiki.war.vec': '1f5d443d6f612b59a53820dd6f39fd886a6ad30f',
+ 'wiki.wa.vec': '18f9ca1a585e1d18c3630029141a2e19d7d34a8e',
+ 'wiki.wo.vec': '2ad96a7a9e640bc0dbcf316b1f414b92802dcb8e',
+ 'wiki.wuu.vec': 'e1cbae1d3ad52329d0f36ada764016fbacf07049',
+ 'wiki.xal.vec': 'b738222d84cb8c8fdb2b30a7219aa5d3bdc2f61c',
+ 'wiki.xh.vec': 'bf37f741b0b75953281d11df2b4d80100df9e666',
+ 'wiki.xmf.vec': 'dc1923cfd1a7002d5d60426b60e6756854ab4a14',
+ 'wiki.yi.vec': '299d61958b7dcc38774768f1489121384726d860',
+ 'wiki.yo.vec': 'e35c8aff2924ba07936be9d0d94bd298f09702a4',
+ 'wiki.za.vec': 'e3a0e58bd2e5b1891c71f1f7e37ff71997a20361',
+ 'wiki.zea.vec': 'ee12db26aab3f2b3b2745a298ef414e7aeb5a058',
+ 'wiki.zh_classical.vec': '840981c83dd8e5cb02d1cd695e2fe0870941316c',
+ 'wiki.zh_min_nan.vec': 'f91ccb013e200bb7ed560082ddf4bdd9c2f315bb',
+ 'wiki.zh.vec': '117ab34faa80e381641fbabf3a24bc8cfba44050',
+ 'wiki.zh_yue.vec': 'd2ac1ab9eb1a908797644f83f259c90cb3c1a350',
+ 'wiki.zu.vec': '4b244b9697a8280e6646842c5fc81bb3a6bc8ec7'}
diff --git a/python/mxnet/gluon/text/embedding.py b/python/mxnet/gluon/text/embedding.py
new file mode 100644
index 0000000..1839212
--- /dev/null
+++ b/python/mxnet/gluon/text/embedding.py
@@ -0,0 +1,581 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=consider-iterating-dictionary
+
+"""Text token embedding."""
+from __future__ import absolute_import
+from __future__ import print_function
+
+import io
+import logging
+import os
+import tarfile
+import warnings
+import zipfile
+
+from . import _constants as C
+from ... import nd
+from ... import registry
+from ..utils import check_sha1, download, _get_repo_file_url
+
+
+def register(embedding_cls):
+ """Registers a new token embedding.
+
+
+ Once an embedding is registered, we can create an instance of this embedding with
+ :func:`~mxnet.gluon.text.embedding.create`.
+
+
+ Examples
+ --------
+ >>> @mxnet.gluon.text.embedding.register
+ ... class MyTextEmbed(mxnet.gluon.text.embedding.TokenEmbedding):
+ ... def __init__(self, file_name='my_pretrain_file'):
+ ... pass
+ >>> embed = mxnet.gluon.text.embedding.create('MyTokenEmbed')
+ >>> print(type(embed))
+ <class '__main__.MyTokenEmbed'>
+ """
+
+ register_text_embedding = registry.get_register_func(TokenEmbedding, 'token embedding')
+ return register_text_embedding(embedding_cls)
+
+
+def create(embedding_name, **kwargs):
+ """Creates an instance of token embedding.
+
+
+ Creates a token embedding instance by loading embedding vectors from an externally hosted
+ pre-trained token embedding file, such as those of GloVe and FastText. To get all the valid
+ `embedding_name` and `file_name`, use `mxnet.gluon.text.embedding.get_file_names()`.
+
+
+ Parameters
+ ----------
+ embedding_name : str
+ The token embedding name (case-insensitive).
+
+
+ Returns
+ -------
+ An instance of `mxnet.gluon.text.embedding.TokenEmbedding`:
+ A token embedding instance that loads embedding vectors from an externally hosted
+ pre-trained token embedding file.
+ """
+
+ create_text_embedding = registry.get_create_func(TokenEmbedding, 'token embedding')
+ return create_text_embedding(embedding_name, **kwargs)
+
+
+def get_file_names(embedding_name=None):
+ """Get valid token embedding names and their pre-trained file names.
+
+
+ To load token embedding vectors from an externally hosted pre-trained token embedding file,
+ such as those of GloVe and FastText, one should use
+ `mxnet.gluon.text.embedding.create(embedding_name, file_name)`. This method returns all the
+ valid names of `file_name` for the specified `embedding_name`. If `embedding_name` is set to
+ None, this method returns all the valid names of `embedding_name` with their associated
+ `file_name`.
+
+
+ Parameters
+ ----------
+ embedding_name : str or None, default None
+ The pre-trained token embedding name.
+
+
+ Returns
+ -------
+ dict or list:
+ A list of all the valid pre-trained token embedding file names (`file_name`) for the
+ specified token embedding name (`embedding_name`). If the text embeding name is set to None,
+ returns a dict mapping each valid token embedding name to a list of valid pre-trained files
+ (`file_name`). They can be plugged into
+ `mxnet.gluon.text.embedding.create(embedding_name, file_name)`.
+ """
+
+ text_embedding_reg = registry.get_registry(TokenEmbedding)
+
+ if embedding_name is not None:
+ if embedding_name not in text_embedding_reg:
+ raise KeyError('Cannot find `embedding_name` %s. Use '
+ '`get_file_names(embedding_name=None).keys()` to get all the valid'
+ 'embedding names.' % embedding_name)
+ return list(text_embedding_reg[embedding_name].pretrained_file_name_sha1.keys())
+ else:
+ return {embedding_name: list(embedding_cls.pretrained_file_name_sha1.keys())
+ for embedding_name, embedding_cls in registry.get_registry(TokenEmbedding).items()}
+
+
+class TokenEmbedding(object):
+ """Token embedding base class.
+
+
+ To load token embedding from an externally hosted pre-trained token embedding file, such as
+ those of GloVe and FastText, use :func:`~mxnet.gluon.text.embedding.create(embedding_name,
+ file_name)`. To get all the available `embedding_name` and `file_name`, use
+ :func:`~mxnet.gluon.text.embedding.get_file_names()`.
+
+ Alternatively, to load embedding vectors from a custom pre-trained token embedding file, use
+ :func:`~mxnet.gluon.text.embedding.from_file()`.
+
+ For every unknown token, if its representation `self.unknown_token` is encountered in the
+ pre-trained token embedding file, index 0 of `self.idx_to_vec` maps to the pre-trained token
+ embedding vector loaded from the file; otherwise, index 0 of `self.idx_to_vec` maps to the
+ token embedding vector initialized by `init_unknown_vec`.
+
+ If a token is encountered multiple times in the pre-trained token embedding file, only the
+ first-encountered token embedding vector will be loaded and the rest will be skipped.
+
+
+ Parameters
+ ----------
+ unknown_token : hashable object, default '<unk>'
+ The representation for any unknown token. In other words, any unknown token will be indexed
+ as the same representation.
+
+
+ Properties
+ ----------
+ idx_to_vec : mxnet.ndarray.NDArray
+ For all the indexed tokens in this embedding, this NDArray maps each token's index to an
+ embedding vector.
+ unknown_token : hashable object
+ The representation for any unknown token. In other words, any unknown token will be indexed
+ as the same representation.
+ """
+
+ def __init__(self, unknown_token='<unk>'):
+ self._unknown_token = unknown_token
+ self._idx_to_token = [unknown_token]
+ self._token_to_idx = {token: idx for idx, token in enumerate(self._idx_to_token)}
+ self._idx_to_vec = None
+
+ @classmethod
+ def _get_download_file_name(cls, file_name):
+ return file_name
+
+ @classmethod
+ def _get_pretrained_file_url(cls, pretrained_file_name):
+ cls_name = cls.__name__.lower()
+
+ namespace = 'gluon/embeddings/{}'.format(cls_name)
+ return _get_repo_file_url(namespace, cls._get_download_file_name(pretrained_file_name))
+
+ @classmethod
+ def _get_pretrained_file(cls, embedding_root, pretrained_file_name):
+ cls_name = cls.__name__.lower()
+ embedding_root = os.path.expanduser(embedding_root)
+ url = cls._get_pretrained_file_url(pretrained_file_name)
+
+ embedding_dir = os.path.join(embedding_root, cls_name)
+ pretrained_file_path = os.path.join(embedding_dir, pretrained_file_name)
+ downloaded_file = os.path.basename(url)
+ downloaded_file_path = os.path.join(embedding_dir, downloaded_file)
+
+ expected_file_hash = cls.pretrained_file_name_sha1[pretrained_file_name]
+
+ if hasattr(cls, 'pretrained_archive_name_sha1'):
+ expected_downloaded_hash = \
+ cls.pretrained_archive_name_sha1[downloaded_file]
+ else:
+ expected_downloaded_hash = expected_file_hash
+
+ if not os.path.exists(pretrained_file_path) \
+ or not check_sha1(pretrained_file_path, expected_file_hash):
+ download(url, downloaded_file_path, sha1_hash=expected_downloaded_hash)
+
+ ext = os.path.splitext(downloaded_file)[1]
+ if ext == '.zip':
+ with zipfile.ZipFile(downloaded_file_path, 'r') as zf:
+ zf.extractall(embedding_dir)
+ elif ext == '.gz':
+ with tarfile.open(downloaded_file_path, 'r:gz') as tar:
+ tar.extractall(path=embedding_dir)
+ return pretrained_file_path
+
+ def _load_embedding(self, pretrained_file_path, elem_delim, init_unknown_vec, encoding='utf8'):
+ """Load embedding vectors from a pre-trained token embedding file.
+
+
+ For every unknown token, if its representation `self.unknown_token` is encountered in the
+ pre-trained token embedding file, index 0 of `self.idx_to_vec` maps to the pre-trained token
+ embedding vector loaded from the file; otherwise, index 0 of `self.idx_to_vec` maps to the
+ text embedding vector initialized by `init_unknown_vec`.
+
+ If a token is encountered multiple times in the pre-trained text embedding file, only the
+ first-encountered token embedding vector will be loaded and the rest will be skipped.
+ """
+
+ pretrained_file_path = os.path.expanduser(pretrained_file_path)
+
+ if not os.path.isfile(pretrained_file_path):
+ raise ValueError('`pretrained_file_path` must be a valid path to the pre-trained '
+ 'token embedding file.')
+
+ logging.info('Loading pre-trained token embedding vectors from %s', pretrained_file_path)
+ vec_len = None
+ all_elems = []
+ tokens = set()
+ loaded_unknown_vec = None
+ line_num = 0
+ with io.open(pretrained_file_path, 'r', encoding=encoding) as f:
+ for line in f:
+ line_num += 1
+ elems = line.rstrip().split(elem_delim)
+
+ assert len(elems) > 1, 'At line %d of the pre-trained token embedding file: the ' \
+ 'data format of the pre-trained token embedding file %s ' \
+ 'is unexpected.' % (line_num, pretrained_file_path)
+
+ token, elems = elems[0], [float(i) for i in elems[1:]]
+
+ if token == self.unknown_token and loaded_unknown_vec is None:
+ loaded_unknown_vec = elems
+ tokens.add(self.unknown_token)
+ elif token in tokens:
+ warnings.warn('At line %d of the pre-trained token embedding file: the '
+ 'embedding vector for token %s has been loaded and a duplicate '
+ 'embedding for the same token is seen and skipped.' %
+ (line_num, token))
+ elif len(elems) == 1:
+ warnings.warn('At line %d of the pre-trained token embedding file: token %s '
+ 'with 1-dimensional vector %s is likely a header and is '
+ 'skipped.' % (line_num, token, elems))
+ else:
+ if vec_len is None:
+ vec_len = len(elems)
+ # Reserve a vector slot for the unknown token at the very beggining because
+ # the unknown token index is 0.
+ all_elems.extend([0] * vec_len)
+ else:
+ assert len(elems) == vec_len, \
+ 'At line %d of the pre-trained token embedding file: the dimension ' \
+ 'of token %s is %d but the dimension of previous tokens is %d. ' \
+ 'Dimensions of all the tokens must be the same.' \
+ % (line_num, token, len(elems), vec_len)
+ all_elems.extend(elems)
+ self._idx_to_token.append(token)
+ self._token_to_idx[token] = len(self._idx_to_token) - 1
+ tokens.add(token)
+
+ self._idx_to_vec = nd.array(all_elems).reshape((-1, vec_len))
+
+ if loaded_unknown_vec is None:
+ self._idx_to_vec[C.UNKNOWN_IDX] = init_unknown_vec(shape=vec_len)
+ else:
+ self._idx_to_vec[C.UNKNOWN_IDX] = nd.array(loaded_unknown_vec)
+
+ @property
+ def idx_to_vec(self):
+ return self._idx_to_vec
+
+ @property
+ def unknown_token(self):
+ return self._unknown_token
+
+ def __contains__(self, x):
+ return x in self._token_to_idx
+
+ def __getitem__(self, tokens):
+ """Looks up embedding vectors of text tokens.
+
+
+ Parameters
+ ----------
+ tokens : str or list of strs
+ A token or a list of tokens.
+
+
+ Returns
+ -------
+ mxnet.ndarray.NDArray:
+ The embedding vector(s) of the token(s). According to numpy conventions, if `tokens` is
+ a string, returns a 1-D NDArray (vector); if `tokens` is a list of
+ strings, returns a 2-D NDArray (matrix) of shape=(len(tokens), vec_len).
+ """
+
+ to_reduce = not isinstance(tokens, (list, tuple))
+ if to_reduce:
+ tokens = [tokens]
+
+ indices = [self._token_to_idx.get(token, C.UNKNOWN_IDX) for token in tokens]
+
+ vecs = nd.Embedding(nd.array(indices), self.idx_to_vec, self.idx_to_vec.shape[0],
+ self.idx_to_vec.shape[1])
+
+ return vecs[0] if to_reduce else vecs
+
+ def __setitem__(self, tokens, new_vectors):
+ """Updates embedding vectors for tokens.
+
+
+ Parameters
+ ----------
+ tokens : str or a list of strs
+ A token or a list of tokens whose embedding vector are to be updated.
+ new_vectors : mxnet.ndarray.NDArray
+ An NDArray to be assigned to the embedding vectors of `tokens`. Its length must be equal
+ to the number of `tokens` and its width must be equal to the dimension of embedding of
+ the glossary. If `tokens` is a singleton, it must be 1-D or 2-D. If `tokens` is a list
+ of multiple strings, it must be 2-D.
+ """
+
+ assert self._idx_to_vec is not None, '`idx_to_vec` has not been initialized.'
+
+ if not isinstance(tokens, list) or len(tokens) == 1:
+ assert isinstance(new_vectors, nd.NDArray) and len(new_vectors.shape) in [1, 2], \
+ '`new_vectors` must be a 1-D or 2-D NDArray if `tokens` is a singleton.'
+ if not isinstance(tokens, list):
+ tokens = [tokens]
+ if len(new_vectors.shape) == 1:
+ new_vectors = new_vectors.expand_dims(0)
+
+ else:
+ assert isinstance(new_vectors, nd.NDArray) and len(new_vectors.shape) == 2, \
+ '`new_vectors` must be a 2-D NDArray if `tokens` is a list of multiple strings.'
+ assert new_vectors.shape == (len(tokens), self._idx_to_vec.shape[1]), \
+ 'The length of new_vectors must be equal to the number of tokens and the width of' \
+ 'new_vectors must be equal to the dimension of embedding of the glossary.'
+
+ indices = []
+ for token in tokens:
+ if token in self._token_to_idx:
+ indices.append(self._token_to_idx[token])
+ else:
+ raise ValueError('Token %s is unknown. To update the embedding vector for an '
+ 'unknown token, please specify it explicitly as the '
+ '`unknown_token` %s in `tokens`. This is to avoid unintended '
+ 'updates.' % (token, self._idx_to_token[C.UNKNOWN_IDX]))
+
+ self._idx_to_vec[nd.array(indices)] = new_vectors
+
+ @classmethod
+ def _check_pretrained_file_names(cls, file_name):
+ """Checks if a pre-trained token embedding file name is valid.
+
+
+ Parameters
+ ----------
+ file_name : str
+ The pre-trained token embedding file.
+ """
+
+ embedding_name = cls.__name__.lower()
+ if file_name not in cls.pretrained_file_name_sha1:
+ raise KeyError('Cannot find pre-trained file %s for token embedding %s. Valid '
+ 'pre-trained file names for embedding %s: %s' %
+ (file_name, embedding_name, embedding_name,
+ ', '.join(cls.pretrained_file_name_sha1.keys())))
+
+ @staticmethod
+ def from_file(file_path, elem_delim=' ', encoding='utf8', init_unknown_vec=nd.zeros, **kwargs):
+ """Creates a user-defined token embedding from a pre-trained embedding file.
+
+
+ This is to load embedding vectors from a user-defined pre-trained token embedding file.
+ For example, if `elem_delim` = ' ', the expected format of a custom pre-trained token
+ embedding file may look like:
+
+ 'hello 0.1 0.2 0.3 0.4 0.5\\\\nworld 1.1 1.2 1.3 1.4 1.5\\\\n'
+
+ where embedding vectors of words `hello` and `world` are [0.1, 0.2, 0.3, 0.4, 0.5] and
+ [1.1, 1.2, 1.3, 1.4, 1.5] respectively.
+
+
+ Parameters
+ ----------
+ file_path : str
+ The path to the user-defined pre-trained token embedding file.
+ elem_delim : str, default ' '
+ The delimiter for splitting a token and every embedding vector element value on the same
+ line of the custom pre-trained token embedding file.
+ encoding : str, default 'utf8'
+ The encoding scheme for reading the custom pre-trained token embedding file.
+ init_unknown_vec : callback
+ The callback used to initialize the embedding vector for the unknown token.
+
+
+ Returns
+ -------
+ instance of `~mxnet.gluon.text.embedding.TokenEmbedding`
+ The user-defined token embedding instance.
+ """
+ embedding = TokenEmbedding(**kwargs)
+ embedding._load_embedding(file_path, elem_delim, init_unknown_vec, encoding)
+
+ return embedding
+
+
+@register
+class GloVe(TokenEmbedding):
+ """The GloVe word embedding.
+
+
+ GloVe is an unsupervised learning algorithm for obtaining vector representations for words.
+ Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and
+ the resulting representations showcase interesting linear substructures of the word vector
+ space. (Source from https://nlp.stanford.edu/projects/glove/)
+
+ Reference:
+
+ GloVe: Global Vectors for Word Representation.
+ Jeffrey Pennington, Richard Socher, and Christopher D. Manning.
+ https://nlp.stanford.edu/pubs/glove.pdf
+
+ Website:
+
+ https://nlp.stanford.edu/projects/glove/
+
+ To get the updated URLs to the externally hosted pre-trained token embedding
+ files, visit https://nlp.stanford.edu/projects/glove/
+
+ License for pre-trained embedding:
+
+ https://opendatacommons.org/licenses/pddl/
+
+
+ Parameters
+ ----------
+ file_name : str, default 'glove.6B.50d.txt'
+ The name of the pre-trained token embedding file.
+ embedding_root : str, default os.path.join('~', '.mxnet', 'embedding')
+ The root directory for storing embedding-related files.
+ init_unknown_vec : callback
+ The callback used to initialize the embedding vector for the unknown token.
+
+
+ Properties
+ ----------
+ idx_to_vec : mxnet.ndarray.NDArray
+ For all the indexed tokens in this embedding, this NDArray maps each token's index to an
+ embedding vector.
+ unknown_token : hashable object
+ The representation for any unknown token. In other words, any unknown token will be indexed
+ as the same representation.
+ """
+
+ # Map a pre-trained token embedding archive file and its SHA-1 hash.
+ pretrained_archive_name_sha1 = C.GLOVE_PRETRAINED_FILE_SHA1
+
+ # Map a pre-trained token embedding file and its SHA-1 hash.
+ pretrained_file_name_sha1 = C.GLOVE_PRETRAINED_ARCHIVE_SHA1
+
+ @classmethod
+ def _get_download_file_name(cls, file_name):
+ # Map a pre-trained embedding file to its archive to download.
+ src_archive = {archive.split('.')[1]: archive for archive in
+ GloVe.pretrained_archive_name_sha1.keys()}
+ archive = src_archive[file_name.split('.')[1]]
+ return archive
+
+ def __init__(self, file_name='glove.6B.50d.txt',
+ embedding_root=os.path.join('~', '.mxnet', 'embedding'),
+ init_unknown_vec=nd.zeros, **kwargs):
+ GloVe._check_pretrained_file_names(file_name)
+
+ super(GloVe, self).__init__(**kwargs)
+ pretrained_file_path = GloVe._get_pretrained_file(embedding_root, file_name)
+
+ self._load_embedding(pretrained_file_path, ' ', init_unknown_vec)
+
+
+@register
+class FastText(TokenEmbedding):
+ """The fastText word embedding.
+
+
+ FastText is an open-source, free, lightweight library that allows users to learn text
+ representations and text classifiers. It works on standard, generic hardware. Models can later
+ be reduced in size to even fit on mobile devices. (Source from https://fasttext.cc/)
+
+
+ References:
+
+ Enriching Word Vectors with Subword Information.
+ Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.
+ https://arxiv.org/abs/1607.04606
+
+ Bag of Tricks for Efficient Text Classification.
+ Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov.
+ https://arxiv.org/abs/1607.01759
+
+ FastText.zip: Compressing text classification models.
+ Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, and Tomas Mikolov.
+ https://arxiv.org/abs/1612.03651
+
+ For 'wiki.multi' embedding:
+ Word Translation Without Parallel Data
+ Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou.
+ https://arxiv.org/abs/1710.04087
+
+ Website:
+
+ https://fasttext.cc/
+
+ To get the updated URLs to the externally hosted pre-trained token embedding files, visit
+ https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md
+
+ License for pre-trained embedding:
+
+ https://creativecommons.org/licenses/by-sa/3.0/
+
+
+ Parameters
+ ----------
+ file_name : str, default 'glove.6B.50d.txt'
+ The name of the pre-trained token embedding file.
+ embedding_root : str, default os.path.join('~', '.mxnet', 'embedding')
+ The root directory for storing embedding-related files.
+ init_unknown_vec : callback
+ The callback used to initialize the embedding vector for the unknown token.
+
+
+ Properties
+ ----------
+ idx_to_vec : mxnet.ndarray.NDArray
+ For all the indexed tokens in this embedding, this NDArray maps each token's index to an
+ embedding vector.
+ unknown_token : hashable object
+ The representation for any unknown token. In other words, any unknown token will be indexed
+ as the same representation.
+ """
+
+ # Map a pre-trained token embedding archive file and its SHA-1 hash.
+ pretrained_archive_name_sha1 = C.FAST_TEXT_ARCHIVE_SHA1
+
+ # Map a pre-trained token embedding file and its SHA-1 hash.
+ pretrained_file_name_sha1 = C.FAST_TEXT_FILE_SHA1
+
+ @classmethod
+ def _get_download_file_name(cls, file_name):
+ # Map a pre-trained embedding file to its archive to download.
+ return '.'.join(file_name.split('.')[:-1]) + '.zip'
+
+ def __init__(self, file_name='wiki.simple.vec',
+ embedding_root=os.path.join('~', '.mxnet', 'embedding'),
+ init_unknown_vec=nd.zeros, **kwargs):
+ FastText._check_pretrained_file_names(file_name)
+
+ super(FastText, self).__init__(**kwargs)
+ pretrained_file_path = FastText._get_pretrained_file(embedding_root, file_name)
+
+ self._load_embedding(pretrained_file_path, ' ', init_unknown_vec)
diff --git a/python/mxnet/gluon/text/utils.py b/python/mxnet/gluon/text/utils.py
new file mode 100644
index 0000000..a812be8
--- /dev/null
+++ b/python/mxnet/gluon/text/utils.py
@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+
+"""Utilities for text data processing."""
+from __future__ import absolute_import
+from __future__ import print_function
+
+import collections
+import re
+
+
+def count_tokens_from_str(source_str, token_delim=' ', seq_delim='\n', to_lower=False,
+ counter_to_update=None):
+ """Counts tokens in the specified string.
+
+
+ For token_delim='(td)' and seq_delim='(sd)', a specified string of two sequences of tokens may
+ look like::
+
+ (td)token1(td)token2(td)token3(td)(sd)(td)token4(td)token5(td)(sd)
+
+
+ Parameters
+ ----------
+ source_str : str
+ A source string of tokens.
+ token_delim : str, default ' '
+ A token delimiter.
+ seq_delim : str, default '\\\\n'
+ A sequence delimiter.
+ to_lower : bool, default False
+ Whether to convert the source source_str to the lower case.
+ counter_to_update : collections.Counter or None, default None
+ The collections.Counter instance to be updated with the token counts of `source_str`. If
+ None, return a new collections.Counter instance counting tokens from `source_str`.
+
+
+ Returns
+ -------
+ collections.Counter
+ The `counter_to_update` collections.Counter instance after being updated with the token
+ counts of `source_str`. If `counter_to_update` is None, return a new collections.Counter
+ instance counting tokens from `source_str`.
+
+
+ Examples
+ --------
+ >>> source_str = ' Life is great ! \\n life is good . \\n'
+ >>> count_tokens_from_str(token_line, ' ', '\\n', True)
+ Counter({'!': 1, '.': 1, 'good': 1, 'great': 1, 'is': 2, 'life': 2})
+ """
+
+ source_str = filter(None, re.split(token_delim + '|' + seq_delim, source_str))
+ if to_lower:
+ source_str = [t.lower() for t in source_str]
+
+ if counter_to_update is None:
+ return collections.Counter(source_str)
+ else:
+ counter_to_update.update(source_str)
+ return counter_to_update
diff --git a/python/mxnet/gluon/text/vocab.py b/python/mxnet/gluon/text/vocab.py
new file mode 100644
index 0000000..aa962af
--- /dev/null
+++ b/python/mxnet/gluon/text/vocab.py
@@ -0,0 +1,323 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=consider-iterating-dictionary
+
+"""Vocabulary."""
+from __future__ import absolute_import
+from __future__ import print_function
+
+import collections
+from ... import nd
+
+from . import _constants as C
+from . import embedding as ebd
+
+
+class Vocabulary(object):
+ """Indexing and embedding attachment for text tokens.
+
+
+ Parameters
+ ----------
+ counter : collections.Counter or None, default None
+ Counts text token frequencies in the text data. Its keys will be indexed according to
+ frequency thresholds such as `max_size` and `min_freq`. Keys of `counter`,
+ `unknown_token`, and values of `reserved_tokens` must be of the same hashable type.
+ Examples: str, int, and tuple.
+ max_size : None or int, default None
+ The maximum possible number of the most frequent tokens in the keys of `counter` that can be
+ indexed. Note that this argument does not count any token from `reserved_tokens`. Suppose
+ that there are different keys of `counter` whose frequency are the same, if indexing all of
+ them will exceed this argument value, such keys will be indexed one by one according to
+ their __cmp__() order until the frequency threshold is met. If this argument is None or
+ larger than its largest possible value restricted by `counter` and `reserved_tokens`, this
+ argument has no effect.
+ min_freq : int, default 1
+ The minimum frequency required for a token in the keys of `counter` to be indexed.
+ unknown_token : hashable object, default '<unk>'
+ The representation for any unknown token. In other words, any unknown token will be indexed
+ as the same representation. Keys of `counter`, `unknown_token`, and values of
+ `reserved_tokens` must be of the same hashable type. Examples: str, int, and tuple.
+ reserved_tokens : list of hashable objects or None, default None
+ A list of reserved tokens (excluding `unknown_token`) that will always be indexed, such as
+ special symbols representing padding, beginning of sentence, and end of sentence. It cannot
+ contain `unknown_token` or duplicate reserved tokens. Keys of `counter`, `unknown_token`,
+ and values of `reserved_tokens` must be of the same hashable type. Examples: str, int, and
+ tuple.
+
+
+ Properties
+ ----------
+ embedding : instance of :class:`~mxnet.gluon.text.embedding.TokenEmbedding`
+ The embedding of the indexed tokens.
+ idx_to_token : list of strs
+ A list of indexed tokens where the list indices and the token indices are aligned.
+ reserved_tokens : list of strs or None
+ A list of reserved tokens that will always be indexed.
+ token_to_idx : dict mapping str to int
+ A dict mapping each token to its index integer.
+ unknown_token : hashable object
+ The representation for any unknown token. In other words, any unknown token will be indexed
+ as the same representation.
+
+
+ Examples
+ --------
+
+ >>> text_data = " hello world \\\\n hello nice world \\\\n hi world \\\\n"
+ >>> counter = text.count_tokens_from_str(text_data)
+ >>> my_vocab = text.Vocabulary(counter)
+ >>> fasttext = text.embedding.create('fasttext', file_name='wiki.simple.vec')
+ >>> my_vocab.set_embedding(fasttext)
+ >>> my_vocab.embedding[['hello', 'world']]
+ [[ 3.95669997e-01 2.14540005e-01 -3.53889987e-02 -2.42990002e-01
+ ...
+ -7.54180014e-01 -3.14429998e-01 2.40180008e-02 -7.61009976e-02]
+ [ 1.04440004e-01 -1.08580001e-01 2.72119999e-01 1.32990003e-01
+ ...
+ -3.73499990e-01 5.67310005e-02 5.60180008e-01 2.90190000e-02]]
+ <NDArray 2x300 @cpu(0)>
+
+ >>> my_vocab[['hello', 'world']]
+ [2, 1]
+
+ >>> input_dim, output_dim = my_vocab.embedding.idx_to_vec.shape
+ >>> layer = gluon.nn.Embedding(input_dim, output_dim)
+ >>> layer.initialize()
+ >>> layer.weight.set_data(my_vocab.embedding.idx_to_vec)
+ >>> layer(nd.array([2, 1]))
+ [[ 3.95669997e-01 2.14540005e-01 -3.53889987e-02 -2.42990002e-01
+ ...
+ -7.54180014e-01 -3.14429998e-01 2.40180008e-02 -7.61009976e-02]
+ [ 1.04440004e-01 -1.08580001e-01 2.72119999e-01 1.32990003e-01
+ ...
+ -3.73499990e-01 5.67310005e-02 5.60180008e-01 2.90190000e-02]]
+ <NDArray 2x300 @cpu(0)>
+
+ >>> glove = text.embedding.create('glove', file_name='glove.6B.50d.txt')
+ >>> my_vocab.set_embedding(glove)
+ >>> my_vocab.embedding[['hello', 'world']]
+ [[ -0.38497001 0.80092001
+ ...
+ 0.048833 0.67203999]
+ [ -0.41486001 0.71847999
+ ...
+ -0.37639001 -0.67541999]]
+ <NDArray 2x50 @cpu(0)>
+
+ """
+
+ def __init__(self, counter=None, max_size=None, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=None):
+
+ # Sanity checks.
+ assert min_freq > 0, '`min_freq` must be set to a positive value.'
+
+ if reserved_tokens is not None:
+ reserved_token_set = set(reserved_tokens)
+ assert unknown_token not in reserved_token_set, \
+ '`reserved_token` cannot contain `unknown_token`.'
+ assert len(reserved_token_set) == len(reserved_tokens), \
+ '`reserved_tokens` cannot contain duplicate reserved tokens.'
+
+ self._index_unknown_and_reserved_tokens(unknown_token, reserved_tokens)
+
+ if counter is not None:
+ self._index_counter_keys(counter, unknown_token, reserved_tokens, max_size, min_freq)
+
+ self._embedding = None
+
+ def _index_unknown_and_reserved_tokens(self, unknown_token, reserved_tokens):
+ """Indexes unknown and reserved tokens."""
+
+ self._unknown_token = unknown_token
+ self._idx_to_token = [unknown_token]
+
+ if reserved_tokens is None:
+ self._reserved_tokens = None
+ else:
+ self._reserved_tokens = reserved_tokens[:]
+ self._idx_to_token.extend(reserved_tokens)
+
+ self._token_to_idx = {token: idx for idx, token in enumerate(self._idx_to_token)}
+
+ def _index_counter_keys(self, counter, unknown_token, reserved_tokens, max_size,
+ min_freq):
+ """Indexes keys of `counter`.
+
+
+ Indexes keys of `counter` according to frequency thresholds such as `max_size` and
+ `min_freq`.
+ """
+
+ assert isinstance(counter, collections.Counter), \
+ '`counter` must be an instance of collections.Counter.'
+
+ unknown_and_reserved_tokens = set(reserved_tokens) if reserved_tokens is not None else set()
+ unknown_and_reserved_tokens.add(unknown_token)
+
+ token_freqs = sorted(counter.items(), key=lambda x: x[0])
+ token_freqs.sort(key=lambda x: x[1], reverse=True)
+
+ token_cap = len(unknown_and_reserved_tokens) + (
+ len(counter) if max_size is None else max_size)
+
+ for token, freq in token_freqs:
+ if freq < min_freq or len(self._idx_to_token) == token_cap:
+ break
+ if token not in unknown_and_reserved_tokens:
+ self._idx_to_token.append(token)
+ self._token_to_idx[token] = len(self._idx_to_token) - 1
+
+ @property
+ def embedding(self):
+ return self._embedding
+
+ @property
+ def idx_to_token(self):
+ return self._idx_to_token
+
+ @property
+ def reserved_tokens(self):
+ return self._reserved_tokens
+
+ @property
+ def token_to_idx(self):
+ return self._token_to_idx
+
+ @property
+ def unknown_token(self):
+ return self._unknown_token
+
+ def __contains__(self, token):
+ """Checks whether a text token exists in the vocabulary.
+
+
+ Parameters
+ ----------
+ token : str
+ A text token.
+
+
+ Returns
+ -------
+ bool
+ Whether the text token exists in the vocabulary (including `unknown_token`).
+ """
+
+ return token in self._token_to_idx
+
+ def __getitem__(self, tokens):
+ """Looks up indices of text tokens according to the vocabulary.
+
+
+ Parameters
+ ----------
+ tokens : str or list of strs
+ A source token or tokens to be converted.
+
+
+ Returns
+ -------
+ int or list of ints
+ A token index or a list of token indices according to the vocabulary.
+ """
+
+ if not isinstance(tokens, (list, tuple)):
+ return self._token_to_idx[tokens] if tokens in self._token_to_idx \
+ else C.UNKNOWN_IDX
+ else:
+ return [self._token_to_idx[token] if token in self._token_to_idx
+ else C.UNKNOWN_IDX for token in tokens]
+
+ def __len__(self):
+ return len(self._idx_to_token)
+
+ def set_embedding(self, *embeddings):
+ """Attaches embeddings to the indexed text tokens.
+
+
+ Parameters
+ ----------
+ embeddings : None or tuple of :class:`~mxnet.gluon.text.embedding.TokenEmbedding` instances
+ The embedding to be attached to the indexed tokens. If a tuple of multiple embeddings
+ are provided, their embedding vectors will be concatenated for the same token.
+ """
+
+ if len(embeddings) == 1 and embeddings[0] is None:
+ self._embedding = None
+ return
+
+ for embedding in embeddings:
+ assert isinstance(embedding, ebd.TokenEmbedding), \
+ 'The argument `embeddings` must be an instance or a list of instances of ' \
+ '`mxnet.gluon.text.embedding.TokenEmbedding`.'
+
+ new_embedding = ebd.TokenEmbedding(self.unknown_token)
+ new_embedding._token_to_idx = self.token_to_idx
+ new_embedding._idx_to_token = self.idx_to_token
+
+ new_vec_len = sum(embedding.idx_to_vec.shape[1] for embedding in embeddings
+ if embedding and embedding.idx_to_vec is not None)
+ new_idx_to_vec = nd.zeros(shape=(len(self), new_vec_len))
+
+ col_start = 0
+ # Concatenate all the embedding vectors in embedding.
+ for embedding in embeddings:
+ if embedding and embedding.idx_to_vec is not None:
+ col_end = col_start + embedding.idx_to_vec.shape[1]
+ # Cancatenate vectors of the unknown token.
+ new_idx_to_vec[0, col_start:col_end] = embedding[0]
+ new_idx_to_vec[1:, col_start:col_end] = embedding[self._idx_to_token[1:]]
+ col_start = col_end
+
+ new_embedding._idx_to_vec = new_idx_to_vec
+ self._embedding = new_embedding
+
+ def to_tokens(self, indices):
+ """Converts token indices to tokens according to the vocabulary.
+
+
+ Parameters
+ ----------
+ indices : int or list of ints
+ A source token index or token indices to be converted.
+
+
+ Returns
+ -------
+ str or list of strs
+ A token or a list of tokens according to the vocabulary.
+ """
+
+ to_reduce = False
+ if not isinstance(indices, (list, tuple)):
+ indices = [indices]
+ to_reduce = True
+
+ max_idx = len(self._idx_to_token) - 1
+
+ tokens = []
+ for idx in indices:
+ if not isinstance(idx, int) or idx > max_idx:
+ raise ValueError('Token index %d in the provided `indices` is invalid.' % idx)
+ else:
+ tokens.append(self._idx_to_token[idx])
+
+ return tokens[0] if to_reduce else tokens
diff --git a/tests/python/unittest/test_gluon_text.py b/tests/python/unittest/test_gluon_text.py
new file mode 100644
index 0000000..ad4f4a0
--- /dev/null
+++ b/tests/python/unittest/test_gluon_text.py
@@ -0,0 +1,681 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# 'License'); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+from collections import Counter
+
+from common import assertRaises
+from mxnet import ndarray as nd
+from mxnet.gluon import text
+from mxnet.test_utils import *
+
+
+def _get_test_str_of_tokens(token_delim, seq_delim):
+ seq1 = token_delim + token_delim.join(['Life', 'is', 'great', '!']) + token_delim + seq_delim
+ seq2 = token_delim + token_delim.join(['life', 'is', 'good', '.']) + token_delim + seq_delim
+ seq3 = token_delim + token_delim.join(['life', "isn't", 'bad', '.']) + token_delim + seq_delim
+ seqs = seq1 + seq2 + seq3
+ return seqs
+
+
+def _test_count_tokens_from_str_with_delims(token_delim, seq_delim):
+ source_str = _get_test_str_of_tokens(token_delim, seq_delim)
+
+ cnt1 = text.count_tokens_from_str(source_str, token_delim, seq_delim, to_lower=False)
+ assert cnt1 == Counter(
+ {'is': 2, 'life': 2, '.': 2, 'Life': 1, 'great': 1, '!': 1, 'good': 1, "isn't": 1,
+ 'bad': 1})
+
+ cnt2 = text.count_tokens_from_str(source_str, token_delim, seq_delim, to_lower=True)
+ assert cnt2 == Counter(
+ {'life': 3, 'is': 2, '.': 2, 'great': 1, '!': 1, 'good': 1, "isn't": 1, 'bad': 1})
+
+ counter_to_update = Counter({'life': 2})
+
+ cnt3 = text.utils.count_tokens_from_str(source_str, token_delim, seq_delim, to_lower=False,
+ counter_to_update=counter_to_update.copy())
+ assert cnt3 == Counter(
+ {'is': 2, 'life': 4, '.': 2, 'Life': 1, 'great': 1, '!': 1, 'good': 1, "isn't": 1,
+ 'bad': 1})
+
+ cnt4 = text.count_tokens_from_str(source_str, token_delim, seq_delim, to_lower=True,
+ counter_to_update=counter_to_update.copy())
+ assert cnt4 == Counter(
+ {'life': 5, 'is': 2, '.': 2, 'great': 1, '!': 1, 'good': 1, "isn't": 1, 'bad': 1})
+
+
+def test_count_tokens_from_str():
+ _test_count_tokens_from_str_with_delims(' ', '\n')
+ _test_count_tokens_from_str_with_delims('IS', 'LIFE')
+
+
+def test_vocabulary_getitem():
+ counter = Counter(['a', 'b', 'b', 'c', 'c', 'c', 'some_word$'])
+
+ vocab = text.Vocabulary(counter, max_size=None, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=None)
+
+ i1 = vocab['c']
+ assert i1 == 1
+
+ i2 = vocab[['c']]
+ assert i2 == [1]
+
+ i3 = vocab[['<unk>', 'non-exist']]
+ assert i3 == [0, 0]
+
+ i4 = vocab[['a', 'non-exist', 'a', 'b']]
+ assert i4 == [3, 0, 3, 2]
+
+
+def test_vocabulary_to_tokens():
+ counter = Counter(['a', 'b', 'b', 'c', 'c', 'c', 'some_word$'])
+
+ vocab = text.Vocabulary(counter, max_size=None, min_freq=1,unknown_token='<unknown>',
+ reserved_tokens=None)
+ i1 = vocab.to_tokens(1)
+ assert i1 == 'c'
+
+ i2 = vocab.to_tokens([1])
+ assert i2 == ['c']
+
+ i3 = vocab.to_tokens([0, 0])
+ assert i3 == ['<unknown>', '<unknown>']
+
+ i4 = vocab.to_tokens([3, 0, 3, 2])
+ assert i4 == ['a', '<unknown>', 'a', 'b']
+
+ assertRaises(ValueError, vocab.to_tokens, 5)
+ assertRaises(ValueError, vocab.to_tokens, [5, 6])
+
+
+def test_vocabulary():
+ counter = Counter(['a', 'b', 'b', 'c', 'c', 'c', 'some_word$'])
+
+ v1 = text.Vocabulary(counter, max_size=None, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=None)
+ assert len(v1) == 5
+ assert v1.token_to_idx == {'<unk>': 0, 'c': 1, 'b': 2, 'a': 3, 'some_word$': 4}
+ assert v1.idx_to_token[1] == 'c'
+ assert v1.unknown_token == '<unk>'
+ assert v1.reserved_tokens is None
+ assert v1.embedding is None
+ assert 'a' in v1
+ assert v1.unknown_token in v1
+
+ v2 = text.Vocabulary(counter, max_size=None, min_freq=2, unknown_token='<unk>',
+ reserved_tokens=None)
+ assert len(v2) == 3
+ assert v2.token_to_idx == {'<unk>': 0, 'c': 1, 'b': 2}
+ assert v2.idx_to_token[1] == 'c'
+ assert v2.unknown_token == '<unk>'
+ assert v2.reserved_tokens is None
+ assert v2.embedding is None
+ assert 'a' not in v2
+ assert v2.unknown_token in v2
+
+ v3 = text.Vocabulary(counter, max_size=None, min_freq=100, unknown_token='<unk>',
+ reserved_tokens=None)
+ assert len(v3) == 1
+ assert v3.token_to_idx == {'<unk>': 0}
+ assert v3.idx_to_token[0] == '<unk>'
+ assert v3.unknown_token == '<unk>'
+ assert v3.reserved_tokens is None
+ assert v3.embedding is None
+ assert 'a' not in v3
+
+ v4 = text.Vocabulary(counter, max_size=2, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=None)
+ assert len(v4) == 3
+ assert v4.token_to_idx == {'<unk>': 0, 'c': 1, 'b': 2}
+ assert v4.idx_to_token[1] == 'c'
+ assert v4.unknown_token == '<unk>'
+ assert v4.reserved_tokens is None
+ assert v4.embedding is None
+ assert 'a' not in v4
+
+ v5 = text.Vocabulary(counter, max_size=3, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=None)
+ assert len(v5) == 4
+ assert v5.token_to_idx == {'<unk>': 0, 'c': 1, 'b': 2, 'a': 3}
+ assert v5.idx_to_token[1] == 'c'
+ assert v5.unknown_token == '<unk>'
+ assert v5.reserved_tokens is None
+ assert v5.embedding is None
+ assert 'a' in v5
+
+ v6 = text.Vocabulary(counter, max_size=100, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=None)
+ assert len(v6) == 5
+ assert v6.token_to_idx == {'<unk>': 0, 'c': 1, 'b': 2, 'a': 3,
+ 'some_word$': 4}
+ assert v6.idx_to_token[1] == 'c'
+ assert v6.unknown_token == '<unk>'
+ assert v6.reserved_tokens is None
+ assert v6.embedding is None
+ assert 'a' in v6
+
+ v7 = text.Vocabulary(counter, max_size=1, min_freq=2, unknown_token='<unk>',
+ reserved_tokens=None)
+ assert len(v7) == 2
+ assert v7.token_to_idx == {'<unk>': 0, 'c': 1}
+ assert v7.idx_to_token[1] == 'c'
+ assert v7.unknown_token == '<unk>'
+ assert v7.reserved_tokens is None
+ assert v7.embedding is None
+ assert 'a' not in v7
+
+ assertRaises(AssertionError, text.Vocabulary, counter, max_size=None,
+ min_freq=0, unknown_token='<unknown>', reserved_tokens=['b'])
+
+ assertRaises(AssertionError, text.Vocabulary, counter, max_size=None,
+ min_freq=1, unknown_token='<unknown>', reserved_tokens=['b', 'b'])
+
+ assertRaises(AssertionError, text.Vocabulary, counter, max_size=None,
+ min_freq=1, unknown_token='<unknown>', reserved_tokens=['b', '<unknown>'])
+
+ v8 = text.Vocabulary(counter, max_size=None, min_freq=1, unknown_token='<unknown>',
+ reserved_tokens=['b'])
+ assert len(v8) == 5
+ assert v8.token_to_idx == {'<unknown>': 0, 'b': 1, 'c': 2, 'a': 3, 'some_word$': 4}
+ assert v8.idx_to_token[1] == 'b'
+ assert v8.unknown_token == '<unknown>'
+ assert v8.reserved_tokens == ['b']
+ assert v8.embedding is None
+ assert 'a' in v8
+
+ v9 = text.Vocabulary(counter, max_size=None, min_freq=2, unknown_token='<unk>',
+ reserved_tokens=['b', 'a'])
+ assert len(v9) == 4
+ assert v9.token_to_idx == {'<unk>': 0, 'b': 1, 'a': 2, 'c': 3}
+ assert v9.idx_to_token[1] == 'b'
+ assert v9.unknown_token == '<unk>'
+ assert v9.reserved_tokens == ['b', 'a']
+ assert v9.embedding is None
+ assert 'a' in v9
+
+ v10 = text.Vocabulary(counter, max_size=None, min_freq=100, unknown_token='<unk>',
+ reserved_tokens=['b', 'c'])
+ assert len(v10) == 3
+ assert v10.token_to_idx == {'<unk>': 0, 'b': 1, 'c': 2}
+ assert v10.idx_to_token[1] == 'b'
+ assert v10.unknown_token == '<unk>'
+ assert v10.reserved_tokens == ['b', 'c']
+ assert v10.embedding is None
+ assert 'a' not in v10
+
+ v11 = text.Vocabulary(counter, max_size=1, min_freq=2, unknown_token='<unk>',
+ reserved_tokens=['<pad>', 'b'])
+ assert len(v11) == 4
+ assert v11.token_to_idx == {'<unk>': 0, '<pad>': 1, 'b': 2, 'c': 3}
+ assert v11.idx_to_token[1] == '<pad>'
+ assert v11.unknown_token == '<unk>'
+ assert v11.reserved_tokens == ['<pad>', 'b']
+ assert v11.embedding is None
+ assert 'a' not in v11
+
+ v12 = text.Vocabulary(counter, max_size=None, min_freq=2, unknown_token='b',
+ reserved_tokens=['<pad>'])
+ assert len(v12) == 3
+ assert v12.token_to_idx == {'b': 0, '<pad>': 1, 'c': 2}
+ assert v12.idx_to_token[1] == '<pad>'
+ assert v12.unknown_token == 'b'
+ assert v12.reserved_tokens == ['<pad>']
+ assert v12.embedding is None
+ assert 'a' not in v12
+
+ v13 = text.Vocabulary(counter, max_size=None, min_freq=2, unknown_token='a',
+ reserved_tokens=['<pad>'])
+ assert len(v13) == 4
+ assert v13.token_to_idx == {'a': 0, '<pad>': 1, 'c': 2, 'b': 3}
+ assert v13.idx_to_token[1] == '<pad>'
+ assert v13.unknown_token == 'a'
+ assert v13.reserved_tokens == ['<pad>']
+ assert v13.embedding is None
+ assert 'a' in v13
+
+ counter_tuple = Counter([('a', 'a'), ('b', 'b'), ('b', 'b'), ('c', 'c'), ('c', 'c'), ('c', 'c'),
+ ('some_word$', 'some_word$')])
+
+ v14 = text.Vocabulary(counter_tuple, max_size=None, min_freq=1,
+ unknown_token=('<unk>', '<unk>'), reserved_tokens=None)
+ assert len(v14) == 5
+ assert v14.token_to_idx == {('<unk>', '<unk>'): 0, ('c', 'c'): 1, ('b', 'b'): 2, ('a', 'a'): 3,
+ ('some_word$', 'some_word$'): 4}
+ assert v14.idx_to_token[1] == ('c', 'c')
+ assert v14.unknown_token == ('<unk>', '<unk>')
+ assert v14.reserved_tokens is None
+ assert v14.embedding is None
+ assert ('a', 'a') in v14
+ assert ('<unk>', '<unk>') in v14
+
+
+def _mk_my_pretrain_file(path, token_delim, pretrain_file):
+ path = os.path.expanduser(path)
+ if not os.path.exists(path):
+ os.makedirs(path)
+ seq1 = token_delim.join(['a', '0.1', '0.2', '0.3', '0.4', '0.5']) + '\n'
+ seq2 = token_delim.join(['b', '0.6', '0.7', '0.8', '0.9', '1.0']) + '\n'
+ seqs = seq1 + seq2
+ with open(os.path.join(path, pretrain_file), 'w') as fout:
+ fout.write(seqs)
+
+
+def _mk_my_pretrain_file2(path, token_delim, pretrain_file):
+ path = os.path.expanduser(path)
+ if not os.path.exists(path):
+ os.makedirs(path)
+ seq1 = token_delim.join(['a', '0.01', '0.02', '0.03', '0.04', '0.05']) + '\n'
+ seq2 = token_delim.join(['c', '0.06', '0.07', '0.08', '0.09', '0.1']) + '\n'
+ seqs = seq1 + seq2
+ with open(os.path.join(path, pretrain_file), 'w') as fout:
+ fout.write(seqs)
+
+
+def _mk_my_pretrain_file3(path, token_delim, pretrain_file):
+ path = os.path.expanduser(path)
+ if not os.path.exists(path):
+ os.makedirs(path)
+ seq1 = token_delim.join(['a', '0.1', '0.2', '0.3', '0.4', '0.5']) + '\n'
+ seq2 = token_delim.join(['b', '0.6', '0.7', '0.8', '0.9', '1.0']) + '\n'
+ seq3 = token_delim.join(['<unk1>', '1.1', '1.2', '1.3', '1.4',
+ '1.5']) + '\n'
+ seqs = seq1 + seq2 + seq3
+ with open(os.path.join(path, pretrain_file), 'w') as fout:
+ fout.write(seqs)
+
+
+def _mk_my_pretrain_file4(path, token_delim, pretrain_file):
+ path = os.path.expanduser(path)
+ if not os.path.exists(path):
+ os.makedirs(path)
+ seq1 = token_delim.join(['a', '0.01', '0.02', '0.03', '0.04', '0.05']) + '\n'
+ seq2 = token_delim.join(['c', '0.06', '0.07', '0.08', '0.09', '0.1']) + '\n'
+ seq3 = token_delim.join(['<unk2>', '0.11', '0.12', '0.13', '0.14', '0.15']) + '\n'
+ seqs = seq1 + seq2 + seq3
+ with open(os.path.join(path, pretrain_file), 'w') as fout:
+ fout.write(seqs)
+
+
+def _mk_my_invalid_pretrain_file(path, token_delim, pretrain_file):
+ path = os.path.expanduser(path)
+ if not os.path.exists(path):
+ os.makedirs(path)
+ seq1 = token_delim.join(['a', '0.1', '0.2', '0.3', '0.4', '0.5']) + '\n'
+ seq2 = token_delim.join(['b', '0.6', '0.7', '0.8', '0.9', '1.0']) + '\n'
+ seq3 = token_delim.join(['c']) + '\n'
+ seqs = seq1 + seq2 + seq3
+ with open(os.path.join(path, pretrain_file), 'w') as fout:
+ fout.write(seqs)
+
+
+def _mk_my_invalid_pretrain_file2(path, token_delim, pretrain_file):
+ path = os.path.expanduser(path)
+ if not os.path.exists(path):
+ os.makedirs(path)
+ seq1 = token_delim.join(['a', '0.1', '0.2', '0.3', '0.4', '0.5']) + '\n'
+ seq2 = token_delim.join(['b', '0.6', '0.7', '0.8', '0.9', '1.0']) + '\n'
+ seq3 = token_delim.join(['c', '0.6', '0.7', '0.8']) + '\n'
+ seqs = seq1 + seq2 + seq3
+ with open(os.path.join(path, pretrain_file), 'w') as fout:
+ fout.write(seqs)
+
+
+def test_token_embedding_from_file():
+ embed_root = 'embedding'
+ embed_name = 'my_embed'
+ elem_delim = '\t'
+ pretrain_file = 'my_pretrain_file.txt'
+
+ _mk_my_pretrain_file(os.path.join(embed_root, embed_name), elem_delim, pretrain_file)
+
+ pretrain_file_path = os.path.join(embed_root, embed_name, pretrain_file)
+
+ my_embed = text.embedding.TokenEmbedding.from_file(pretrain_file_path, elem_delim)
+
+ assert 'a' in my_embed
+ assert my_embed.unknown_token == '<unk>'
+ assert my_embed.unknown_token in my_embed
+
+ first_vec = my_embed.idx_to_vec[0]
+ assert_almost_equal(first_vec.asnumpy(), np.array([0, 0, 0, 0, 0]))
+
+ # Test __getitem__.
+ unk_vec = my_embed['A']
+ assert_almost_equal(unk_vec.asnumpy(), np.array([0, 0, 0, 0, 0]))
+
+ a_vec = my_embed['a']
+ assert_almost_equal(a_vec.asnumpy(), np.array([0.1, 0.2, 0.3, 0.4, 0.5]))
+
+ # Test __setitem__.
+ my_embed['a'] = nd.array([1, 2, 3, 4, 5])
+ assert_almost_equal(my_embed['a'].asnumpy(), np.array([1, 2, 3, 4, 5]))
+ assertRaises(ValueError, my_embed.__setitem__, 'unknown$$$', nd.array([0, 0, 0, 0, 0]))
+
+ assertRaises(AssertionError, my_embed.__setitem__, '<unk>',
+ nd.array([[0, 0, 0, 0, 0], [0, 0, 0, 0, 0]]))
+
+ assertRaises(AssertionError, my_embed.__setitem__, '<unk>', nd.array([0]))
+
+ unk_vecs = my_embed['<un...@unk>', '<un...@unk>']
+ assert_almost_equal(unk_vecs.asnumpy(), np.array([[0, 0, 0, 0, 0], [0, 0, 0, 0, 0]]))
+
+ # Test loaded unknown vectors.
+ pretrain_file2 = 'my_pretrain_file2.txt'
+ _mk_my_pretrain_file3(os.path.join(embed_root, embed_name), elem_delim, pretrain_file2)
+ pretrain_file_path = os.path.join(embed_root, embed_name, pretrain_file2)
+ my_embed2 = text.embedding.TokenEmbedding.from_file(pretrain_file_path, elem_delim,
+ init_unknown_vec=nd.ones,
+ unknown_token='<unk>')
+ unk_vec2 = my_embed2['<unk>']
+ assert_almost_equal(unk_vec2.asnumpy(), np.array([1, 1, 1, 1, 1]))
+ unk_vec2 = my_embed2['<un...@unk>']
+ assert_almost_equal(unk_vec2.asnumpy(), np.array([1, 1, 1, 1, 1]))
+
+ my_embed3 = text.embedding.TokenEmbedding.from_file(pretrain_file_path, elem_delim,
+ init_unknown_vec=nd.ones,
+ unknown_token='<unk1>')
+ unk_vec3 = my_embed3['<unk1>']
+ assert_almost_equal(unk_vec3.asnumpy(), np.array([1.1, 1.2, 1.3, 1.4, 1.5]))
+ unk_vec3 = my_embed3['<un...@unk>']
+ assert_almost_equal(unk_vec3.asnumpy(), np.array([1.1, 1.2, 1.3, 1.4, 1.5]))
+
+ # Test error handling.
+ invalid_pretrain_file = 'invalid_pretrain_file.txt'
+ _mk_my_invalid_pretrain_file(os.path.join(embed_root, embed_name), elem_delim,
+ invalid_pretrain_file)
+ pretrain_file_path = os.path.join(embed_root, embed_name, invalid_pretrain_file)
+ assertRaises(AssertionError, text.embedding.TokenEmbedding.from_file, pretrain_file_path,
+ elem_delim)
+
+ invalid_pretrain_file2 = 'invalid_pretrain_file2.txt'
+ _mk_my_invalid_pretrain_file2(os.path.join(embed_root, embed_name), elem_delim,
+ invalid_pretrain_file2)
+ pretrain_file_path = os.path.join(embed_root, embed_name, invalid_pretrain_file2)
+ assertRaises(AssertionError, text.embedding.TokenEmbedding.from_file, pretrain_file_path,
+ elem_delim)
+
+
+def test_embedding_get_and_pretrain_file_names():
+ assert len(text.embedding.get_file_names(embedding_name='fasttext')) == 327
+ assert len(text.embedding.get_file_names(embedding_name='glove')) == 10
+
+ reg = text.embedding.get_file_names(embedding_name=None)
+
+ assert len(reg['glove']) == 10
+ assert len(reg['fasttext']) == 327
+
+ assertRaises(KeyError, text.embedding.get_file_names, 'unknown$$')
+
+
+def test_vocab_set_embedding_with_one_custom_embedding():
+ embed_root = 'embedding'
+ embed_name = 'my_embed'
+ elem_delim = '\t'
+ pretrain_file = 'my_pretrain_file1.txt'
+
+ _mk_my_pretrain_file(os.path.join(embed_root, embed_name), elem_delim, pretrain_file)
+
+ pretrain_file_path = os.path.join(embed_root, embed_name, pretrain_file)
+
+ counter = Counter(['a', 'b', 'b', 'c', 'c', 'c', 'some_word$'])
+
+ v1 = text.Vocabulary(counter, max_size=None, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=['<pad>'])
+
+ e1 = text.embedding.TokenEmbedding.from_file(pretrain_file_path, elem_delim,
+ init_unknown_vec=nd.ones)
+
+ assert v1.embedding is None
+ v1.set_embedding(e1)
+ assert v1.embedding is not None
+
+ assert_almost_equal(v1.embedding.idx_to_vec.asnumpy(),
+ np.array([[1, 1, 1, 1, 1],
+ [1, 1, 1, 1, 1],
+ [1, 1, 1, 1, 1],
+ [0.6, 0.7, 0.8, 0.9, 1],
+ [0.1, 0.2, 0.3, 0.4, 0.5],
+ [1, 1, 1, 1, 1]])
+ )
+
+ assert_almost_equal(v1.embedding['c'].asnumpy(),
+ np.array([1, 1, 1, 1, 1])
+ )
+
+ assert_almost_equal(v1.embedding[['c']].asnumpy(),
+ np.array([[1, 1, 1, 1, 1]])
+ )
+
+ assert_almost_equal(v1.embedding[['a', 'not_exist']].asnumpy(),
+ np.array([[0.1, 0.2, 0.3, 0.4, 0.5],
+ [1, 1, 1, 1, 1]])
+ )
+
+ assert_almost_equal(v1.embedding[['a', 'b']].asnumpy(),
+ np.array([[0.1, 0.2, 0.3, 0.4, 0.5],
+ [0.6, 0.7, 0.8, 0.9, 1]])
+ )
+
+ assert_almost_equal(v1.embedding[['A', 'b']].asnumpy(),
+ np.array([[1, 1, 1, 1, 1],
+ [0.6, 0.7, 0.8, 0.9, 1]])
+ )
+
+ v1.embedding['a'] = nd.array([2, 2, 2, 2, 2])
+ v1.embedding['b'] = nd.array([3, 3, 3, 3, 3])
+
+ assert_almost_equal(v1.embedding.idx_to_vec.asnumpy(),
+ np.array([[1, 1, 1, 1, 1],
+ [1, 1, 1, 1, 1],
+ [1, 1, 1, 1, 1],
+ [3, 3, 3, 3, 3],
+ [2, 2, 2, 2, 2],
+ [1, 1, 1, 1, 1]])
+ )
+
+ v1.embedding['<unk>'] = nd.array([0, 0, 0, 0, 0])
+ assert_almost_equal(v1.embedding.idx_to_vec.asnumpy(),
+ np.array([[0, 0, 0, 0, 0],
+ [1, 1, 1, 1, 1],
+ [1, 1, 1, 1, 1],
+ [3, 3, 3, 3, 3],
+ [2, 2, 2, 2, 2],
+ [1, 1, 1, 1, 1]])
+ )
+ v1.embedding['<unk>'] = nd.array([10, 10, 10, 10, 10])
+ assert_almost_equal(v1.embedding.idx_to_vec.asnumpy(),
+ np.array([[10, 10, 10, 10, 10],
+ [1, 1, 1, 1, 1],
+ [1, 1, 1, 1, 1],
+ [3, 3, 3, 3, 3],
+ [2, 2, 2, 2, 2],
+ [1, 1, 1, 1, 1]])
+ )
+
+ v1.set_embedding(None)
+ assert v1.embedding is None
+
+
+def test_vocab_set_embedding_with_two_custom_embeddings():
+ embed_root = '.'
+ embed_name = 'my_embed'
+ elem_delim = '\t'
+ pretrain_file1 = 'my_pretrain_file1.txt'
+ pretrain_file2 = 'my_pretrain_file2.txt'
+
+ _mk_my_pretrain_file(os.path.join(embed_root, embed_name), elem_delim, pretrain_file1)
+ _mk_my_pretrain_file2(os.path.join(embed_root, embed_name), elem_delim, pretrain_file2)
+
+ pretrain_file_path1 = os.path.join(embed_root, embed_name, pretrain_file1)
+ pretrain_file_path2 = os.path.join(embed_root, embed_name, pretrain_file2)
+
+ my_embed1 = text.embedding.TokenEmbedding.from_file(pretrain_file_path1, elem_delim,
+ init_unknown_vec=nd.ones)
+ my_embed2 = text.embedding.TokenEmbedding.from_file(pretrain_file_path2, elem_delim)
+
+ counter = Counter(['a', 'b', 'b', 'c', 'c', 'c', 'some_word$'])
+
+ v1 = text.Vocabulary(counter, max_size=None, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=None)
+ v1.set_embedding(my_embed1, my_embed2)
+ assert v1.embedding is not None
+
+ assertRaises(AssertionError, v1.set_embedding, my_embed1, None, my_embed2)
+
+ assert_almost_equal(v1.embedding.idx_to_vec.asnumpy(),
+ np.array([[1, 1, 1, 1, 1, 0, 0, 0, 0, 0],
+ [1, 1, 1, 1, 1, 0.06, 0.07, 0.08, 0.09, 0.1],
+ [0.6, 0.7, 0.8, 0.9, 1, 0, 0, 0, 0, 0],
+ [0.1, 0.2, 0.3, 0.4, 0.5,
+ 0.01, 0.02, 0.03, 0.04, 0.05],
+ [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]])
+ )
+
+ assert_almost_equal(v1.embedding['c'].asnumpy(),
+ np.array([1, 1, 1, 1, 1, 0.06, 0.07, 0.08, 0.09, 0.1])
+ )
+
+ assert_almost_equal(v1.embedding[['b', 'not_exist']].asnumpy(),
+ np.array([[0.6, 0.7, 0.8, 0.9, 1, 0, 0, 0, 0, 0],
+ [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]])
+ )
+
+ v1.embedding['a'] = nd.array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2])
+ v1.embedding['b'] = nd.array([3, 3, 3, 3, 3, 3, 3, 3, 3, 3])
+
+ assert_almost_equal(v1.embedding.idx_to_vec.asnumpy(),
+ np.array([[1, 1, 1, 1, 1, 0, 0, 0, 0, 0],
+ [1, 1, 1, 1, 1, 0.06, 0.07, 0.08, 0.09, 0.1],
+ [3, 3, 3, 3, 3, 3, 3, 3, 3, 3],
+ [2, 2, 2, 2, 2, 2, 2, 2, 2, 2],
+ [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]])
+ )
+
+ # Test loaded unknown tokens
+ pretrain_file3 = 'my_pretrain_file3.txt'
+ pretrain_file4 = 'my_pretrain_file4.txt'
+
+ _mk_my_pretrain_file3(os.path.join(embed_root, embed_name), elem_delim, pretrain_file3)
+ _mk_my_pretrain_file4(os.path.join(embed_root, embed_name), elem_delim, pretrain_file4)
+
+ pretrain_file_path3 = os.path.join(embed_root, embed_name, pretrain_file3)
+ pretrain_file_path4 = os.path.join(embed_root, embed_name, pretrain_file4)
+
+ my_embed3 = text.embedding.TokenEmbedding.from_file(pretrain_file_path3, elem_delim,
+ init_unknown_vec=nd.ones, unknown_token='<unk1>')
+ my_embed4 = text.embedding.TokenEmbedding.from_file(pretrain_file_path4, elem_delim,
+ unknown_token='<unk2>')
+
+ v2 = text.Vocabulary(counter, max_size=None, min_freq=1, unknown_token='<unk>',
+ reserved_tokens=None)
+ v2.set_embedding(my_embed3, my_embed4)
+ assert_almost_equal(v2.embedding.idx_to_vec.asnumpy(),
+ np.array([[1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.11, 0.12, 0.13, 0.14, 0.15],
+ [1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.06, 0.07, 0.08, 0.09, 0.1],
+ [0.6, 0.7, 0.8, 0.9, 1,
+ 0.11, 0.12, 0.13, 0.14, 0.15],
+ [0.1, 0.2, 0.3, 0.4, 0.5,
+ 0.01, 0.02, 0.03, 0.04, 0.05],
+ [1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.11, 0.12, 0.13, 0.14, 0.15]])
+ )
+
+ v3 = text.Vocabulary(counter, max_size=None, min_freq=1, unknown_token='<unk1>',
+ reserved_tokens=None)
+ v3.set_embedding(my_embed3, my_embed4)
+ assert_almost_equal(v3.embedding.idx_to_vec.asnumpy(),
+ np.array([[1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.11, 0.12, 0.13, 0.14, 0.15],
+ [1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.06, 0.07, 0.08, 0.09, 0.1],
+ [0.6, 0.7, 0.8, 0.9, 1,
+ 0.11, 0.12, 0.13, 0.14, 0.15],
+ [0.1, 0.2, 0.3, 0.4, 0.5,
+ 0.01, 0.02, 0.03, 0.04, 0.05],
+ [1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.11, 0.12, 0.13, 0.14, 0.15]])
+ )
+
+ v4 = text.Vocabulary(counter, max_size=None, min_freq=1, unknown_token='<unk2>',
+ reserved_tokens=None)
+ v4.set_embedding(my_embed3, my_embed4)
+ assert_almost_equal(v4.embedding.idx_to_vec.asnumpy(),
+ np.array([[1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.11, 0.12, 0.13, 0.14, 0.15],
+ [1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.06, 0.07, 0.08, 0.09, 0.1],
+ [0.6, 0.7, 0.8, 0.9, 1,
+ 0.11, 0.12, 0.13, 0.14, 0.15],
+ [0.1, 0.2, 0.3, 0.4, 0.5,
+ 0.01, 0.02, 0.03, 0.04, 0.05],
+ [1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.11, 0.12, 0.13, 0.14, 0.15]])
+ )
+
+ counter2 = Counter(['b', 'b', 'c', 'c', 'c', 'some_word$'])
+
+ v5 = text.Vocabulary(counter2, max_size=None, min_freq=1, unknown_token='a',
+ reserved_tokens=None)
+ v5.set_embedding(my_embed3, my_embed4)
+ assert v5.embedding._token_to_idx == {'a': 0, 'c': 1, 'b': 2, 'some_word$': 3}
+ assert v5.embedding._idx_to_token == ['a', 'c', 'b', 'some_word$']
+ assert_almost_equal(v5.embedding.idx_to_vec.asnumpy(),
+ np.array([[1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.11, 0.12, 0.13, 0.14, 0.15],
+ [1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.06, 0.07, 0.08, 0.09, 0.1],
+ [0.6, 0.7, 0.8, 0.9, 1,
+ 0.11, 0.12, 0.13, 0.14, 0.15],
+ [1.1, 1.2, 1.3, 1.4, 1.5,
+ 0.11, 0.12, 0.13, 0.14, 0.15]])
+ )
+
+
+def test_download_embed():
+ @text.embedding.register
+ class Test(text.embedding.TokenEmbedding):
+ # 33 bytes.
+ pretrained_file_name_sha1 = \
+ {'embedding_test.vec': '29b9a6511cf4b5aae293c44a9ec1365b74f2a2f8'}
+ namespace = 'test'
+
+ def __init__(self, embedding_root='embedding', init_unknown_vec=nd.zeros, **kwargs):
+ file_name = 'embedding_test.vec'
+ Test._check_pretrained_file_names(file_name)
+
+ super(Test, self).__init__(**kwargs)
+
+ file_path = Test._get_pretrained_file(embedding_root, file_name)
+
+ self._load_embedding(file_path, ' ', init_unknown_vec)
+
+ test_embed = text.embedding.create('test')
+ assert_almost_equal(test_embed['hello'].asnumpy(), (nd.arange(5) + 1).asnumpy())
+ assert_almost_equal(test_embed['world'].asnumpy(), (nd.arange(5) + 6).asnumpy())
+ assert_almost_equal(test_embed['<unk>'].asnumpy(), nd.zeros((5,)).asnumpy())
+
+
+if __name__ == '__main__':
+ import nose
+ nose.runmodule()
--
To stop receiving notification emails like this one, please contact
zhasheng@apache.org.
[incubator-mxnet] 02/04: CorpusReader (#10061)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
commit 85f564e59ab26e4896225842d803d2213aa84234
Author: Sheng Zha <sz...@users.noreply.github.com>
AuthorDate: Sat Mar 10 14:40:50 2018 -0800
CorpusReader (#10061)
* corpus reader
* update
* delete pair
---
python/mxnet/gluon/data/datareader.py | 2 +-
python/mxnet/gluon/data/text/base.py | 99 +++++++++++++++++----------
python/mxnet/gluon/data/text/lm.py | 10 +--
python/mxnet/gluon/data/text/utils.py | 27 ++------
tests/python/unittest/test_gluon_data_text.py | 22 +++---
5 files changed, 87 insertions(+), 73 deletions(-)
diff --git a/python/mxnet/gluon/data/datareader.py b/python/mxnet/gluon/data/datareader.py
index 9b94ed4..b96575b 100644
--- a/python/mxnet/gluon/data/datareader.py
+++ b/python/mxnet/gluon/data/datareader.py
@@ -31,4 +31,4 @@ class DataReader(object):
raise NotImplementedError
def read_iter(self):
- return self.read()
+ return iter(self.read())
diff --git a/python/mxnet/gluon/data/text/base.py b/python/mxnet/gluon/data/text/base.py
index a9fa25b..6c4cf87 100644
--- a/python/mxnet/gluon/data/text/base.py
+++ b/python/mxnet/gluon/data/text/base.py
@@ -27,12 +27,56 @@ import os
from ..dataset import SimpleDataset
from ..datareader import DataReader
-from .utils import flatten_samples, collate, pair
+from .utils import flatten_samples, collate
-class WordLanguageReader(DataReader):
+class CorpusReader(DataReader):
"""Text reader that reads a whole corpus and produces a dataset based on provided
sample splitter and word tokenizer.
+ The returned dataset includes samples, each of which can either be a list of tokens if tokenizer
+ is specified, or a single string segment from the result of sample_splitter.
+
+ Parameters
+ ----------
+ filename : str
+ Path to the input text file.
+ encoding : str, default 'utf8'
+ File encoding format.
+ flatten : bool, default False
+ Whether to return all samples as flattened tokens. If True, each sample is a token.
+ sample_splitter : function, default str.splitlines
+ A function that splits the dataset string into samples.
+ tokenizer : function or None, default str.split
+ A function that splits each sample string into list of tokens. If None, raw samples are
+ returned according to `sample_splitter`.
+ """
+ def __init__(self, filename, encoding='utf8', flatten=False,
+ sample_splitter=lambda s: s.splitlines(),
+ tokenizer=lambda s: s.split()):
+ assert sample_splitter, 'sample_splitter must be specified.'
+ self._filename = os.path.expanduser(filename)
+ self._encoding = encoding
+ self._flatten = flatten
+ self._sample_splitter = sample_splitter
+ self._tokenizer = tokenizer
+
+ def read(self):
+ with io.open(self._filename, 'r', encoding=self._encoding) as fin:
+ content = fin.read()
+ samples = (s.strip() for s in self._sample_splitter(content))
+ if self._tokenizer:
+ samples = [self._tokenizer(s) for s in samples if s]
+ if self._flatten:
+ samples = flatten(samples)
+ else:
+ samples = [s for s in samples if s]
+ return SimpleDataset(samples)
+
+
+class WordLanguageReader(CorpusReader):
+ """Text reader that reads a whole corpus and produces a language modeling dataset given
+ the provided sample splitter and word tokenizer.
+
The returned dataset includes data (current word) and label (next word).
Parameters
@@ -46,8 +90,9 @@ class WordLanguageReader(DataReader):
tokenizer : function, default str.split
A function that splits each sample string into list of tokens.
seq_len : int or None
- The length of each of the samples. If None, samples are divided according to
- `sample_splitter` only, and may have variable lengths.
+ The length of each of the samples regardless of sample boundary.
+ If None, samples are divided according to `sample_splitter` only,
+ and may have variable lengths.
bos : str or None, default None
The token to add at the begining of each sentence. If None, nothing is added.
eos : str or None, default None
@@ -61,43 +106,27 @@ class WordLanguageReader(DataReader):
"""
def __init__(self, filename, encoding='utf8', sample_splitter=lambda s: s.splitlines(),
tokenizer=lambda s: s.split(), seq_len=None, bos=None, eos=None, pad=None):
- self._filename = os.path.expanduser(filename)
- self._encoding = encoding
- self._sample_splitter = sample_splitter
- self._tokenizer = tokenizer
-
- if bos and eos:
- def process(s):
- out = [bos]
- out.extend(s)
- out.append(eos)
- return pair(out)
- elif bos:
- def process(s):
- out = [bos]
- out.extend(s)
- return pair(out)
- elif eos:
- def process(s):
- s.append(eos)
- return pair(s)
- else:
- def process(s):
- return pair(s)
- self._process = process
+ assert tokenizer, "Tokenizer must be specified for reading word language model corpus."
+ super(WordLanguageReader, self).__init__(filename, encoding, False,
+ sample_splitter, tokenizer)
+ def process(s):
+ tokens = [bos] if bos else []
+ tokens.extend(s)
+ if eos:
+ tokens.append(eos)
+ return tokens
self._seq_len = seq_len
+ self._process = process
self._pad = pad
def read(self):
- with io.open(self._filename, 'r', encoding=self._encoding) as fin:
- content = fin.read()
- samples = [s.strip() for s in self._sample_splitter(content)]
- samples = [self._process(self._tokenizer(s)) for s in samples if s]
+ samples = super(WordLanguageReader, self).read()
+ samples = [self._process(s) for s in samples]
if self._seq_len:
samples = flatten_samples(samples)
if self._pad and len(samples) % self._seq_len:
pad_len = self._seq_len - len(samples) % self._seq_len
samples.extend([self._pad] * pad_len)
- samples = collate(samples, self._seq_len)
- samples = [list(zip(*s)) for s in samples]
- return SimpleDataset(samples)
+ samples = collate(samples, self._seq_len, 1)
+
+ return SimpleDataset(samples).transform(lambda x: (x[:-1], x[1:]))
diff --git a/python/mxnet/gluon/data/text/lm.py b/python/mxnet/gluon/data/text/lm.py
index 63ad004..ed61f98 100644
--- a/python/mxnet/gluon/data/text/lm.py
+++ b/python/mxnet/gluon/data/text/lm.py
@@ -78,8 +78,9 @@ class WikiText2(_WikiText):
segment : str, default 'train'
Dataset segment. Options are 'train', 'val', 'test'.
seq_len : int or None, default 35
- The number of tokens for each sample. If specified, samples are collated by length.
- If None, each sample is of variable length.
+ The length of each of the samples regardless of sample boundary.
+ If None, samples are divided according to `sample_splitter` only,
+ and may have variable lengths.
bos : str or None, default None
The token to add at the begining of each sentence. If None, nothing is added.
eos : str or None, default '<eos>'
@@ -119,8 +120,9 @@ class WikiText103(_WikiText):
segment : str, default 'train'
Dataset segment. Options are 'train', 'val', 'test'.
seq_len : int or None, default 35
- The number of tokens for each sample. If specified, samples are collated by length.
- If None, each sample is of variable length.
+ The length of each of the samples regardless of sample boundary.
+ If None, samples are divided according to `sample_splitter` only,
+ and may have variable lengths.
bos : str or None, default None
The token to add at the begining of each sentence. If None, nothing is added.
eos : str or None, default '<eos>'
diff --git a/python/mxnet/gluon/data/text/utils.py b/python/mxnet/gluon/data/text/utils.py
index 68fa519..057e843 100644
--- a/python/mxnet/gluon/data/text/utils.py
+++ b/python/mxnet/gluon/data/text/utils.py
@@ -20,10 +20,6 @@
"""Utility functions."""
-import sys
-if sys.version_info[0] < 3:
- from itertools import izip as zip
-
def flatten_samples(samples):
"""Flatten list of list of tokens into a single flattened list of tokens.
@@ -38,7 +34,7 @@ def flatten_samples(samples):
"""
return [token for sample in samples for token in sample if token]
-def collate(flat_sample, seq_len):
+def collate(flat_sample, seq_len, overlap=0):
"""Collate a flat list of tokens into list of list of tokens, with each
inner list's length equal to the specified `seq_len`.
@@ -48,26 +44,13 @@ def collate(flat_sample, seq_len):
A flat list of tokens.
seq_len : int
The length of each of the samples.
+ overlap : int, default 0
+ The extra number of items in current sample that should overlap with the
+ next sample.
Returns
-------
List of samples, each of which has length equal to `seq_len`.
"""
num_samples = len(flat_sample) // seq_len
- return [flat_sample[i*seq_len:(i+1)*seq_len] for i in range(num_samples)]
-
-def pair(sample):
- """Produce tuples of tokens from a list of tokens, with current token as the first
- element and the next token as the second element.
-
- Parameters
- ----------
- sample : list of object
- A list of tokens.
-
- Returns
- -------
- Generator of tuples, each of which has current token as the first element and the next token
- as the second element.
- """
- return list(zip(sample[:-1], sample[1:]))
+ return [flat_sample[i*seq_len:((i+1)*seq_len+overlap)] for i in range(num_samples)]
diff --git a/tests/python/unittest/test_gluon_data_text.py b/tests/python/unittest/test_gluon_data_text.py
index a7c083f..49a5988 100644
--- a/tests/python/unittest/test_gluon_data_text.py
+++ b/tests/python/unittest/test_gluon_data_text.py
@@ -29,20 +29,20 @@ def test_wikitext2():
val = data.text.lm.WikiText2(root='data/wikitext-2', segment='val')
test = data.text.lm.WikiText2(root='data/wikitext-2', segment='test')
train_freq, val_freq, test_freq = [get_frequencies(x) for x in [train, val, test]]
- assert len(train) == 58626
- assert len(train_freq) == 33278
- assert len(val) == 6112
- assert len(val_freq) == 13778
- assert len(test) == 6892
- assert len(test_freq) == 14144
- assert test_freq['English'] == 35
- assert len(train[0][0]) == 35
+ assert len(train) == 59306, len(train)
+ assert len(train_freq) == 33279, len(train_freq)
+ assert len(val) == 6182, len(val)
+ assert len(val_freq) == 13778, len(val_freq)
+ assert len(test) == 6975, len(test)
+ assert len(test_freq) == 14144, len(test_freq)
+ assert test_freq['English'] == 33, test_freq['English']
+ assert len(train[0][0]) == 35, len(train[0][0])
test_no_pad = data.text.lm.WikiText2(root='data/wikitext-2', segment='test', pad=None)
- assert len(test_no_pad) == 6891
+ assert len(test_no_pad) == 6974, len(test_no_pad)
train_paragraphs = data.text.lm.WikiText2(root='data/wikitext-2', segment='train', seq_len=None)
- assert len(train_paragraphs) == 23767
- assert len(train_paragraphs[0][0]) != 35
+ assert len(train_paragraphs) == 23767, len(train_paragraphs)
+ assert len(train_paragraphs[0][0]) != 35, len(train_paragraphs[0][0])
if __name__ == '__main__':
--
To stop receiving notification emails like this one, please contact
zhasheng@apache.org.
[incubator-mxnet] 03/04: fix CorpusReader (#10079)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
commit 911ff3bf0cbca4dc626edfcbd63c87388d23b153
Author: Xingjian Shi <xs...@ust.hk>
AuthorDate: Mon Mar 12 15:02:28 2018 -0700
fix CorpusReader (#10079)
try to accelerate
update
Revert "update"
This reverts commit 7d409b54e0fdb006e6d8be50749a90e24b12715d.
Revert "try to accelerate"
This reverts commit b7d93f9184323909ae59d77445d45588ea76aaa5.
---
python/mxnet/gluon/data/text/base.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/python/mxnet/gluon/data/text/base.py b/python/mxnet/gluon/data/text/base.py
index 6c4cf87..3c02424 100644
--- a/python/mxnet/gluon/data/text/base.py
+++ b/python/mxnet/gluon/data/text/base.py
@@ -20,7 +20,7 @@
"""Base classes for text datasets and readers."""
-__all__ = ['WordLanguageReader']
+__all__ = ['CorpusReader', 'WordLanguageReader']
import io
import os
@@ -67,7 +67,7 @@ class CorpusReader(DataReader):
if self._tokenizer:
samples = [self._tokenizer(s) for s in samples if s]
if self._flatten:
- samples = flatten(samples)
+ samples = flatten_samples(samples)
else:
samples = [s for s in samples if s]
return SimpleDataset(samples)
--
To stop receiving notification emails like this one, please contact
zhasheng@apache.org.
[incubator-mxnet] 01/04: gluon language modeling dataset and text
token reader (#9986)
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhasheng pushed a commit to branch nlp_toolkit
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
commit 929f5ef630f8f9e5cf6bef6b419282932f531fd0
Author: Sheng Zha <sz...@users.noreply.github.com>
AuthorDate: Thu Mar 8 16:49:51 2018 -0500
gluon language modeling dataset and text token reader (#9986)
* language modeling dataset and text token reader.
* update
* add padding
* update bos insert
* update doc
---
example/gluon/word_language_model/train.py | 44 ++++---
python/mxnet/gluon/data/__init__.py | 2 +
.../gluon/data/{__init__.py => datareader.py} | 18 ++-
python/mxnet/gluon/data/{ => text}/__init__.py | 10 +-
.../gluon/data/{__init__.py => text/_constants.py} | 12 +-
python/mxnet/gluon/data/text/base.py | 103 +++++++++++++++
python/mxnet/gluon/data/text/lm.py | 145 +++++++++++++++++++++
python/mxnet/gluon/data/text/utils.py | 73 +++++++++++
tests/python/unittest/test_gluon_data_text.py | 50 +++++++
9 files changed, 420 insertions(+), 37 deletions(-)
diff --git a/example/gluon/word_language_model/train.py b/example/gluon/word_language_model/train.py
index b69fd17..c732393 100644
--- a/example/gluon/word_language_model/train.py
+++ b/example/gluon/word_language_model/train.py
@@ -16,13 +16,13 @@
# under the License.
import argparse
+import collections
import time
import math
import mxnet as mx
-from mxnet import gluon, autograd
-from mxnet.gluon import contrib
+from mxnet import gluon, autograd, contrib
+from mxnet.gluon import data
import model
-import data
parser = argparse.ArgumentParser(description='MXNet Autograd RNN/LSTM Language Model on Wikitext-2.')
parser.add_argument('--model', type=str, default='lstm',
@@ -71,32 +71,40 @@ if args.cuda:
else:
context = mx.cpu(0)
-train_dataset = contrib.data.text.WikiText2('./data', 'train', seq_len=args.bptt)
-vocab = train_dataset.vocabulary
-val_dataset, test_dataset = [contrib.data.text.WikiText2('./data', segment,
- vocab=vocab,
- seq_len=args.bptt)
- for segment in ['validation', 'test']]
+train_dataset = data.text.lm.WikiText2('./data', 'train', seq_len=args.bptt,
+ eos='<eos>')
+
+def get_frequencies(dataset):
+ return collections.Counter(x for tup in dataset for x in tup[0] if x)
+
+vocab = contrib.text.vocab.Vocabulary(get_frequencies(train_dataset))
+def index_tokens(data, label):
+ return vocab.to_indices(data), vocab.to_indices(label)
+
+val_dataset, test_dataset = [data.text.lm.WikiText2('./data', segment,
+ seq_len=args.bptt,
+ eos='<eos>')
+ for segment in ['val', 'test']]
nbatch_train = len(train_dataset) // args.batch_size
-train_data = gluon.data.DataLoader(train_dataset,
+train_data = gluon.data.DataLoader(train_dataset.transform(index_tokens),
batch_size=args.batch_size,
- sampler=contrib.data.IntervalSampler(len(train_dataset),
- nbatch_train),
+ sampler=gluon.contrib.data.IntervalSampler(len(train_dataset),
+ nbatch_train),
last_batch='discard')
nbatch_val = len(val_dataset) // args.batch_size
-val_data = gluon.data.DataLoader(val_dataset,
+val_data = gluon.data.DataLoader(val_dataset.transform(index_tokens),
batch_size=args.batch_size,
- sampler=contrib.data.IntervalSampler(len(val_dataset),
- nbatch_val),
+ sampler=gluon.contrib.data.IntervalSampler(len(val_dataset),
+ nbatch_val),
last_batch='discard')
nbatch_test = len(test_dataset) // args.batch_size
-test_data = gluon.data.DataLoader(test_dataset,
+test_data = gluon.data.DataLoader(test_dataset.transform(index_tokens),
batch_size=args.batch_size,
- sampler=contrib.data.IntervalSampler(len(test_dataset),
- nbatch_test),
+ sampler=gluon.contrib.data.IntervalSampler(len(test_dataset),
+ nbatch_test),
last_batch='discard')
diff --git a/python/mxnet/gluon/data/__init__.py b/python/mxnet/gluon/data/__init__.py
index 23ae3e9..14a0e46 100644
--- a/python/mxnet/gluon/data/__init__.py
+++ b/python/mxnet/gluon/data/__init__.py
@@ -26,3 +26,5 @@ from .sampler import *
from .dataloader import *
from . import vision
+
+from . import text
diff --git a/python/mxnet/gluon/data/__init__.py b/python/mxnet/gluon/data/datareader.py
similarity index 64%
copy from python/mxnet/gluon/data/__init__.py
copy to python/mxnet/gluon/data/datareader.py
index 23ae3e9..9b94ed4 100644
--- a/python/mxnet/gluon/data/__init__.py
+++ b/python/mxnet/gluon/data/datareader.py
@@ -16,13 +16,19 @@
# under the License.
# coding: utf-8
-# pylint: disable=wildcard-import
-"""Dataset utilities."""
+# pylint: disable=
+"""Dataset reader."""
+__all__ = ['DataReader']
-from .dataset import *
+class DataReader(object):
+ """Abstract datareader class. Data reader handles I/O and produces raw samples for a dataset.
-from .sampler import *
+ Subclasses need to override `read` that returns a Dataset (and optionally `read_iter` that
+ returns an iterable for large files).
+ """
-from .dataloader import *
+ def read(self):
+ raise NotImplementedError
-from . import vision
+ def read_iter(self):
+ return self.read()
diff --git a/python/mxnet/gluon/data/__init__.py b/python/mxnet/gluon/data/text/__init__.py
similarity index 87%
copy from python/mxnet/gluon/data/__init__.py
copy to python/mxnet/gluon/data/text/__init__.py
index 23ae3e9..5a6c097 100644
--- a/python/mxnet/gluon/data/__init__.py
+++ b/python/mxnet/gluon/data/text/__init__.py
@@ -17,12 +17,8 @@
# coding: utf-8
# pylint: disable=wildcard-import
-"""Dataset utilities."""
+"""Text utilities."""
-from .dataset import *
+from .base import *
-from .sampler import *
-
-from .dataloader import *
-
-from . import vision
+from . import lm
diff --git a/python/mxnet/gluon/data/__init__.py b/python/mxnet/gluon/data/text/_constants.py
similarity index 84%
copy from python/mxnet/gluon/data/__init__.py
copy to python/mxnet/gluon/data/text/_constants.py
index 23ae3e9..9aeac09 100644
--- a/python/mxnet/gluon/data/__init__.py
+++ b/python/mxnet/gluon/data/text/_constants.py
@@ -16,13 +16,13 @@
# under the License.
# coding: utf-8
-# pylint: disable=wildcard-import
-"""Dataset utilities."""
-from .dataset import *
+"""Constants relevant to text processing."""
-from .sampler import *
+UNK_TOKEN = '<unk>'
-from .dataloader import *
+BOS_TOKEN = '<bos>'
-from . import vision
+EOS_TOKEN = '<eos>'
+
+PAD_TOKEN = '<pad>'
diff --git a/python/mxnet/gluon/data/text/base.py b/python/mxnet/gluon/data/text/base.py
new file mode 100644
index 0000000..a9fa25b
--- /dev/null
+++ b/python/mxnet/gluon/data/text/base.py
@@ -0,0 +1,103 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+
+"""Base classes for text datasets and readers."""
+
+__all__ = ['WordLanguageReader']
+
+import io
+import os
+
+from ..dataset import SimpleDataset
+from ..datareader import DataReader
+from .utils import flatten_samples, collate, pair
+
+class WordLanguageReader(DataReader):
+ """Text reader that reads a whole corpus and produces a dataset based on provided
+ sample splitter and word tokenizer.
+
+ The returned dataset includes data (current word) and label (next word).
+
+ Parameters
+ ----------
+ filename : str
+ Path to the input text file.
+ encoding : str, default 'utf8'
+ File encoding format.
+ sample_splitter : function, default str.splitlines
+ A function that splits the dataset string into samples.
+ tokenizer : function, default str.split
+ A function that splits each sample string into list of tokens.
+ seq_len : int or None
+ The length of each of the samples. If None, samples are divided according to
+ `sample_splitter` only, and may have variable lengths.
+ bos : str or None, default None
+ The token to add at the begining of each sentence. If None, nothing is added.
+ eos : str or None, default None
+ The token to add at the end of each sentence. If None, nothing is added.
+ pad : str or None, default None
+ The padding token to add at the end of dataset if `seq_len` is specified and the total
+ number of tokens in the corpus don't evenly divide `seq_len`. If pad is None or seq_len
+ is None, no padding is added. Otherwise, padding token is added to the last sample if
+ its length is less than `seq_len`. If `pad` is None and `seq_len` is specified, the last
+ sample is discarded if it's shorter than `seq_len`.
+ """
+ def __init__(self, filename, encoding='utf8', sample_splitter=lambda s: s.splitlines(),
+ tokenizer=lambda s: s.split(), seq_len=None, bos=None, eos=None, pad=None):
+ self._filename = os.path.expanduser(filename)
+ self._encoding = encoding
+ self._sample_splitter = sample_splitter
+ self._tokenizer = tokenizer
+
+ if bos and eos:
+ def process(s):
+ out = [bos]
+ out.extend(s)
+ out.append(eos)
+ return pair(out)
+ elif bos:
+ def process(s):
+ out = [bos]
+ out.extend(s)
+ return pair(out)
+ elif eos:
+ def process(s):
+ s.append(eos)
+ return pair(s)
+ else:
+ def process(s):
+ return pair(s)
+ self._process = process
+ self._seq_len = seq_len
+ self._pad = pad
+
+ def read(self):
+ with io.open(self._filename, 'r', encoding=self._encoding) as fin:
+ content = fin.read()
+ samples = [s.strip() for s in self._sample_splitter(content)]
+ samples = [self._process(self._tokenizer(s)) for s in samples if s]
+ if self._seq_len:
+ samples = flatten_samples(samples)
+ if self._pad and len(samples) % self._seq_len:
+ pad_len = self._seq_len - len(samples) % self._seq_len
+ samples.extend([self._pad] * pad_len)
+ samples = collate(samples, self._seq_len)
+ samples = [list(zip(*s)) for s in samples]
+ return SimpleDataset(samples)
diff --git a/python/mxnet/gluon/data/text/lm.py b/python/mxnet/gluon/data/text/lm.py
new file mode 100644
index 0000000..63ad004
--- /dev/null
+++ b/python/mxnet/gluon/data/text/lm.py
@@ -0,0 +1,145 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+"""Language model datasets."""
+
+__all__ = ['WikiText2', 'WikiText103']
+
+import os
+import zipfile
+import shutil
+
+from ..dataset import SimpleDataset
+from .base import WordLanguageReader
+from . import _constants as C
+from ...utils import download, check_sha1, _get_repo_file_url
+
+
+class _WikiText(SimpleDataset):
+ def __init__(self, root, namespace, seq_len, bos, eos, pad):
+ root = os.path.expanduser(root)
+ if not os.path.isdir(root):
+ os.makedirs(root)
+ self._root = root
+ self._namespace = 'gluon/dataset/{}'.format(namespace)
+ reader = WordLanguageReader(self._get_data(),
+ seq_len=seq_len, bos=bos, eos=eos, pad=pad)
+ super(_WikiText, self).__init__(reader.read())
+
+ def _get_data(self):
+ archive_file_name, archive_hash = self._archive_file
+ data_file_name, data_hash = self._data_file[self._segment]
+ root = self._root
+ path = os.path.join(root, data_file_name)
+ if not os.path.exists(path) or not check_sha1(path, data_hash):
+ downloaded_file_path = download(_get_repo_file_url(self._namespace, archive_file_name),
+ path=root,
+ sha1_hash=archive_hash)
+
+ with zipfile.ZipFile(downloaded_file_path, 'r') as zf:
+ for member in zf.namelist():
+ filename = os.path.basename(member)
+ if filename:
+ dest = os.path.join(root, filename)
+ with zf.open(member) as source, \
+ open(dest, "wb") as target:
+ shutil.copyfileobj(source, target)
+ return path
+
+
+class WikiText2(_WikiText):
+ """WikiText-2 word-level dataset for language modeling, from Salesforce research.
+
+ From
+ https://einstein.ai/research/the-wikitext-long-term-dependency-language-modeling-dataset
+
+ License: Creative Commons Attribution-ShareAlike
+
+ Parameters
+ ----------
+ root : str, default '~/.mxnet/datasets/wikitext-2'
+ Path to temp folder for storing data.
+ segment : str, default 'train'
+ Dataset segment. Options are 'train', 'val', 'test'.
+ seq_len : int or None, default 35
+ The number of tokens for each sample. If specified, samples are collated by length.
+ If None, each sample is of variable length.
+ bos : str or None, default None
+ The token to add at the begining of each sentence. If None, nothing is added.
+ eos : str or None, default '<eos>'
+ The token to add at the end of each sentence. If None, nothing is added.
+ pad : str or None, default '<pad>'
+ The padding token to add at the end of dataset if `seq_len` is specified and the total
+ number of tokens in the corpus don't evenly divide `seq_len`. If pad is None or seq_len
+ is None, no padding is added. Otherwise, padding token is added to the last sample if
+ its length is less than `seq_len`. If `pad` is None and `seq_len` is specified, the last
+ sample is discarded if it's shorter than `seq_len`.
+ """
+ def __init__(self, root=os.path.join('~', '.mxnet', 'datasets', 'wikitext-2'),
+ segment='train', seq_len=35, bos=None, eos=C.EOS_TOKEN, pad=C.PAD_TOKEN):
+ self._archive_file = ('wikitext-2-v1.zip', '3c914d17d80b1459be871a5039ac23e752a53cbe')
+ self._data_file = {'train': ('wiki.train.tokens',
+ '863f29c46ef9d167fff4940ec821195882fe29d1'),
+ 'val': ('wiki.valid.tokens',
+ '0418625c8b4da6e4b5c7a0b9e78d4ae8f7ee5422'),
+ 'test': ('wiki.test.tokens',
+ 'c7b8ce0aa086fb34dab808c5c49224211eb2b172')}
+ self._segment = segment
+ super(WikiText2, self).__init__(root, 'wikitext-2', seq_len, bos, eos, pad)
+
+
+class WikiText103(_WikiText):
+ """WikiText-103 word-level dataset for language modeling, from Salesforce research.
+
+ From
+ https://einstein.ai/research/the-wikitext-long-term-dependency-language-modeling-dataset
+
+ License: Creative Commons Attribution-ShareAlike
+
+ Parameters
+ ----------
+ root : str, default '~/.mxnet/datasets/wikitext-103'
+ Path to temp folder for storing data.
+ segment : str, default 'train'
+ Dataset segment. Options are 'train', 'val', 'test'.
+ seq_len : int or None, default 35
+ The number of tokens for each sample. If specified, samples are collated by length.
+ If None, each sample is of variable length.
+ bos : str or None, default None
+ The token to add at the begining of each sentence. If None, nothing is added.
+ eos : str or None, default '<eos>'
+ The token to add at the end of each sentence. If None, nothing is added.
+ pad : str or None, default '<pad>'
+ The padding token to add at the end of dataset if `seq_len` is specified and the total
+ number of tokens in the corpus don't evenly divide `seq_len`. If pad is None or seq_len
+ is None, no padding is added. Otherwise, padding token is added to the last sample if
+ its length is less than `seq_len`. If `pad` is None and `seq_len` is specified, the last
+ sample is discarded if it's shorter than `seq_len`.
+ """
+ def __init__(self, root=os.path.join('~', '.mxnet', 'datasets', 'wikitext-103'),
+ segment='train', seq_len=35, bos=None, eos=C.EOS_TOKEN, pad=C.PAD_TOKEN):
+ self._archive_file = ('wikitext-103-v1.zip', '0aec09a7537b58d4bb65362fee27650eeaba625a')
+ self._data_file = {'train': ('wiki.train.tokens',
+ 'b7497e2dfe77e72cfef5e3dbc61b7b53712ac211'),
+ 'val': ('wiki.valid.tokens',
+ 'c326ac59dc587676d58c422eb8a03e119582f92b'),
+ 'test': ('wiki.test.tokens',
+ '8a5befc548865cec54ed4273cf87dbbad60d1e47')}
+ self._segment = segment
+ super(WikiText103, self).__init__(root, 'wikitext-103', seq_len, bos, eos, pad)
diff --git a/python/mxnet/gluon/data/text/utils.py b/python/mxnet/gluon/data/text/utils.py
new file mode 100644
index 0000000..68fa519
--- /dev/null
+++ b/python/mxnet/gluon/data/text/utils.py
@@ -0,0 +1,73 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=redefined-builtin
+
+"""Utility functions."""
+
+import sys
+if sys.version_info[0] < 3:
+ from itertools import izip as zip
+
+def flatten_samples(samples):
+ """Flatten list of list of tokens into a single flattened list of tokens.
+
+ Parameters
+ ----------
+ samples : list of list of object
+ List of samples, each of which is a list of tokens.
+
+ Returns
+ -------
+ Flattened list of tokens.
+ """
+ return [token for sample in samples for token in sample if token]
+
+def collate(flat_sample, seq_len):
+ """Collate a flat list of tokens into list of list of tokens, with each
+ inner list's length equal to the specified `seq_len`.
+
+ Parameters
+ ----------
+ flat_sample : list of object
+ A flat list of tokens.
+ seq_len : int
+ The length of each of the samples.
+
+ Returns
+ -------
+ List of samples, each of which has length equal to `seq_len`.
+ """
+ num_samples = len(flat_sample) // seq_len
+ return [flat_sample[i*seq_len:(i+1)*seq_len] for i in range(num_samples)]
+
+def pair(sample):
+ """Produce tuples of tokens from a list of tokens, with current token as the first
+ element and the next token as the second element.
+
+ Parameters
+ ----------
+ sample : list of object
+ A list of tokens.
+
+ Returns
+ -------
+ Generator of tuples, each of which has current token as the first element and the next token
+ as the second element.
+ """
+ return list(zip(sample[:-1], sample[1:]))
diff --git a/tests/python/unittest/test_gluon_data_text.py b/tests/python/unittest/test_gluon_data_text.py
new file mode 100644
index 0000000..a7c083f
--- /dev/null
+++ b/tests/python/unittest/test_gluon_data_text.py
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import print_function
+import collections
+import mxnet as mx
+from mxnet.gluon import nn, data
+from common import setup_module, with_seed
+
+def get_frequencies(dataset):
+ return collections.Counter(x for tup in dataset for x in tup[0]+tup[1][-1:])
+
+def test_wikitext2():
+ train = data.text.lm.WikiText2(root='data/wikitext-2', segment='train')
+ val = data.text.lm.WikiText2(root='data/wikitext-2', segment='val')
+ test = data.text.lm.WikiText2(root='data/wikitext-2', segment='test')
+ train_freq, val_freq, test_freq = [get_frequencies(x) for x in [train, val, test]]
+ assert len(train) == 58626
+ assert len(train_freq) == 33278
+ assert len(val) == 6112
+ assert len(val_freq) == 13778
+ assert len(test) == 6892
+ assert len(test_freq) == 14144
+ assert test_freq['English'] == 35
+ assert len(train[0][0]) == 35
+ test_no_pad = data.text.lm.WikiText2(root='data/wikitext-2', segment='test', pad=None)
+ assert len(test_no_pad) == 6891
+
+ train_paragraphs = data.text.lm.WikiText2(root='data/wikitext-2', segment='train', seq_len=None)
+ assert len(train_paragraphs) == 23767
+ assert len(train_paragraphs[0][0]) != 35
+
+
+if __name__ == '__main__':
+ import nose
+ nose.runmodule()
--
To stop receiving notification emails like this one, please contact
zhasheng@apache.org.