You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by se...@apache.org on 2018/03/06 20:17:35 UTC
[incubator-mxnet] branch master updated: [Perl] Sparse feature.
(#9988)
This is an automated email from the ASF dual-hosted git repository.
sergeykolychev pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/master by this push:
new 2a9c7d9 [Perl] Sparse feature. (#9988)
2a9c7d9 is described below
commit 2a9c7d9c7e1f2cdb49e029aef9b20e61afacbe46
Author: Sergey Kolychev <se...@gmail.com>
AuthorDate: Tue Mar 6 12:17:32 2018 -0800
[Perl] Sparse feature. (#9988)
* sparse feature.
* moved myself to the list of commiters.
* changed ImperativeInvoke to ImperativeInvokeEx.
* updated MNIFEST
* addressed code review
* recent python's kvstore sparse changes.
* using PDL.
* fix docs, trigger tests.
---
CONTRIBUTORS.md | 4 +-
docker/install/perl.sh | 2 +-
docker_multiarch/Dockerfile.run.ubuntu-17.04.perl | 2 +-
docs/api/perl/index.md | 5 +-
docs/install/build_from_source.md | 2 +-
docs/install/osx_setup.md | 2 +-
docs/install/ubuntu_setup.md | 2 +-
perl-package/AI-MXNet/Changes | 3 +
perl-package/AI-MXNet/MANIFEST | 10 +
perl-package/AI-MXNet/META.json | 9 +-
perl-package/AI-MXNet/META.yml | 7 +-
perl-package/AI-MXNet/Makefile.PL | 12 +-
perl-package/AI-MXNet/README | 2 +-
.../examples/sparse/matrix_factorization/README.md | 9 +
.../sparse/matrix_factorization/get_data.sh | 19 +-
.../examples/sparse/matrix_factorization/train.pl | 184 +++
.../AI-MXNet/examples/sparse/wide_deep/README.md | 7 +
.../AI-MXNet/examples/sparse/wide_deep/get_data.sh | 18 +-
.../AI-MXNet/examples/sparse/wide_deep/train.pl | 301 +++++
perl-package/AI-MXNet/lib/AI/MXNet.pm | 5 +-
perl-package/AI-MXNet/lib/AI/MXNet/AutoGrad.pm | 2 +-
perl-package/AI-MXNet/lib/AI/MXNet/Base.pm | 131 +-
perl-package/AI-MXNet/lib/AI/MXNet/CachedOp.pm | 4 +-
perl-package/AI-MXNet/lib/AI/MXNet/Executor.pm | 2 +-
perl-package/AI-MXNet/lib/AI/MXNet/IO.pm | 23 +-
perl-package/AI-MXNet/lib/AI/MXNet/KVStore.pm | 168 ++-
perl-package/AI-MXNet/lib/AI/MXNet/Module.pm | 8 +-
perl-package/AI-MXNet/lib/AI/MXNet/Module/Base.pm | 6 +-
perl-package/AI-MXNet/lib/AI/MXNet/Monitor.pm | 2 +-
perl-package/AI-MXNet/lib/AI/MXNet/NDArray.pm | 96 +-
perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Base.pm | 43 +-
.../AI-MXNet/lib/AI/MXNet/NDArray/Sparse.pm | 1342 ++++++++++++++++++++
perl-package/AI-MXNet/lib/AI/MXNet/Optimizer.pm | 743 +++++++++--
perl-package/AI-MXNet/lib/AI/MXNet/Symbol.pm | 40 +-
perl-package/AI-MXNet/lib/AI/MXNet/Symbol/Base.pm | 2 +-
.../AI-MXNet/lib/AI/MXNet/Symbol/Sparse.pm | 21 +-
perl-package/AI-MXNet/lib/AI/MXNet/TestUtils.pm | 456 ++++++-
perl-package/AI-MXNet/lib/AI/MXNet/Types.pm | 12 +-
perl-package/AI-MXNet/t/test_autograd.t | 105 +-
perl-package/AI-MXNet/t/test_init.t | 24 +-
perl-package/AI-MXNet/t/test_io.t | 48 +-
perl-package/AI-MXNet/t/test_kvstore.t | 93 +-
perl-package/AI-MXNet/t/test_module.t | 257 +++-
perl-package/AI-MXNet/t/test_optimizers.t | 629 ++++++++-
perl-package/AI-MXNet/t/test_sparse_ndarray.t | 1005 +++++++++++++++
perl-package/AI-MXNetCAPI/Changes | 3 +
perl-package/AI-MXNetCAPI/META.json | 2 +-
perl-package/AI-MXNetCAPI/META.yml | 2 +-
perl-package/AI-MXNetCAPI/README | 2 +-
perl-package/AI-MXNetCAPI/lib/AI/MXNetCAPI.pm | 2 +-
perl-package/AI-MXNetCAPI/mxnet.i | 216 +++-
perl-package/AI-MXNetCAPI/mxnet_typemaps.i | 45 +-
perl-package/AI-NNVMCAPI/Changes | 3 +
perl-package/AI-NNVMCAPI/META.json | 2 +-
perl-package/AI-NNVMCAPI/META.yml | 2 +-
perl-package/AI-NNVMCAPI/README | 2 +-
perl-package/AI-NNVMCAPI/lib/AI/NNVMCAPI.pm | 2 +-
perl-package/README.md | 5 +-
tests/ci_build/install/ubuntu_install_perl.sh | 2 +-
tests/travis/setup.sh | 4 +-
60 files changed, 5818 insertions(+), 343 deletions(-)
diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md
index c09c559..d9f1273 100644
--- a/CONTRIBUTORS.md
+++ b/CONTRIBUTORS.md
@@ -39,6 +39,8 @@ The committers are the granted write access to the project.
- Zixuan is one of major maintainers of mxnet scala package.
* [Yuan Tang](https://github.com/terrytangyuan)
- Yuan is one of major maintainers of mxnet scala package.
+* [Sergey Kolychev](https://github.com/sergeykolychev)
+ - Sergey is original author and current maintainer of Perl5 interface.
### Become a Committer
MXNet is a opensource project and we are actively looking for new committers
@@ -127,8 +129,6 @@ List of Contributors
* [Freddy Chua](https://github.com/freddycct)
* [Jie Zhang](https://github.com/luoyetx)
* [Leonard Lausen](https://github.com/leezu)
-* [Sergey Kolychev](https://github.com/sergeykolychev)
- - Sergey is original author and current maintainer of Perl5 interface.
* [Robert Stone](https://github.com/tlby)
* [Pedro Larroy](https://github.com/larroy)
* [Jun Wu](https://github.com/reminisce)
diff --git a/docker/install/perl.sh b/docker/install/perl.sh
index af49952..29313c7 100755
--- a/docker/install/perl.sh
+++ b/docker/install/perl.sh
@@ -19,4 +19,4 @@
# install libraries for mxnet's perl package on ubuntu
apt-get update && apt-get install -y libmouse-perl pdl cpanminus swig libgraphviz-perl
-cpanm -q Function::Parameters Hash::Ordered
+cpanm -q Function::Parameters Hash::Ordered PDL::CCS
diff --git a/docker_multiarch/Dockerfile.run.ubuntu-17.04.perl b/docker_multiarch/Dockerfile.run.ubuntu-17.04.perl
index a1a637d..17c5d8d 100644
--- a/docker_multiarch/Dockerfile.run.ubuntu-17.04.perl
+++ b/docker_multiarch/Dockerfile.run.ubuntu-17.04.perl
@@ -11,7 +11,7 @@ RUN apt-get install -y\
libmouse-perl pdl cpanminus swig libgraphviz-perl
RUN rm -rf /var/lib/apt/lists/*
-RUN cpanm -q Function::Parameters
+RUN cpanm -q Function::Parameters Hash::Ordered PDL::CCS
WORKDIR /work/mxnet/perl-package/AI-MXNetCAPI
RUN perl Makefile.PL && make install
diff --git a/docs/api/perl/index.md b/docs/api/perl/index.md
index aadff6f..1669d22 100644
--- a/docs/api/perl/index.md
+++ b/docs/api/perl/index.md
@@ -18,7 +18,7 @@ at [Deep Learning - The Straight Dope](http://gluon.mxnet.io/).
For specific Perl Gluon usage please refer to Perl examples and tests directories on github, but be assured that the Python and Perl usage
are extremely close in order to make the use of the Python Gluon docs and examples as easy as possible.
-AI::MXNet is seamlessly glued with PDL, the C++ level state can be easily initialized from PDL and the results can be
+AI::MXNet is seamlessly glued with [PDL](https://metacpan.org/release/PDL), the C++ level state can be easily initialized from PDL and the results can be
transferred to PDL objects in order to allow you to use all the glory and power of the PDL!
Here is how you can perform tensor or matrix computation in Perl with AI::MXNet and PDL:
@@ -60,6 +60,9 @@ pdl> print mx->nd->array(sequence(2,3))->aspdl ## 3 rows, 2 columns
[2 3]
[4 5]
]
+
+Export/import to/from sparse MXNet tensors are supported via [PDL::CCS](https://metacpan.org/release/PDL-CCS).
+Please check out the examples directory for the examples on how to use the sparse matrices.
```
## Perl API Reference
* [Module API is a flexible high-level interface for training neural networks.](module.md)
diff --git a/docs/install/build_from_source.md b/docs/install/build_from_source.md
index 5c558a9..2505f00 100644
--- a/docs/install/build_from_source.md
+++ b/docs/install/build_from_source.md
@@ -473,7 +473,7 @@ Run the following command from the MXNet source root directory to build the MXNe
```bash
sudo apt-get install libmouse-perl pdl cpanminus swig libgraphviz-perl
- cpanm -q -L "${HOME}/perl5" Function::Parameters Hash::Ordered
+ cpanm -q -L "${HOME}/perl5" Function::Parameters Hash::Ordered PDL::CCS
MXNET_HOME=${PWD}
export LD_LIBRARY_PATH=${MXNET_HOME}/lib
diff --git a/docs/install/osx_setup.md b/docs/install/osx_setup.md
index a8567f0..c1fa0fc 100644
--- a/docs/install/osx_setup.md
+++ b/docs/install/osx_setup.md
@@ -188,7 +188,7 @@ After you build the shared library, run the following command from the MXNet sou
```bash
brew install swig
sudo sh -c 'curl -L https://cpanmin.us | perl - App::cpanminus'
- sudo cpanm -q -n PDL Mouse Function::Parameters Hash::Ordered
+ sudo cpanm -q -n PDL Mouse Function::Parameters Hash::Ordered PDL::CCS
MXNET_HOME=${PWD}
export PERL5LIB=${HOME}/perl5/lib/perl5
diff --git a/docs/install/ubuntu_setup.md b/docs/install/ubuntu_setup.md
index d33c042..7796b11 100644
--- a/docs/install/ubuntu_setup.md
+++ b/docs/install/ubuntu_setup.md
@@ -238,7 +238,7 @@ Before you build MXNet for Perl from source code, you must complete [building th
```bash
sudo apt-get install libmouse-perl pdl cpanminus swig libgraphviz-perl
- cpanm -q -L "${HOME}/perl5" Function::Parameters Hash::Ordered
+ cpanm -q -L "${HOME}/perl5" Function::Parameters Hash::Ordered PDL::CCS
MXNET_HOME=${PWD}
export LD_LIBRARY_PATH=${MXNET_HOME}/lib
diff --git a/perl-package/AI-MXNet/Changes b/perl-package/AI-MXNet/Changes
index e9fd9f7..ddf6b3a 100644
--- a/perl-package/AI-MXNet/Changes
+++ b/perl-package/AI-MXNet/Changes
@@ -1,5 +1,8 @@
Revision history for Perl extension AI::MXNet
+1.2 Sun Mar 4 16:29:19 PST 2018
+ - Support for sparse tensors
+
1.1 Sun Oct 1 10:19:08 PDT 2017
- Major update, added support for new imperative MXNet interface Gluon and realtime GPU kernels callable from Perl space.
- Bugfixes for distributed training.
diff --git a/perl-package/AI-MXNet/MANIFEST b/perl-package/AI-MXNet/MANIFEST
index abbd285..690df1e 100644
--- a/perl-package/AI-MXNet/MANIFEST
+++ b/perl-package/AI-MXNet/MANIFEST
@@ -8,6 +8,12 @@ examples/gluon/mnist.pl
examples/lstm_bucketing.pl
examples/mnist.pl
examples/plot_network.pl
+examples/sparse/matrix_factorization/get_data.sh
+examples/sparse/matrix_factorization/README.md
+examples/sparse/matrix_factorization/train.pl
+examples/sparse/wide_deep/get_data.sh
+examples/sparse/wide_deep/README.md
+examples/sparse/wide_deep/train.pl
lib/AI/MXNet.pm
lib/AI/MXNet/AutoGrad.pm
lib/AI/MXNet/Base.pm
@@ -55,6 +61,7 @@ lib/AI/MXNet/NDArray.pm
lib/AI/MXNet/NDArray/Base.pm
lib/AI/MXNet/NDArray/Doc.pm
lib/AI/MXNet/NDArray/Slice.pm
+lib/AI/MXNet/NDArray/Sparse.pm
lib/AI/MXNet/Optimizer.pm
lib/AI/MXNet/Profiler.pm
lib/AI/MXNet/Random.pm
@@ -68,6 +75,7 @@ lib/AI/MXNet/Symbol/Base.pm
lib/AI/MXNet/Symbol/Doc.pm
lib/AI/MXNet/Symbol/NameManager.pm
lib/AI/MXNet/Symbol/Random.pm
+lib/AI/MXNet/Symbol/Sparse.pm
lib/AI/MXNet/TestUtils.pm
lib/AI/MXNet/Types.pm
lib/AI/MXNet/Util/Printable.pm
@@ -80,6 +88,7 @@ README
t/AI-MXNet.t
t/test_attr.t
t/test_autograd.t
+t/test_base.t
t/test_conv.t
t/test_cuda_module.t
t/test_executor.t
@@ -101,5 +110,6 @@ t/test_optimizers.t
t/test_random.t
t/test_recordio.t
t/test_rnn.t
+t/test_sparse_ndarray.t
t/test_symbol.t
t/test_viz.t
diff --git a/perl-package/AI-MXNet/META.json b/perl-package/AI-MXNet/META.json
index 5ab7725..a37d407 100644
--- a/perl-package/AI-MXNet/META.json
+++ b/perl-package/AI-MXNet/META.json
@@ -30,13 +30,14 @@
},
"runtime" : {
"requires" : {
- "AI::MXNetCAPI" : "1.1",
- "AI::NNVMCAPI" : "1.1",
+ "AI::MXNetCAPI" : "1.2",
+ "AI::NNVMCAPI" : "1.2",
"Function::Parameters" : "1.0705",
"Hash::Ordered" : "0.012",
"GraphViz" : "2.14",
"Mouse" : "v2.1.0",
- "PDL" : "2.007"
+ "PDL" : "2.007",
+ "PDL::CCS" : "1.23.4"
}
},
"test" : {
@@ -44,5 +45,5 @@
}
},
"release_status" : "stable",
- "version" : "1.1"
+ "version" : "1.2"
}
diff --git a/perl-package/AI-MXNet/META.yml b/perl-package/AI-MXNet/META.yml
index 1d1d760..8505cef 100644
--- a/perl-package/AI-MXNet/META.yml
+++ b/perl-package/AI-MXNet/META.yml
@@ -17,11 +17,12 @@ no_index:
- t
- inc
requires:
- AI::MXNetCAPI: '1.1'
- AI::NNVMCAPI: '1.1'
+ AI::MXNetCAPI: '1.2'
+ AI::NNVMCAPI: '1.2'
Function::Parameters: '1.0705'
Hash::Ordered: '0.012'
GraphViz: '2.14'
Mouse: v2.1.0
PDL: '2.007'
-version: '1.1'
+ PDL::CCS: '1.23.4'
+version: '1.2'
diff --git a/perl-package/AI-MXNet/Makefile.PL b/perl-package/AI-MXNet/Makefile.PL
index 01fb731..731b693 100644
--- a/perl-package/AI-MXNet/Makefile.PL
+++ b/perl-package/AI-MXNet/Makefile.PL
@@ -36,16 +36,17 @@ my %WriteMakefileArgs = (
"LICENSE" => "apache_2_0",
"NAME" => "AI::MXNet",
"PREREQ_PM" => {
- "AI::MXNetCAPI" => "1.1",
- "AI::NNVMCAPI" => "1.1",
+ "AI::MXNetCAPI" => "1.2",
+ "AI::NNVMCAPI" => "1.2",
"Function::Parameters" => "1.0705",
"Hash::Ordered" => "0.012",
"Mouse" => "v2.1.0",
"PDL" => "2.007",
+ "PDL::CCS" => "1.23.4",
"GraphViz" => "2.14"
},
"TEST_REQUIRES" => {},
- "VERSION" => "1.1",
+ "VERSION" => "1.2",
"test" => {
"TESTS" => "t/*.t"
}
@@ -53,12 +54,13 @@ my %WriteMakefileArgs = (
my %FallbackPrereqs = (
- "AI::MXNetCAPI" => "1.1",
- "AI::NNVMCAPI" => "1.1",
+ "AI::MXNetCAPI" => "1.2",
+ "AI::NNVMCAPI" => "1.2",
"Function::Parameters" => "1.0705",
"Hash::Ordered" => "0.012",
"Mouse" => "v2.1.0",
"PDL" => "2.007",
+ "PDL::CCS" => "1.23.4",
"GraphViz" => "2.14"
);
diff --git a/perl-package/AI-MXNet/README b/perl-package/AI-MXNet/README
index 22eae0e..59d1dd0 100644
--- a/perl-package/AI-MXNet/README
+++ b/perl-package/AI-MXNet/README
@@ -1,5 +1,5 @@
This archive contains the distribution AI-MXNet,
-version 1.1:
+version 1.2:
Perl interface to MXNet machine learning library
diff --git a/perl-package/AI-MXNet/examples/sparse/matrix_factorization/README.md b/perl-package/AI-MXNet/examples/sparse/matrix_factorization/README.md
new file mode 100644
index 0000000..debad27
--- /dev/null
+++ b/perl-package/AI-MXNet/examples/sparse/matrix_factorization/README.md
@@ -0,0 +1,9 @@
+Matrix Factorization w/ Sparse Embedding
+===========
+The example demonstrates the basic usage of the SparseEmbedding operator in MXNet, adapted based on @leopd's recommender examples.
+The operator is available on both CPU and GPU. This is for demonstration purpose only.
+
+- get_data.sh
+- perl train.pl
+- To compare the training speed with (dense) Embedding, run perl train.pl --use-dense 1
+- To run the example on the GPU, run perl train.pl --use-gpu 1
diff --git a/tests/ci_build/install/ubuntu_install_perl.sh b/perl-package/AI-MXNet/examples/sparse/matrix_factorization/get_data.sh
similarity index 71%
copy from tests/ci_build/install/ubuntu_install_perl.sh
copy to perl-package/AI-MXNet/examples/sparse/matrix_factorization/get_data.sh
index 217f322..b8b14e1 100755
--- a/tests/ci_build/install/ubuntu_install_perl.sh
+++ b/perl-package/AI-MXNet/examples/sparse/matrix_factorization/get_data.sh
@@ -16,9 +16,18 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-set -e
-set -x
-# install libraries for mxnet's perl package on ubuntu
-apt-get update && apt-get install -y libmouse-perl pdl cpanminus swig libgraphviz-perl
-cpanm -q Function::Parameters Hash::Ordered
+DIR=$(cd `dirname $0`; pwd)
+DATA_DIR="${DIR}/data/"
+
+if [[ ! -d "${DATA_DIR}" ]]; then
+ echo "${DATA_DIR} doesn't exist, will create one";
+ mkdir -p ${DATA_DIR}
+fi
+
+wget -P ${DATA_DIR} http://files.grouplens.org/datasets/movielens/ml-10m.zip
+cd ${DATA_DIR}
+unzip ml-10m.zip
+cd ml-10M100K
+chmod +x allbut.pl
+sh split_ratings.sh
\ No newline at end of file
diff --git a/perl-package/AI-MXNet/examples/sparse/matrix_factorization/train.pl b/perl-package/AI-MXNet/examples/sparse/matrix_factorization/train.pl
new file mode 100755
index 0000000..fa6a763
--- /dev/null
+++ b/perl-package/AI-MXNet/examples/sparse/matrix_factorization/train.pl
@@ -0,0 +1,184 @@
+#!/usr/bin/env perl
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+use strict;
+use warnings;
+use AI::MXNet qw(mx);
+use Getopt::Long qw(HelpMessage);
+
+GetOptions(
+ 'print-every=i' => \(my $print_every = 100),
+ 'factor-size=i' => \(my $factor_size = 128),
+ 'use-gpu=i' => \(my $use_gpu = 0 ),
+ 'num-epoch=i' => \(my $num_epoch = 3 ),
+ 'batch-size=i' => \(my $batch_size = 128),
+ 'use-dense=i' => \(my $use_dense = 0 ),
+ 'help' => sub { HelpMessage(0) },
+) or HelpMessage(1);
+
+=head1 NAME
+
+ train.pl - Run matrix factorization with sparse embedding
+
+=head1 SYNOPSIS
+
+ --print-every logging frequency, 100
+ --factor-size the factor size of the embedding operation, 128
+ --use-gpu use gpu, 0
+ --num-epoch number of epochs to train, 3
+ --batch-size number of examples per batch, 128
+ --use-dense use the dense embedding operator, 0
+
+=cut
+
+my %MOVIELENS = (
+ dataset => 'ml-10m',
+ train => './data/ml-10M100K/r1.train',
+ val => './data/ml-10M100K/r1.test',
+ max_user => 71569,
+ max_movie => 65135,
+);
+
+sub get_movielens_iter
+{
+ my ($filename, $batch_size) = @_;
+ print "Preparing data iterators for $filename ... \n";
+ my @user;
+ my @item;
+ my @score;
+ open(F, $filename) or die $!;
+ my $num_samples = 0;
+ while(my $line = <F>)
+ {
+ my @tks = split('::', $line);
+ next unless @tks == 4;
+ $num_samples++;
+ push @user, [$tks[0]];
+ push @item, [$tks[1]];
+ push @score, [$tks[2]];
+ }
+ # convert to ndarrays
+ my $user = mx->nd->array(\@user, dtype=>'int32');
+ my $item = mx->nd->array(\@item);
+ my $score = mx->nd->array(\@score);
+ return mx->io->NDArrayIter(
+ data => Hash::Ordered->new(user => $user, item => $item),
+ label => Hash::Ordered->new(score => $score),
+ batch_size => $batch_size,
+ shuffle => 1
+ );
+}
+
+sub matrix_fact_net
+{
+ my ($factor_size, $num_hidden, $max_user, $max_item, $sparse_embed) = @_;
+ $sparse_embed //= 1;
+ # input
+ my $user = mx->symbol->Variable('user');
+ my $item = mx->symbol->Variable('item');
+ my $score = mx->symbol->Variable('score');
+ if($sparse_embed)
+ {
+ # user feature lookup
+ my $user_weight = mx->symbol->Variable('user_weight', stype=>'row_sparse');
+ $user = mx->symbol->contrib->SparseEmbedding(data=>$user, weight=>$user_weight,
+ input_dim=>$max_user, output_dim=>$factor_size);
+ # item feature lookup
+ my $item_weight = mx->symbol->Variable('item_weight', stype=>'row_sparse');
+ $item = mx->symbol->contrib->SparseEmbedding(data=>$item, weight=>$item_weight,
+ input_dim=>$max_item, output_dim=>$factor_size);
+ }
+ else
+ {
+ # user feature lookup
+ $user = mx->symbol->Embedding(data=>$user, input_dim=>$max_user, output_dim=>$factor_size);
+ # item feature lookup
+ $item = mx->symbol->Embedding(data=>$item, input_dim=>$max_item, output_dim=>$factor_size);
+ }
+ # non-linear transformation of user features
+ $user = mx->symbol->Activation(data=>$user, act_type=>'relu');
+ $user = mx->symbol->FullyConnected(data=>$user, num_hidden=>$num_hidden);
+ # non-linear transformation of item features
+ $item = mx->symbol->Activation(data=>$item, act_type=>'relu');
+ $item = mx->symbol->FullyConnected(data=>$item, num_hidden=>$num_hidden);
+ # predict by the inner product, which is elementwise product and then sum
+ my $pred = $user * $item;
+ $pred = mx->symbol->sum(data=>$pred, axis => 1);
+ $pred = mx->symbol->Flatten(data=>$pred);
+ # loss layer
+ $pred = mx->symbol->LinearRegressionOutput(data=>$pred, label=>$score);
+ return $pred;
+}
+
+my $optimizer = 'sgd';
+my $use_sparse = not $use_dense;
+
+my $momentum = 0.9;
+my $ctx = $use_gpu ? mx->gpu(0) : mx->cpu(0);
+my $learning_rate = 0.1;
+
+# prepare dataset and iterators
+my $max_user = $MOVIELENS{max_user};
+my $max_movies = $MOVIELENS{max_movie};
+my $train_iter = get_movielens_iter($MOVIELENS{train}, $batch_size);
+my $val_iter = get_movielens_iter($MOVIELENS{val} , $batch_size);
+
+# construct the model
+my $net = matrix_fact_net($factor_size, $factor_size, $max_user, $max_movies, $use_sparse);
+
+# initialize the module
+my $mod = mx->module->Module(symbol=>$net, context=>$ctx, data_names=>['user', 'item'],
+ label_names=>['score']);
+$mod->bind(data_shapes=>$train_iter->provide_data, label_shapes=>$train_iter->provide_label);
+$mod->init_params(initializer=>mx->init->Xavier(factor_type=>"in", magnitude=>2.34));
+my $optim = mx->optimizer->create($optimizer, learning_rate=>$learning_rate, momentum=>$momentum,
+ wd=>1e-4, rescale_grad=>1.0/$batch_size);
+$mod->init_optimizer(optimizer=>$optim);
+
+# use MSE as the metric
+my $metric = mx->metric->create(['MSE']);
+my $speedometer = mx->callback->Speedometer($batch_size, $print_every);
+print "Training started ...\n";
+for my $epoch (0..$num_epoch-1)
+{
+ my $nbatch = 0;
+ $metric->reset();
+ while(my $batch = <$train_iter>)
+ {
+ $nbatch += 1;
+ $mod->forward_backward($batch);
+ # update all parameters
+ $mod->update();
+ # update training metric
+ $mod->update_metric($metric, $batch->label);
+ my $speedometer_param = AI::MXNet::BatchEndParam->new(
+ epoch=>$epoch, nbatch=>$nbatch,
+ eval_metric=>$metric
+ );
+ $speedometer->($speedometer_param);
+ }
+ # evaluate metric on validation dataset
+ my $score = $mod->score($val_iter, ['MSE']);
+ printf("epoch %d, eval MSE = %s \n", $epoch, $score->{mse});
+ # reset the iterator for next pass of data
+ $train_iter->reset();
+ $val_iter->reset();
+}
+print "Training completed.\n";
+
diff --git a/perl-package/AI-MXNet/examples/sparse/wide_deep/README.md b/perl-package/AI-MXNet/examples/sparse/wide_deep/README.md
new file mode 100644
index 0000000..9a481d6
--- /dev/null
+++ b/perl-package/AI-MXNet/examples/sparse/wide_deep/README.md
@@ -0,0 +1,7 @@
+## Wide and Deep Learning
+
+The example demonstrates how to train [wide and deep model](https://arxiv.org/abs/1606.07792). The [Census Income Data Set](https://archive.ics.uci.edu/ml/datasets/Census+Income) that this example uses for training is hosted by the [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/). Tricks of feature engineering are adapted from tensorflow's [wide and deep tutorial](https://github.com/tensorflow/models/tree/master/official/wide_deep).
+
+The final accuracy should be around 84%.
+
+- perl train.pl
diff --git a/tests/ci_build/install/ubuntu_install_perl.sh b/perl-package/AI-MXNet/examples/sparse/wide_deep/get_data.sh
similarity index 65%
copy from tests/ci_build/install/ubuntu_install_perl.sh
copy to perl-package/AI-MXNet/examples/sparse/wide_deep/get_data.sh
index 217f322..69e5b4d 100755
--- a/tests/ci_build/install/ubuntu_install_perl.sh
+++ b/perl-package/AI-MXNet/examples/sparse/wide_deep/get_data.sh
@@ -16,9 +16,17 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-set -e
-set -x
-# install libraries for mxnet's perl package on ubuntu
-apt-get update && apt-get install -y libmouse-perl pdl cpanminus swig libgraphviz-perl
-cpanm -q Function::Parameters Hash::Ordered
+DIR=$(cd `dirname $0`; pwd)
+DATA_DIR="${DIR}/data/"
+
+if [[ ! -d "${DATA_DIR}" ]]; then
+ echo "${DATA_DIR} doesn't exist, will create one";
+ mkdir -p ${DATA_DIR}
+fi
+
+wget -P ${DATA_DIR} https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data
+wget -P ${DATA_DIR} https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test
+cd ${DATA_DIR}
+sed -i '1d' adult.test
+sed -i 's/.$//' adult.test
\ No newline at end of file
diff --git a/perl-package/AI-MXNet/examples/sparse/wide_deep/train.pl b/perl-package/AI-MXNet/examples/sparse/wide_deep/train.pl
new file mode 100755
index 0000000..cd18c61
--- /dev/null
+++ b/perl-package/AI-MXNet/examples/sparse/wide_deep/train.pl
@@ -0,0 +1,301 @@
+#!/usr/bin/env perl
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+use strict;
+use warnings;
+use AI::MXNet qw(mx);
+use AI::MXNet::Base qw(pdl enumerate digitize hash array_index range);
+use Getopt::Long qw(HelpMessage);
+
+GetOptions(
+ 'log-interval=i' => \(my $log_interval = 100),
+ 'optimizer=s' => \(my $optimizer = 'adam'),
+ 'cuda' => \(my $cuda = 0 ),
+ 'num-epoch=i' => \(my $num_epoch = 10 ),
+ 'batch-size=i' => \(my $batch_size = 100),
+ 'lr=f' => \(my $lr = 0.001 ),
+ 'help' => sub { HelpMessage(0) },
+) or HelpMessage(1);
+
+=head1 NAME
+
+ train.pl - Run sparse wide and deep classification
+
+=head1 SYNOPSIS
+
+ --log-interval number of batches to wait before logging training status, 100
+ --optimizer what optimizer to use, 'adam'
+ --cuda train on gpu with cuda, 0
+ --num-epoch number of epochs to train, 10
+ --batch-size number of examples per batch, 100
+ --lr learning rate, 0.001
+
+=cut
+
+my %allowed_optimizers = qw(adam 1 sgd 1 ftrl 1);
+Carp::confess("optimizer can only be one of 'adam', 'sgd', 'ftrl'")
+ unless exists $allowed_optimizers{ $optimizer };
+
+sub wide_deep_model
+{
+ my ($num_linear_features, $num_embed_features, $num_cont_features,
+ $input_dims, $hidden_units) = @_;
+ # wide model
+ my $csr_data = mx->symbol->Variable("csr_data", stype=>'csr');
+ my $label = mx->symbol->Variable("softmax_label");
+
+ my $norm_init = mx->initializer->Normal(sigma=>0.01);
+ # weight with row_sparse storage type to enable sparse gradient updates
+ my $weight = mx->symbol->Variable("linear_weight", shape=>[$num_linear_features, 2],
+ init=>$norm_init, stype=>'row_sparse');
+ my $bias = mx->symbol->Variable("linear_bias", shape=>[2]);
+ my $dot = mx->symbol->sparse->dot($csr_data, $weight);
+ my $linear_out = mx->symbol->broadcast_add($dot, $bias);
+ # deep model
+ my $dns_data = mx->symbol->Variable("dns_data");
+ # embedding features
+ my $x = mx->symbol->slice(data=>$dns_data, begin=>[0, 0],
+ end=>[undef, $num_embed_features]);
+ my $embeds = mx->symbol->split(data=>$x, num_outputs=>$num_embed_features, squeeze_axis=>1);
+ # continuous features
+ $x = mx->symbol->slice(data=>$dns_data, begin=>[0, $num_embed_features],
+ end=>[undef, $num_embed_features + $num_cont_features]);
+ my @features = ($x);
+
+ enumerate(sub {
+ my ($i, $embed) = @_;
+ my $embed_weight = mx->symbol->Variable("embed_${i}_weight", stype=>'row_sparse');
+ push @features, mx->symbol->contrib->SparseEmbedding(data=>$embed, weight=>$embed_weight,
+ input_dim=>$input_dims->[$i], output_dim=>$hidden_units->[0]);
+
+ }, $embeds);
+
+ my $hidden = mx->symbol->concat(@features, dim=>1);
+ $hidden = mx->symbol->FullyConnected(data=>$hidden, num_hidden=>$hidden_units->[1]);
+ $hidden = mx->symbol->Activation(data=>$hidden, act_type=>'relu');
+ $hidden = mx->symbol->FullyConnected(data=>$hidden, num_hidden=>$hidden_units->[2]);
+ $hidden = mx->symbol->Activation(data=>$hidden, act_type=>'relu');
+ my $deep_out = mx->symbol->FullyConnected(data=>$hidden, num_hidden=>2);
+
+ my $out = mx->symbol->SoftmaxOutput($linear_out + $deep_out, $label, name=>'model');
+ return $out;
+}
+
+sub preprocess_uci_adult
+{
+ my ($data_name) = @_;
+ # Some tricks of feature engineering are adapted
+ # from tensorflow's wide and deep tutorial.
+ my @csv_columns = (
+ "age", "workclass", "fnlwgt", "education", "education_num",
+ "marital_status", "occupation", "relationship", "race", "gender",
+ "capital_gain", "capital_loss", "hours_per_week", "native_country",
+ "income_bracket"
+ );
+
+ my %vocabulary_dict = (
+ "gender" => [
+ "Female", "Male"
+ ],
+ "education" => [
+ "Bachelors", "HS-grad", "11th", "Masters", "9th",
+ "Some-college", "Assoc-acdm", "Assoc-voc", "7th-8th",
+ "Doctorate", "Prof-school", "5th-6th", "10th", "1st-4th",
+ "Preschool", "12th"
+ ],
+ "marital_status" => [
+ "Married-civ-spouse", "Divorced", "Married-spouse-absent",
+ "Never-married", "Separated", "Married-AF-spouse", "Widowed"
+ ],
+ "relationship" => [
+ "Husband", "Not-in-family", "Wife", "Own-child", "Unmarried",
+ "Other-relative"
+ ],
+ "workclass" => [
+ "Self-emp-not-inc", "Private", "State-gov", "Federal-gov",
+ "Local-gov", "?", "Self-emp-inc", "Without-pay", "Never-worked"
+ ]
+ );
+ # wide columns
+ my @crossed_columns = (
+ ["education", "occupation"],
+ ["native_country", "occupation"],
+ ["age_buckets", "education", "occupation"],
+ );
+ my @age_boundaries = (18, 25, 30, 35, 40, 45, 50, 55, 60, 65);
+ # deep columns
+ my @indicator_columns = ('workclass', 'education', 'gender', 'relationship');
+
+ my @embedding_columns = ('native_country', 'occupation');
+
+ my @continuous_columns = ('age', 'education_num', 'capital_gain', 'capital_loss', 'hours_per_week');
+ # income_bracket column is the label
+ my %labels = ("<=50K" => 0, ">50K" => 1);
+
+ my $hash_bucket_size = 1000;
+
+ my $csr_ncols = @crossed_columns * $hash_bucket_size;
+ my $dns_ncols = @continuous_columns + @embedding_columns;
+ for my $col (@indicator_columns)
+ {
+ $dns_ncols += @{ $vocabulary_dict{$col} };
+ }
+
+ my @label_list;
+ my @csr_list;
+ my @dns_list;
+
+ open(F, $data_name) or die $!;
+ while(<F>)
+ {
+ chomp;
+ my %row;
+ @row{ @csv_columns } = split(/\s*,\s*/);
+ next if not defined $row{income_bracket};
+ push @label_list, $labels{$row{income_bracket}};
+ enumerate(sub {
+ my ($i, $cols) = @_;
+ if($cols->[0] eq "age_buckets")
+ {
+ my $age_bucket = digitize($row{age}, \@age_boundaries);
+ my $s = join('_', map { $row{$_} } @{ $cols }[1..@{$cols}-1]);
+ $s .= '_' . $age_bucket;
+ push @csr_list, [$i * $hash_bucket_size + hash($s) % $hash_bucket_size, 1];
+ }
+ else
+ {
+ my $s = join('_', map { $row{$_} } @{ $cols });
+ push @csr_list, [$i * $hash_bucket_size + hash($s) % $hash_bucket_size, 1];
+ }
+ }, \@crossed_columns);
+ my @dns_row = (0) x $dns_ncols;
+ my $dns_dim = 0;
+ for my $col (@embedding_columns)
+ {
+ $dns_row[$dns_dim] = hash($row{$col}) % $hash_bucket_size;
+ $dns_dim += 1;
+ }
+
+ for my $col (@indicator_columns)
+ {
+ $dns_row[$dns_dim + array_index($row{$col}, $vocabulary_dict{$col})] = 1;
+ $dns_dim += @{$vocabulary_dict{$col}};
+ }
+
+ for my $col (@continuous_columns)
+ {
+ $dns_row[$dns_dim] = $row{col};
+ $dns_dim += 1;
+ }
+ push @dns_list, \@dns_row;
+ }
+ my @data_list = map { $_->[1] } @csr_list;
+ my @indices_list = map { $_->[0] } @csr_list;
+ my @indptr_list = range(0, @indices_list + 1, scalar @crossed_columns);
+ # convert to ndarrays
+ my $csr = mx->nd->sparse->csr_matrix([\@data_list, \@indices_list, \@indptr_list],
+ shape=>[scalar @label_list, $hash_bucket_size * @crossed_columns]);
+ my $dns = pdl(\@dns_list);
+ my $label = pdl(\@label_list);
+ return ($csr, $dns, $label);
+}
+
+# Related to feature engineering, please see preprocess in data.py
+my %ADULT = (
+ train => './data/adult.data',
+ test => './data/adult.test',
+ num_linear_features => 3000,
+ num_embed_features => 2,
+ num_cont_features => 38,
+ embed_input_dims => [1000, 1000],
+ hidden_units => [8, 50, 100],
+);
+
+
+my $ctx = $cuda ? mx->gpu : mx->cpu;
+
+# dataset
+my ($train_csr, $train_dns, $train_label) = preprocess_uci_adult($ADULT{train});
+my ($val_csr, $val_dns, $val_label) = preprocess_uci_adult($ADULT{test});
+my $model = wide_deep_model(
+ $ADULT{num_linear_features}, $ADULT{num_embed_features},
+ $ADULT{num_cont_features}, $ADULT{embed_input_dims},
+ $ADULT{hidden_units}
+);
+
+# data iterator
+my $train_data = mx->io->NDArrayIter(
+ data => Hash::Ordered->new(csr_data => $train_csr, dns_data => $train_dns),
+ label => Hash::Ordered->new(softmax_label => $train_label),
+ batch_size => $batch_size,
+ shuffle => 1,
+ last_batch_handle => 'discard'
+);
+my $eval_data = mx->io->NDArrayIter(
+ data => Hash::Ordered->new(csr_data => $val_csr, val_data => $val_dns),
+ label => Hash::Ordered->new(softmax_label => $val_label),
+ batch_size => $batch_size,
+ shuffle => 0,
+ last_batch_handle => 'discard'
+);
+
+# module
+my $mod = mx->mod->Module(
+ symbol => $model, context => $ctx, data_names=>['csr_data', 'dns_data'],
+ label_names => ['softmax_label']
+);
+$mod->bind(data_shapes => $train_data->provide_data, label_shapes => $train_data->provide_label);
+$mod->init_params();
+my $optim = mx->optimizer->create($optimizer, learning_rate=>$lr, rescale_grad=>1/$batch_size);
+$mod->init_optimizer(optimizer=>$optim);
+# use accuracy as the metric
+my $metric = mx->metric->create('acc');
+# get the sparse weight parameter
+my $speedometer = mx->callback->Speedometer($batch_size, $log_interval);
+
+print "Training started ...\n";
+
+for my $epoch (0..$num_epoch-1)
+{
+ my $nbatch = 0;
+ $metric->reset;
+ while(my $batch = <$train_data>)
+ {
+ $nbatch++;
+ $mod->forward_backward($batch);
+ # update all parameters (including the weight parameter)
+ $mod->update;
+ # update training metric
+ $mod->update_metric($metric, $batch->label);
+ my $speedometer_param = AI::MXNet::BatchEndParam->new(
+ epoch=>$epoch, nbatch=>$nbatch,
+ eval_metric=>$metric
+ );
+ $speedometer->($speedometer_param);
+ }
+ # evaluate metric on validation dataset
+ my $score = $mod->score($eval_data, 'acc');
+ printf("epoch %d, validation accuracy = %.4f\n", $epoch, $score->{accuracy});
+
+ $mod->save_checkpoint("checkpoint", $epoch, 1);
+ # reset the iterator for next pass of data
+ $train_data->reset;
+}
+
+print "Training completed.\n";
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet.pm b/perl-package/AI-MXNet/lib/AI/MXNet.pm
index 5a98715..87d0b6d 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet.pm
@@ -47,7 +47,9 @@ use AI::MXNet::Contrib;
use AI::MXNet::CachedOp;
use AI::MXNet::AutoGrad;
use AI::MXNet::Gluon;
-our $VERSION = '1.1';
+use AI::MXNet::NDArray::Sparse;
+use AI::MXNet::Symbol::Sparse;
+our $VERSION = '1.2';
sub import
{
@@ -78,6 +80,7 @@ sub import
sub io { 'AI::MXNet::IO' }
sub metric { 'AI::MXNet::Metric' }
sub mod { 'AI::MXNet::Module' }
+ sub module { 'AI::MXNet::Module' }
sub mon { 'AI::MXNet::Monitor' }
sub viz { 'AI::MXNet::Visualization' }
sub rnn { 'AI::MXNet::RNN' }
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/AutoGrad.pm b/perl-package/AI-MXNet/lib/AI/MXNet/AutoGrad.pm
index 221840e..160cd96 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/AutoGrad.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/AutoGrad.pm
@@ -335,7 +335,7 @@ method grad(
my @ret;
for(zip($grad_vars, $grad_stypes)) {
my ($handle, $stype) = @$_;
- push @ret, AI::MXNet::NDArray->new(handle => $handle, stype => $stype);
+ push @ret, AI::MXNet::NDArray->_ndarray_cls($handle, 1, $stype);
}
if(blessed $variables)
{
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Base.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Base.pm
index f748ecb..33b3d4d 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Base.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Base.pm
@@ -19,21 +19,26 @@ package AI::MXNet::Base;
use strict;
use warnings;
use PDL;
-use PDL::Types qw();
-use AI::MXNetCAPI 1.1;
-use AI::NNVMCAPI 1.1;
+use PDL::Types ();
+use PDL::CCS::Nd;
+use AI::MXNetCAPI 1.2;
+use AI::NNVMCAPI 1.2;
use AI::MXNet::Types;
use Time::HiRes;
+use Scalar::Util qw(blessed);
use Carp;
use Exporter;
use base qw(Exporter);
use List::Util qw(shuffle);
@AI::MXNet::Base::EXPORT = qw(product enumerate assert zip check_call build_param_doc
- pdl cat dog svd bisect_left pdl_shuffle as_array
+ pdl cat dog svd bisect_left pdl_shuffle as_array ascsr rand_sparse
DTYPE_STR_TO_MX DTYPE_MX_TO_STR DTYPE_MX_TO_PDL
- DTYPE_PDL_TO_MX DTYPE_MX_TO_PERL GRAD_REQ_MAP);
-@AI::MXNet::Base::EXPORT_OK = qw(pzeros pceil);
+ DTYPE_PDL_TO_MX DTYPE_MX_TO_PERL GRAD_REQ_MAP
+ STORAGE_TYPE_UNDEFINED STORAGE_TYPE_DEFAULT
+ STORAGE_TYPE_ROW_SPARSE STORAGE_TYPE_CSR
+ STORAGE_TYPE_STR_TO_ID STORAGE_TYPE_ID_TO_STR STORAGE_AUX_TYPES);
+@AI::MXNet::Base::EXPORT_OK = qw(pzeros pceil pones digitize hash array_index range);
use constant DTYPE_STR_TO_MX => {
float32 => 0,
@@ -97,6 +102,29 @@ use constant GRAD_REQ_MAP => {
write => 1,
add => 3
};
+use constant {
+ STORAGE_TYPE_UNDEFINED => -1,
+ STORAGE_TYPE_DEFAULT => 0,
+ STORAGE_TYPE_ROW_SPARSE => 1,
+ STORAGE_TYPE_CSR => 2
+};
+use constant STORAGE_TYPE_STR_TO_ID => {
+ undefined => STORAGE_TYPE_UNDEFINED,
+ default => STORAGE_TYPE_DEFAULT,
+ row_sparse => STORAGE_TYPE_ROW_SPARSE,
+ csr => STORAGE_TYPE_CSR
+};
+use constant STORAGE_TYPE_ID_TO_STR => {
+ STORAGE_TYPE_UNDEFINED() => 'undefined',
+ STORAGE_TYPE_DEFAULT() => 'default',
+ STORAGE_TYPE_ROW_SPARSE() => 'row_sparse',
+ STORAGE_TYPE_CSR() => 'csr'
+};
+use constant STORAGE_AUX_TYPES => {
+ row_sparse => ['int64'],
+ csr => ['int64', 'int64']
+};
+
=head1 NAME
@@ -198,18 +226,11 @@ sub bisect_left
as shuffled last dimension's indexes
=cut
-
sub pdl_shuffle
{
my ($pdl, $preshuffle) = @_;
- my $c = $pdl->copy;
my @shuffle = $preshuffle ? @{ $preshuffle } : shuffle(0..$pdl->dim(-1)-1);
- my $rem = $pdl->ndims-1;
- for my $i (0..$pdl->dim(-1)-1)
- {
- $c->slice(('X')x$rem, $i) .= $pdl->slice(('X')x$rem, $shuffle[$i])
- }
- $c;
+ return $pdl->dice_axis(-1, pdl(\@shuffle));
}
=head2 assert
@@ -348,10 +369,76 @@ END {
}
*pzeros = \&zeros;
+*pones = \&ones;
*pceil = \&ceil;
## making sure that we can stringify arbitrarily large piddles
$PDL::toolongtoprint = 1000_000_000;
## convenience subs
+
+sub ascsr
+{
+ my ($data, $indptr, $indices, $shape) = @_;
+ my @which;
+ my $i = 0;
+ my $j = 0;
+ while($i < $indices->nelem)
+ {
+ for($i = $indptr->at($j); $i < $indptr->at($j+1); $i++)
+ {
+ push @which, [$j, $indices->at($i)];
+ }
+ $j++;
+ }
+ return PDL::CCS::Nd->newFromWhich(
+ pdl(\@which), $data, pdims => blessed $shape ? $shape : pdl($shape)
+ )->xchg(0, 1);
+}
+
+package AI::MXNet::COO::Nd;
+use Mouse;
+has ['data', 'row', 'col'] => (is => 'rw');
+no Mouse;
+
+package AI::MXNet::Base;
+
+sub tocoo
+{
+ my $csr = shift;
+ return AI::MXNet::COO::Nd->new(
+ data => $csr->data,
+ row => $csr->_whichND->slice(0)->flat,
+ col => $csr->_whichND->slice(1)->flat
+ );
+}
+
+sub rand_sparse
+{
+ my ($num_rows, $num_cols, $density, $dtype, $format) = @_;
+ $dtype //= 'float32';
+ $format //= 'csr';
+ my $pdl_type = PDL::Type->new(DTYPE_MX_TO_PDL->{ $dtype });
+ my $dense = random($pdl_type, $num_cols, $num_rows);
+ my $missing = 0;
+ $dense->where(random($num_cols, $num_rows)<=1-$density) .= $missing;
+ if($format eq 'csr')
+ {
+ return $dense->tocsr;
+ }
+ return $dense;
+}
+
+{
+ no warnings 'once';
+ *PDL::CCS::Nd::data = sub { shift->_nzvals };
+ *PDL::CCS::Nd::indptr = sub { my $self = shift; ($self->hasptr ? $self->getptr : $self->ptr)[0] };
+ *PDL::CCS::Nd::indices = sub { shift->_whichND->slice(1)->flat };
+ *PDL::CCS::Nd::tocoo = sub { tocoo(shift) };
+ *PDL::CCS::Nd::shape = sub { shift->pdims };
+ *PDL::CCS::Nd::dtype = sub { DTYPE_MX_TO_STR->{ DTYPE_PDL_TO_MX->{ shift->type->numval } } };
+ *PDL::tocsr = sub { shift->xchg(0, 1)->toccs->xchg(0, 1) };
+ *PDL::rand_sparse = sub { shift; rand_sparse(@_) };
+}
+
{
my $orig_at = PDL->can('at');
no warnings 'redefine';
@@ -365,4 +452,20 @@ $PDL::toolongtoprint = 1000_000_000;
*PDL::dtype = sub { DTYPE_MX_TO_STR->{ DTYPE_PDL_TO_MX->{ shift->type->numval } } };
}
+sub digitize
+{
+ my ($d, $bins) = @_;
+ for(my $i = 0; $i < @$bins; $i++)
+ {
+ return $i if $d < $bins->[$i];
+ }
+ return scalar(@$bins);
+}
+
+use B;
+sub hash { hex(B::hash(shift)) }
+use List::Util ();
+sub array_index { my ($s, $array) = @_; return List::Util::first { $array->[$_] eq $s } 0..@$array-1 }
+sub range { my ($begin, $end, $step) = @_; $step //= 1; grep { not (($_-$begin) % $step) } $begin..$end-1 }
+
1;
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/CachedOp.pm b/perl-package/AI-MXNet/lib/AI/MXNet/CachedOp.pm
index f3c21ed..9222afb 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/CachedOp.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/CachedOp.pm
@@ -95,11 +95,11 @@ sub call
return $original_output if defined $original_output;
if(@$output == 1)
{
- return AI::MXNet::NDArray->new(handle => $output->[0]);
+ return AI::MXNet::NDArray->_ndarray_cls($output->[0]);
}
else
{
- return [map { AI::MXNet::NDArray->new(handle => $_) } @$output];
+ return [map { AI::MXNet::NDArray->_ndarray_cls($_) } @$output];
}
}
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Executor.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Executor.pm
index 3a1a343..ebda90f 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Executor.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Executor.pm
@@ -126,7 +126,7 @@ method _get_outputs()
{
return [
map {
- AI::MXNet::NDArray->new(handle => $_)
+ AI::MXNet::NDArray->_ndarray_cls($_)
}
@{ check_call(AI::MXNetCAPI::ExecutorOutputs($self->handle)) }
];
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/IO.pm b/perl-package/AI-MXNet/lib/AI/MXNet/IO.pm
index 6ca98ca..fc3f960 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/IO.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/IO.pm
@@ -459,20 +459,35 @@ sub BUILD
my $self = shift;
my $data = AI::MXNet::IO->init_data($self->data, allow_empty => 0, default_name => 'data');
my $label = AI::MXNet::IO->init_data($self->label, allow_empty => 1, default_name => $self->label_name);
+ if(
+ (
+ (blessed $data and $data->isa('AI::MXNet::NDArray::CSR'))
+ or
+ (blessed $label and $label->isa('AI::MXNet::NDArray::CSR'))
+ )
+ and
+ ($self->last_batch_handle != 'discard')
+ )
+ {
+ confess("`NDArrayIter` only supports AI::MXNet::NDArray::CSR with `last_batch_handle` set to discard.");
+ }
my $num_data = $data->[0][1]->shape->[0];
confess("size of data dimension 0 $num_data < batch_size ${\ $self->batch_size }")
unless($num_data >= $self->batch_size);
if($self->shuffle)
{
my @idx = List::Util::shuffle(0..$num_data-1);
- $_->[1] = AI::MXNet::NDArray->array(pdl_shuffle($_->[1]->aspdl, \@idx)) for @$data;
- $_->[1] = AI::MXNet::NDArray->array(pdl_shuffle($_->[1]->aspdl, \@idx)) for @$label;
+ $_->[1] = AI::MXNet::NDArray->array(
+ pdl_shuffle($_->[1]->stype eq 'csr' ? $_->[1]->aspdlccs : $_->[1]->aspdl, \@idx),
+ ctx => $_->[1]->context
+ ) for (@$data, @$label);
}
if($self->last_batch_handle eq 'discard')
{
my $new_n = $num_data - $num_data % $self->batch_size - 1;
$_->[1] = $_->[1]->slice([0, $new_n]) for @$data;
$_->[1] = $_->[1]->slice([0, $new_n]) for @$label;
+ $num_data = $new_n + 1;
}
my $data_list = [map { $_->[1] } (@{ $data }, @{ $label })];
my $num_source = @{ $data_list };
@@ -721,13 +736,13 @@ method iter_next()
method getdata()
{
my $handle = check_call(AI::MXNetCAPI::DataIterGetData($self->handle));
- return AI::MXNet::NDArray->new(handle => $handle);
+ return AI::MXNet::NDArray->_ndarray_cls($handle);
}
method getlabel()
{
my $handle = check_call(AI::MXNetCAPI::DataIterGetLabel($self->handle));
- return AI::MXNet::NDArray->new(handle => $handle);
+ return AI::MXNet::NDArray->_ndarray_cls($handle);
}
method getindex()
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/KVStore.pm b/perl-package/AI-MXNet/lib/AI/MXNet/KVStore.pm
index 84a890d..de66d91 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/KVStore.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/KVStore.pm
@@ -227,6 +227,170 @@ method pull(
);
}
+=head2 row_sparse_pull
+
+ Pulls a single AI::MXNet::NDArray::RowSparse value or an array ref of AI::MXNet::NDArray::RowSparse values
+ from the store with specified row_ids. When there is only one row_id, KVStoreRowSparsePull
+ is invoked just once and the result is broadcast to all the rest of outputs.
+
+ `row_sparse_pull` is executed asynchronously after all previous
+ `pull`/`row_sparse_pull` calls and the last `push` call for the
+ same input key(s) are finished.
+
+ The returned values are guaranteed to be the latest values in the store.
+
+ Parameters
+ ----------
+ key : str, int, or sequence of str or int
+ Keys.
+
+ out: AI::MXNet::NDArray::RowSparse or array ref of AI::MXNet::NDArray::RowSparse or array ref of array ref of AI::MXNet::NDArray::RowSparse
+ Values corresponding to the keys. The stype is expected to be row_sparse
+
+ priority : int, optional
+ The priority of the pull operation.
+ Higher priority pull operations are likely to be executed before
+ other pull actions.
+
+ row_ids : AI::MXNet::NDArray or array ref of AI::MXNet::NDArray
+ The row_ids for which to pull for each value. Each row_id is an 1D NDArray
+ whose values don't have to be unique nor sorted.
+
+ Examples
+ --------
+ >>> $shape = [3, 3]
+ >>> $kv->init('3', mx->nd->ones($shape)->tostype('row_sparse'))
+ >>> $a = mx->nd->sparse->zeros('row_sparse', $shape)
+ >>> $row_ids = mx->nd->array([0, 2], dtype=>'int64')
+ >>> $kv->row_sparse_pull('3', out=>$a, row_ids=>$row_ids)
+ >>> print $a->aspdl
+ [[ 1. 1. 1.]
+ [ 0. 0. 0.]
+ [ 1. 1. 1.]]
+ >>> $duplicate_row_ids = mx->nd->array([2, 2], dtype=>'int64')
+ >>> $kv->row_sparse_pull('3', out=>$a, row_ids=>$duplicate_row_ids)
+ >>> print $a->aspdl
+ [[ 0. 0. 0.]
+ [ 0. 0. 0.]
+ [ 1. 1. 1.]]
+ >>> $unsorted_row_ids = mx->nd->array([1, 0], dtype=>'int64')
+ >>> $kv->row_sparse_pull('3', out=>$a, row_ids=>$unsorted_row_ids)
+ >>> print $a->aspdl
+ [[ 1. 1. 1.]
+ [ 1. 1. 1.]
+ [ 0. 0. 0.]]
+=cut
+
+
+method row_sparse_pull(
+ Str|ArrayRef[Str] $key,
+ AI::MXNet::NDArray::RowSparse|ArrayRef[AI::MXNet::NDArray::RowSparse]|ArrayRef[ArrayRef[AI::MXNet::NDArray::RowSparse]] :$out,
+ Int :$priority=0,
+ AI::MXNet::NDArray|ArrayRef[AI::MXNet::NDArray]|ArrayRef[ArrayRef[AI::MXNet::NDArray]] :$row_ids
+)
+{
+ if(blessed $row_ids)
+ {
+ $row_ids = [$row_ids];
+ }
+ my $first_out = $out;
+ # whether row_ids are the same
+ my $single_rowid = 0;
+ if(@$row_ids == 1 and ref $out eq 'ARRAY')
+ {
+ $single_rowid = 1;
+ $first_out = [$out->[0]];
+ }
+ my ($ckeys, $cvals) = _key_value($key, $first_out);
+ my (undef, $crow_ids) = _key_value($key, $row_ids);
+ assert(
+ (@$crow_ids == @$cvals),
+ "the number of row_ids doesn't match the number of values"
+ );
+ check_call(
+ AI::MXNetCAPI::KVStorePullRowSparseEx(
+ $self->handle, scalar(@$ckeys), $ckeys, $cvals, $crow_ids, $priority
+ )
+ );
+ # the result can be copied to other devices without invoking row_sparse_pull
+ # if the indices are the same
+ if($single_rowid)
+ {
+ for my $out_i (@{ $out } [1..@{ $out }-1])
+ {
+ $out->[0]->copyto($out_i);
+ }
+ }
+}
+
+=head2 set_gradient_compression
+
+ Specifies type of low-bit quantization for gradient compression \
+ and additional arguments depending on the type of compression being used.
+
+ 2bit Gradient Compression takes a positive float `threshold`.
+ The technique works by thresholding values such that positive values in the
+ gradient above threshold will be set to threshold. Negative values whose absolute
+ values are higher than threshold, will be set to the negative of threshold.
+ Values whose absolute values are less than threshold will be set to 0.
+ By doing so, each value in the gradient is in one of three states. 2bits are
+ used to represent these states, and every 16 float values in the original
+ gradient can be represented using one float. This compressed representation
+ can reduce communication costs. The difference between these thresholded values and
+ original values is stored at the sender's end as residual and added to the
+ gradient in the next iteration.
+
+ When kvstore is 'local', gradient compression is used to reduce communication
+ between multiple devices (gpus). Gradient is quantized on each GPU which
+ computed the gradients, then sent to the GPU which merges the gradients. This
+ receiving GPU dequantizes the gradients and merges them. Note that this
+ increases memory usage on each GPU because of the residual array stored.
+
+ When kvstore is 'dist', gradient compression is used to reduce communication
+ from worker to sender. Gradient is quantized on each worker which
+ computed the gradients, then sent to the server which dequantizes
+ this data and merges the gradients from each worker. Note that this
+ increases CPU memory usage on each worker because of the residual array stored.
+ Only worker to server communication is compressed in this setting.
+ If each machine has multiple GPUs, currently this GPU to GPU or GPU to CPU communication
+ is not compressed. Server to worker communication (in the case of pull)
+ is also not compressed.
+
+ To use 2bit compression, we need to specify `type` as `2bit`.
+ Only specifying `type` would use default value for the threshold.
+ To completely specify the arguments for 2bit compression, we would need to pass
+ a dictionary which includes `threshold` like:
+ {'type': '2bit', 'threshold': 0.5}
+
+ Parameters
+ ----------
+ compression_params : HashRef
+ A dictionary specifying the type and parameters for gradient compression.
+ The key `type` in this dictionary is a
+ required string argument and specifies the type of gradient compression.
+ Currently `type` can be only `2bit`
+ Other keys in this dictionary are optional and specific to the type
+ of gradient compression.
+=cut
+
+method set_gradient_compression(HashRef[Str] $compression_params)
+{
+ if($self->type =~ /(?:device|dist)/)
+ {
+ check_call(
+ AI::MXNetCAPI::KVStoreSetGradientCompression(
+ $self->handle,
+ scalar(keys %$compression_params),
+ $compression_params
+ )
+ );
+ }
+ else
+ {
+ confess('Gradient compression is not supported for this type of kvstore');
+ }
+}
+
=head2 set_optimizer
Register an optimizer to the store
@@ -381,8 +545,8 @@ method _set_updater(Updater $updater_func)
my ($index, $input_handle, $storage_handle) = @_;
$updater_func->(
$index,
- AI::MXNet::NDArray->new(handle => $input_handle),
- AI::MXNet::NDArray->new(handle => $storage_handle)
+ AI::MXNet::NDArray->_ndarray_cls($input_handle),
+ AI::MXNet::NDArray->_ndarray_cls($storage_handle)
);
}
);
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Module.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Module.pm
index 3229d22..231d63b 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Module.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Module.pm
@@ -627,7 +627,7 @@ method bind(
{
assert(not defined $self->_p->_arg_params and not $self->_p->_aux_params);
my @param_arrays = (
- map { AI::MXNet::NDArray->zeros($_->[0]->shape, dtype => $_->[0]->dtype) }
+ map { AI::MXNet::NDArray->zeros($_->[0]->shape, dtype => $_->[0]->dtype, stype => $_->[0]->stype) }
@{ $self->_p->_exec_group->_p->param_arrays }
);
my %arg_params;
@@ -961,4 +961,10 @@ method _kvstore()
$self->_p->_kvstore;
}
+method _arg_params()
+{
+ $self->_p->_arg_params;
+}
+
+
1;
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Module/Base.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Module/Base.pm
index 4deca46..b9d5011 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Module/Base.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Module/Base.pm
@@ -253,7 +253,7 @@ method forward_backward(AI::MXNet::DataBatch $data_batch)
method score(
AI::MXNet::DataIter $eval_data,
- EvalMetric $eval_metric,
+ EvalMetric|ArrayRef[EvalMetric] $eval_metric,
Maybe[Int] :$num_batch=,
Maybe[Callback]|ArrayRef[Callback] :$batch_end_callback=,
Maybe[Callback]|ArrayRef[Callback] :$score_end_callback=,
@@ -484,7 +484,7 @@ method predict(
method fit(
AI::MXNet::DataIter $train_data,
Maybe[AI::MXNet::DataIter] :$eval_data=,
- EvalMetric :$eval_metric='acc',
+ EvalMetric|ArrayRef[EvalMetric] :$eval_metric='acc',
Maybe[Callback]|ArrayRef[Callback] :$epoch_end_callback=,
Maybe[Callback]|ArrayRef[Callback] :$batch_end_callback=,
KVStore :$kvstore='local',
@@ -500,7 +500,7 @@ method fit(
Bool :$force_init=0,
Int :$begin_epoch=0,
Int :$num_epoch,
- Maybe[EvalMetric] :$validation_metric=,
+ Maybe[EvalMetric|ArrayRef[EvalMetric]] :$validation_metric=,
Maybe[AI::MXNet::Monitor] :$monitor=
)
{
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Monitor.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Monitor.pm
index 3861641..0e46c31 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Monitor.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Monitor.pm
@@ -80,7 +80,7 @@ has 'stat_helper' => (
return sub {
my ($name, $handle) = @_;
return if(not $self->activated or not $name =~ $self->re_pattern);
- my $array = AI::MXNet::NDArray->new(handle => $handle, writable => 0);
+ my $array = AI::MXNet::NDArray->_ndarray_cls($handle, 0);
push @{ $self->queue }, [$self->step, $name, $self->stat_func->($array)];
}
},
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/NDArray.pm b/perl-package/AI-MXNet/lib/AI/MXNet/NDArray.pm
index ffee129..62e64d2 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/NDArray.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/NDArray.pm
@@ -114,9 +114,13 @@ method at(Index @indices)
method len() { $self->shape->[0] }
-method slice(Slice|AdvancedSlice @slices)
+method slice(Slice|AdvancedSlice|InternalSlice @slices)
{
confess("No slices supplied") unless @slices;
+ if(grep { not ref and /^(?:begin|end|slice)$/ } @slices)
+ {
+ return $self->SUPER::slice(@slices);
+ }
if(ref $slices[0] eq 'ARRAY' and ref $slices[0]->[0])
{
my @indices;
@@ -338,7 +342,7 @@ method _slice (
$stop
)
);
- return __PACKAGE__->new(handle => $handle, writable => $self->writable);
+ return __PACKAGE__->_ndarray_cls($handle, $self->writable);
}
=head2 _at
@@ -359,7 +363,7 @@ method _at(Index $idx)
$self->handle, $idx >=0 ? $idx : $self->shape->[0] + $idx
)
);
- return __PACKAGE__->new(handle => $handle, writable => $self->writable);
+ return __PACKAGE__->_ndarray_cls($handle, $self->writable);
}
=head2 reshape
@@ -397,7 +401,7 @@ method reshape(ArrayRef[Int] $new_shape)
$new_shape
)
);
- return __PACKAGE__->new(handle => $handle, writable => $self->writable);
+ return __PACKAGE__->_ndarray_cls($handle, $self->writable);
}
=head2 ndim
@@ -954,14 +958,21 @@ method imodulo(AI::MXNet::NDArray|Num $other, $reverse=)
:$dtype : Dtype, optional
The dtype of the NDArray, defaults to 'float32'.
+ :$stype: Stype, optional
+ The stype of the NDArray, defaults to 'default'
+
Returns
-------
out: Array
The created NDArray.
=cut
-method empty(Shape $shape, AI::MXNet::Context :$ctx=AI::MXNet::Context->current_ctx, Dtype :$dtype='float32')
+method empty(Shape $shape, AI::MXNet::Context :$ctx=AI::MXNet::Context->current_ctx, Dtype :$dtype='float32', Stype :$stype='default')
{
+ if($stype ne 'default')
+ {
+ return AI::MXNet::NDArray::Sparse->empty($stype, $shape, ctx => $ctx, dtype => $dtype);
+ }
return __PACKAGE__->new(
handle => _new_alloc_handle(
$shape,
@@ -987,6 +998,8 @@ method empty(Shape $shape, AI::MXNet::Context :$ctx=AI::MXNet::Context->current_
:$dtype : Dtype, optional
The dtype of the NDArray, defaults to 'float32'.
+ :$stype: Stype, optional
+ The stype of the NDArray, defaults to 'default'
Returns
-------
out: Array
@@ -999,9 +1012,14 @@ method zeros(
Dtype :$dtype='float32',
Maybe[AI::MXNet::NDArray] :$out=,
Maybe[Str] :$name=,
- Maybe[Str] :$__layout__=
+ Maybe[Str] :$__layout__=,
+ Stype :$stype='default'
)
{
+ if($stype ne 'default')
+ {
+ return AI::MXNet::NDArray::Sparse->zeros($stype, $shape, ctx => $ctx, dtype => $dtype, out => $out);
+ }
return __PACKAGE__->_zeros({ shape => $shape, ctx => "$ctx", dtype => $dtype, ($out ? (out => $out) : ()) });
}
@@ -1032,7 +1050,7 @@ method ones(
Dtype :$dtype='float32',
Maybe[AI::MXNet::NDArray] :$out=,
Maybe[Str] :$name=,
- Maybe[Str] :$__layout__=
+ Maybe[Str] :$__layout__=,
)
{
return __PACKAGE__->_ones({ shape => $shape, ctx => "$ctx", dtype => $dtype, ($out ? (out => $out) : ()) });
@@ -1094,14 +1112,19 @@ method full(
The created NDArray.
=cut
-method array(PDL|PDL::Matrix|ArrayRef|AI::MXNet::NDArray $source_array, AI::MXNet::Context :$ctx=AI::MXNet::Context->current_ctx, Dtype :$dtype='float32')
+method array(PDL|PDL::Matrix|PDL::CCS::Nd|ArrayRef|AI::MXNet::NDArray $source_array, AI::MXNet::Context :$ctx=AI::MXNet::Context->current_ctx, Dtype :$dtype='float32')
{
if(blessed $source_array and $source_array->isa('AI::MXNet::NDArray'))
{
+ return AI::MXNet::NDArray::Sparse->array($source_array, ctx => $ctx, dtype => $dtype) unless $source_array->stype eq 'default';
my $arr = __PACKAGE__->empty($source_array->shape, ctx => $ctx, dtype => $dtype);
$arr .= $source_array;
return $arr;
}
+ elsif(blessed $source_array and $source_array->isa('PDL::CCS::Nd'))
+ {
+ return AI::MXNet::NDArray::Sparse->array($source_array, ctx => $ctx, dtype => $dtype);
+ }
my $pdl_type = PDL::Type->new(DTYPE_MX_TO_PDL->{ $dtype });
if(not blessed($source_array))
{
@@ -1261,7 +1284,7 @@ method load(Str $filename)
my ($handles, $names) = check_call(AI::MXNetCAPI::NDArrayLoad($filename));
if (not @$names)
{
- return [map { __PACKAGE__->new(handle => $_) } @$handles];
+ return [map { __PACKAGE__->_ndarray_cls($_) } @$handles];
}
else
{
@@ -1269,7 +1292,7 @@ method load(Str $filename)
my $h = @$handles;
confess("Handles [$h] and names [$n] count mismatch") unless $h == $n;
my %ret;
- @ret{ @$names } = map { __PACKAGE__->new(handle => $_) } @$handles;
+ @ret{ @$names } = map { __PACKAGE__->_ndarray_cls($_) } @$handles;
return \%ret;
}
}
@@ -1386,17 +1409,49 @@ sub _new_empty_handle
func _new_alloc_handle($shape, $ctx, $delay_alloc, $dtype)
{
- my $hdl = check_call(AI::MXNetCAPI::NDArrayCreateEx(
- $shape,
- scalar(@$shape),
- $ctx->device_type_id,
- $ctx->device_id,
- $delay_alloc,
- $dtype)
+ my $hdl = check_call(
+ AI::MXNetCAPI::NDArrayCreateEx(
+ $shape,
+ scalar(@$shape),
+ $ctx->device_type_id,
+ $ctx->device_id,
+ $delay_alloc,
+ $dtype
+ )
+ );
+ return $hdl;
+}
+
+method _new_from_shared_mem($shared_pid, $shared_id, $shape, $dtype)
+{
+ my $hdl = check_call(
+ AI::MXNetCAPI::NDArrayCreateFromSharedMem(
+ $shared_pid,
+ $shared_id,
+ $shape,
+ scalar(@$shape),
+ DTYPE_STR_TO_MX->{$dtype}
+ )
);
return $hdl;
}
+=head2 tostype
+
+ Return a copy of the array with chosen storage type.
+
+ Returns
+ -------
+ AI::MXNet::NDArray or AI::MXNet::NDArray::CSR or AI::MXNet::NDArray::RowSparse
+ A copy of the array with the chosen storage stype
+=cut
+
+method tostype(Stype $stype)
+{
+ return $self->cast_storage(stype => $stype);
+}
+
+
=head2 waitall
Wait for all async operations to finish in MXNet.
@@ -1444,7 +1499,7 @@ method _fresh_grad(Maybe[Bool] $state=)
method detach()
{
my $handle = check_call(AI::MXNetCAPI::NDArrayDetach($self->handle));
- return __PACKAGE__->new(handle => $handle);
+ return __PACKAGE__->_ndarray_cls($handle);
}
=head2 attach_grad
@@ -1468,7 +1523,7 @@ method attach_grad(GradReq :$grad_req='write', Maybe[Str] :$stype=)
my $grad;
if(defined $stype)
{
- $grad = __PACKAGE__->_zeros($self->shape, stype=>$stype);
+ $grad = __PACKAGE__->zeros($self->shape, stype => $stype);
}
else
{
@@ -1494,7 +1549,7 @@ method grad()
{
my $handle = check_call(AI::MXNetCAPI::NDArrayGetGrad($self->handle));
return undef unless defined $handle;
- return __PACKAGE__->new(handle => $handle);
+ return __PACKAGE__->_ndarray_cls($handle);
}
=head2 backward
@@ -1543,5 +1598,6 @@ EOV
sub contrib { 'AI::MXNet::Contrib::NDArray' }
sub random { 'AI::MXNet::Random' }
+sub sparse { 'AI::MXNet::NDArray::Sparse' }
__PACKAGE__->meta->make_immutable;
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Base.pm b/perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Base.pm
index 8efe9d7..6b4b3e1 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Base.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Base.pm
@@ -156,10 +156,10 @@ func _make_ndarray_function($handle, $func_name)
}
for my $key (keys %kwargs)
{
- $kwargs{ $key } = "(" .join(", ", @{ $kwargs{ $key } }) .")"
+ $kwargs{ $key } = "(" .join(", ", map { defined($_) ? $_ : 'None' } @{ $kwargs{ $key } }) .")"
if ref $kwargs{ $key } eq 'ARRAY';
}
- my $out = check_call(AI::MXNetCAPI::ImperativeInvoke(
+ my ($out, $stypes) = check_call(AI::MXNetCAPI::ImperativeInvokeEx(
$handle,
scalar(@ndargs),
\@ndargs,
@@ -170,11 +170,12 @@ func _make_ndarray_function($handle, $func_name)
return $original_output if $original_output;
if(@$out == 1)
{
- return $class->new(handle => $out->[0]);
+ return __PACKAGE__->_ndarray_cls($out->[0], 1, $stypes->[0]);
}
else
{
- return [map { $class->new(handle => $_) } @$out];
+ my $i = 0;
+ return [map { __PACKAGE__->_ndarray_cls($_, 1, $stypes->[$i++]) } @$out];
}
};
$function_meta{ $generic_ndarray_function }{__name__} = $func_name;
@@ -182,6 +183,40 @@ func _make_ndarray_function($handle, $func_name)
return $generic_ndarray_function;
}
+method _ndarray_cls($handle, $writable=1, $stype=STORAGE_TYPE_UNDEFINED)
+{
+ if($stype eq STORAGE_TYPE_UNDEFINED)
+ {
+ $stype = __PACKAGE__->_storage_type($handle);
+ }
+ if($stype eq STORAGE_TYPE_DEFAULT)
+ {
+ return AI::MXNet::NDArray->new(handle => $handle, writable => $writable);
+ }
+ elsif($stype eq STORAGE_TYPE_CSR)
+ {
+ return AI::MXNet::NDArray::CSR->new(handle => $handle, writable => $writable);
+ }
+ elsif($stype eq STORAGE_TYPE_ROW_SPARSE)
+ {
+ return AI::MXNet::NDArray::RowSparse->new(handle => $handle, writable => $writable);
+ }
+ else
+ {
+ confess("unknown storage type: $stype");
+ }
+}
+
+method _storage_type($handle)
+{
+ scalar(check_call(AI::MXNetCAPI::NDArrayGetStorageType($handle)));
+}
+
+method stype()
+{
+ return STORAGE_TYPE_ID_TO_STR->{ __PACKAGE__->_storage_type($self->handle) };
+}
+
method _init_ndarray_module()
{
my $op_names = check_call(AI::MXNetCAPI::ListAllOpNames());
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Sparse.pm b/perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Sparse.pm
new file mode 100644
index 0000000..cbc9f7d
--- /dev/null
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Sparse.pm
@@ -0,0 +1,1342 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+package AI::MXNet::NDArray::Sparse;
+use strict;
+use warnings;
+use AI::MXNet::Base;
+use AI::MXNet::Function::Parameters;
+use Mouse;
+extends 'AI::MXNet::NDArray';
+
+=head1 NAME
+
+ AI::MXNet::NDArray::Sparse - Sparse NDArray API of MXNet
+=cut
+
+=head1 DESCRIPTION
+
+ The base class of an NDArray stored in a sparse storage format.
+ See AI::MXNet::NDArray::CSR and AI::MXNet::NDArray::RowSparse for more details.
+=cut
+
+method _new_alloc_handle(
+ Stype $stype,
+ Shape $shape,
+ AI::MXNet::Context $ctx,
+ Bool $delay_alloc,
+ Dtype $dtype,
+ AuxTypes $aux_types,
+ Maybe[ArrayRef[Shape]] $aux_shapes=
+)
+{
+ confess("only int64 is supported for aux types")
+ if (grep { $_ ne 'int64' } @$aux_types);
+ my $aux_type_ids = [map { DTYPE_STR_TO_MX->{$_} } @$aux_types];
+ $aux_shapes //= [map { [0] } @$aux_types];
+ my $aux_shape_lens = [map { scalar(@$_) } @$aux_shapes];
+ @$aux_shapes = map { @$_ } @$aux_shapes;
+ my $num_aux = @{ $aux_types };
+ my $handle = check_call(
+ AI::MXNetCAPI::NDArrayCreateSparseEx(
+ STORAGE_TYPE_STR_TO_ID->{$stype},
+ $shape,
+ scalar(@$shape),
+ $ctx->device_type_id,
+ $ctx->device_id,
+ $delay_alloc,
+ DTYPE_STR_TO_MX->{$dtype},
+ scalar(@$aux_types),
+ $aux_type_ids,
+ $aux_shape_lens,
+ $aux_shapes
+ )
+ );
+}
+
+method _class_name()
+{
+ my $class = ref $self || $self;
+ $class;
+}
+
+sub not_implemented { confess "Not implemented" }
+use overload '""' => sub {
+ my $self = shift;
+ my $shape_info = join('x', @{ $self->shape });
+ sprintf("\n<%s, %s @%s>", $self->_class_name, $shape_info, $self->context);
+ },
+ '+=' => \¬_implemented,
+ '-=' => \¬_implemented,
+ '*=' => \¬_implemented,
+ '/=' => \¬_implemented;
+{
+ no warnings 'redefine';
+ *_sync_copyfrom = *_at = *_slice = *reshape = *size = \¬_implemented;
+}
+
+method _aux_type(Int $i)
+{
+ return DTYPE_MX_TO_STR->{
+ check_call(
+ AI::MXNetCAPI::NDArrayGetAuxType(
+ $self->handle, $i
+ )
+ )
+ }
+}
+
+method _num_aux()
+{
+ return scalar(@{ STORAGE_AUX_TYPES->{ $self->stype } });
+}
+
+method _aux_types()
+{
+ [map { $self->_aux_type($_) } 0..$self->_num_aux-1];
+}
+
+=head2 aspdl
+
+ Return a dense PDL object with value copied from this array
+=cut
+
+method aspdl()
+{
+ return $self->tostype('default')->aspdl;
+}
+
+=head2 astype
+
+ Returns a copy of the array after casting to a specified type.
+ Parameters
+ ----------
+ dtype : Dtype
+ The type of the returned array.
+ Examples
+ --------
+ >>> $x = mx->nd->sparse->zeros('row_sparse', [2,3], dtype=>'float32')
+ >>> $y = $x->astype('int32')
+ >>> $y->dtype
+ <type 'int32'>
+=cut
+
+method astype(Dtype $dtype)
+{
+ my $res = $self->zeros(
+ $self->stype, $self->shape, ctx => $self->context,
+ dtype => $dtype
+ );
+ $self->copyto($res);
+ return $res;
+}
+
+=head2 copyto
+
+ Copies the value of this array to another array.
+
+ Parameters
+ ----------
+ other : NDArray or NDArray::CSR or NDArray::RowSparse or Context
+ The destination array or context.
+
+ Returns
+ -------
+ NDArray or CSRNDArray::CSR or NDArray::RowSparse
+ The copied array.
+=cut
+
+method copyto(AI::MXNet::NDArray|AI::MXNet::Context $other)
+{
+ if($other->isa('AI::MXNet::NDArray'))
+ {
+ if($self->handle eq $other->handle)
+ {
+ Carp::cluck('You are attempting to copy an array to itself');
+ return;
+ }
+ else
+ {
+ return __PACKAGE__->_copyto($self, out => $other);
+ }
+ }
+ elsif($other->isa('AI::MXNet::Context'))
+ {
+ my $hret = __PACKAGE__->_ndarray_cls(
+ __PACKAGE__->_new_alloc_handle(
+ $self->stype, $self->shape, $other, 1, $self->dtype, $self->_aux_types
+ )
+ );
+ return __PACKAGE__->_copyto($self, out=>$hret)
+ }
+}
+
+=head2 check_format
+
+ Check whether the NDArray format is valid.
+
+ Parameters
+ ----------
+ full_check : bool, optional
+ If `True`, rigorous check, O(N) operations. Otherwise
+ basic check, O(1) operations (default True).
+=cut
+
+method check_format(Bool $full_check=1)
+{
+ scalar(check_call(AI::MXNetCAPI::NDArraySyncCheckFormat($self->handle, $full_check)));
+}
+
+=head2 _data
+
+ A deep copy NDArray of the data array associated with the BaseSparseNDArray.
+
+ This function blocks. Do not use it in performance critical code.
+=cut
+
+method _data()
+{
+ $self->wait_to_read;
+ my $handle = check_call(AI::MXNetCAPI::NDArrayGetDataNDArray($self->handle));
+ return AI::MXNet::NDArray->new(handle => $handle);
+}
+
+=head2 _aux_data
+
+ Get a deep copy NDArray of the i-th aux data array associated with the
+ AI::MXNet::NDArray::Sparse
+
+ This function blocks. Do not use it in performance critical code.
+=cut
+
+method _aux_data(Int $i)
+{
+ $self->wait_to_read;
+ my $handle = check_call(AI::MXNetCAPI::NDArrayGetAuxNDArray($self->handle, $i));
+ return AI::MXNet::NDArray->new(handle => $handle);
+}
+
+package AI::MXNet::NDArray::CSR;
+use AI::MXNet::Base;
+use Mouse;
+extends 'AI::MXNet::NDArray::Sparse';
+
+=head1 NAME
+
+ AI::MXNet::NDArray::CSR - A sparse representation of 2D NDArray in the Compressed Sparse Row format.
+=cut
+
+=head1 DESCRIPTION
+
+ A AI::MXNet::NDArray::CSR represents an AI::MXNet::NDArray as three separate arrays: `data`,
+ `indptr` and `indices`. It uses the CSR representation where the column indices for
+ row i are stored in ``indices[indptr[i]:indptr[i+1]]`` and their corresponding values are stored
+ in ``data[indptr[i]:indptr[i+1]]``.
+
+ The column indices for a given row are expected to be sorted in ascending order.
+ Duplicate column entries for the same row are not allowed.
+
+ Example
+ -------
+ >>> $a = mx->nd->array([[0, 1, 0], [2, 0, 0], [0, 0, 0], [0, 0, 3]]);
+ >>> $a = $a->tostype('csr');
+ >>> $a->data->aspdl;
+ [ 1 2 3]
+ >>> $a->indices->aspdl
+ [1 0 2]
+ >>> $a->indptr->aspdl
+ [0 1 2 2 3]
+
+ See Also
+ --------
+ csr_matrix: Several ways to construct a CSRNDArray
+=cut
+
+# def __reduce__(self):
+# return CSRNDArray, (None,), super(CSRNDArray, self).__getstate__()
+
+use overload '+=' => sub { ($_[0] + $_[1])->copyto($_[0]) },
+ '-=' => sub { ($_[0] - $_[1])->copyto($_[0]) },
+ '*=' => sub { ($_[0] * $_[1])->copyto($_[0]) },
+ '/=' => sub { ($_[0] / $_[1])->copyto($_[0]) };
+
+=head2 slice
+
+ Returns a sliced view of this array.
+
+ Parameters
+ ----------
+ key : int or array ref
+ Indexing key.
+
+ Examples
+ --------
+ >>> $indptr = [0, 2, 3, 6];
+ >>> $indices = [0, 2, 2, 0, 1, 2];
+ >>> $data = [1, 2, 3, 4, 5, 6];
+ >>> $a = mx->nd->sparse->csr_matrix([$data, $indices, $indptr], shape=>[3, 3])
+ >>> $a->aspdl
+ [[ 1 0 2]
+ [ 0 0 3]
+ [ 4 5 6]]
+ >>> $a->slice([1,2])->aspdl
+ [[ 0 0 3]]
+ >>> $a->slice(1)->aspdl
+ [[ 0 0 3]]
+ >>> $a->[-1]->aspdl
+ [[ 4 5 6]]
+=cut
+
+method slice(Slice|InternalSlice @slices)
+{
+ if(grep { /^begin|end|slice$/ } @slices)
+ {
+ return $self->SUPER::slice(@slices);
+ }
+ my $slice = $slices[0];
+ my ($begin, $end);
+ if(not ref $slice)
+ {
+ if($slice < 0)
+ {
+ $begin = $self->shape->[0] + $slice;
+ }
+ else
+ {
+ $begin = $slice;
+ }
+ $end = $begin;
+ }
+ else
+ {
+ ($begin, $end) = @{ $slice };
+ $end //= $self->shape->[0] - 1;
+ if($begin < 0)
+ {
+ $begin += $self->shape->[0];
+ }
+ if($end < 0)
+ {
+ $end += $self->shape->[0];
+ }
+ }
+ return $self->SUPER::slice(begin => $begin, end => $end + 1);
+}
+
+
+=head2 set
+
+ Set self to value. Also usable as overloaded .=
+
+ Parameters
+ ----------
+ value : AI::MXNet::NDArray or AI::MXNet::NDArray::CSR
+ or PDL/PDL::CCS::Nd/perl array ref in PDL constructor format
+ The value to set.
+
+ Examples
+ --------
+ >>> $src = mx->nd->sparse->zeros('csr', [3,3])
+ >>> $src->aspdl
+ [[ 0 0 0]
+ [ 0 0 0]
+ [ 0 0 0]]
+ >>> # AI::MXNet::NDArray::CSR with same storage type
+ >>> $x = mx->nd->ones('row_sparse', [3,3])->tostype('csr')
+ >>> $x .= $src
+ >>> $x->aspdl
+ [[ 1 1 1]
+ [ 1 1 1]
+ [ 1 1 1]]
+ >>> # assign NDArray to AI::MXNet::NDArray::CSR
+ >>> $x .= mx->nd->ones([3,3]) * 2
+ >>> $x->aspdl
+ [[ 2 2 2]
+ [ 2 2 2]
+ [ 2 2 2]]
+=cut
+
+method set(AcceptableInput $other, $reverse=)
+{
+ confess('Failed to assign to a readonly CSR') unless $self->writable;
+ if($other->isa('AI::MXNet::NDArray'))
+ {
+ if($self->handle ne $other->handle)
+ {
+ $other->copyto($self);
+ }
+ }
+ else
+ {
+ my $tmp = __PACKAGE__->array($other, ((not ref $other) ? ( pdl => $self->aspdl) : ()));
+ $tmp->copyto($self);
+ }
+}
+
+use overload '.=' => \&set;
+
+=head2 indices
+
+ A deep copy NDArray of the indices array of the AI::MXNet::NDArray::CSR.
+ This generates a deep copy of the column indices of the current `csr` matrix.
+
+ Returns
+ -------
+ NDArray
+ This AI::MXNet::NDArray::CSR indices array.
+=cut
+
+method indices()
+{
+ return $self->_aux_data(1);
+}
+
+=head2 indptr
+
+ A deep copy NDArray of the inptr array of the AI::MXNet::NDArray::CSR.
+ This generates a deep copy of the indptr of the current `csr` matrix.
+
+ Returns
+ -------
+ NDArray
+ This AI::MXNet::NDArray::CSR indptr array.
+=cut
+
+method indptr()
+{
+ return $self->_aux_data(0);
+}
+
+=head2 data
+
+ A deep copy NDArray of the data array of the AI::MXNet::NDArray::CSR.
+ This generates a deep copy of the data of the current `csr` matrix.
+
+ Returns
+ -------
+ NDArray
+ This AI::MXNet::NDArray::CSR data array.
+=cut
+
+method data()
+{
+ return $self->_data;
+}
+
+=head2 tostype
+
+ Return a copy of the array with chosen storage type.
+
+ Returns
+ -------
+ NDArray or AI::MXNet::NDArray::CSR
+ A copy of the array with the chosen storage stype
+=cut
+
+method tostype(Stype $stype)
+{
+ if($stype eq 'row_sparse')
+ {
+ confess("cast_storage from csr to row_sparse is not supported");
+ }
+ return $self->cast_storage(stype => $stype);
+}
+
+=head2 copyto
+
+ Copies the value of this array to another array.
+
+ If $other is a AI::MXNet::NDArray or AI::MXNet::NDArray::CSR object, then $other->shape and
+ $self->shape should be the same. This function copies the value from
+ $self to $other.
+
+ If $other is a context, a new AI::MXNet::NDArray::CSR will be first created on
+ the target context, and the value of $self is copied.
+
+ Parameters
+ ----------
+ $other : AI::MXNet::NDArray or AI::MXNet::NDArray::CSR or AI::MXNet::Context
+ The destination array or context.
+
+ Returns
+ -------
+ AI::MXNet::NDArray or AI::MXNet::NDArray::CSR
+=cut
+
+method copyto(AI::MXNet::Context|AI::MXNet::NDArray $other)
+{
+ if($other->isa('AI::MXNet::Context'))
+ {
+ return $self->SUPER::copyto($other);
+ }
+ else
+ {
+ my $stype = $other->stype;
+ if($stype eq 'default' or $stype eq 'csr')
+ {
+ return return $self->SUPER::copyto($other);
+ }
+ else
+ {
+ confess("copyto does not support destination NDArray stype $stype");
+ }
+ }
+}
+
+=head2
+
+ Returns a PDL::CCS::Nd object with value copied from this array
+=cut
+
+method aspdlccs()
+{
+ return ascsr($self->data->aspdl, $self->indptr->aspdl, $self->indices->aspdl, $self->shape);
+}
+
+package AI::MXNet::NDArray::RowSparse;
+use Mouse;
+extends 'AI::MXNet::NDArray::Sparse';
+
+=head1 NAME
+
+ AI::MXNet::NDArray::RowSparse - A sparse representation of a set of NDArray row slices at given indices.
+=cut
+
+=head1 DESCRIPTION
+
+ A AI::MXNet::NDArray::RowSparse represents a multidimensional NDArray using two separate arrays: `data` and
+ `indices`. The number of dimensions has to be at least 2.
+
+ - data: an NDArray of any dtype with shape [D0, D1, ..., Dn].
+ - indices: a 1-D int64 NDArray with shape [D0] with values sorted in ascending order.
+
+ The `indices` stores the indices of the row slices with non-zeros,
+ while the values are stored in `data`. The corresponding NDArray ``dense``
+ represented by AI::MXNet::NDArray::RowSparse ``rsp`` has
+
+ ``dense[rsp.indices[i], :, :, :, ...] = rsp.data[i, :, :, :, ...]``
+
+ >>> $dense->aspdl
+ [[ 1 2 3 ]
+ [ 0 0 0 ]
+ [ 4 0 5 ]
+ [ 0 0 0 ]
+ [ 0 0 0 ]]
+ >>> $rsp = $dense->tostype('row_sparse');
+ >>> $rsp->indices->aspdl
+ [ 0 2 ]
+ >>> $rsp->data->aspdl
+ [[ 1 2 3 ]
+ [ 4 0 5 ]]
+
+ A AI::MXNet::NDArray::RowSparse is typically used to represent non-zero row slices of a large NDArray
+ of shape [LARGE0, D1, .. , Dn] where LARGE0 >> D0 and most row slices are zeros.
+
+ AI::MXNet::NDArray::RowSparse is used principally in the definition of gradients for operations
+ that have sparse gradients (e.g. sparse dot and sparse embedding).
+
+ See Also
+ --------
+ row_sparse_array: Several ways to construct a AI::MXNet::NDArray::RowSparse
+=cut
+
+use overload '+=' => sub { ($_[0] + $_[1])->copyto($_[0]) },
+ '-=' => sub { ($_[0] - $_[1])->copyto($_[0]) },
+ '*=' => sub { ($_[0] * $_[1])->copyto($_[0]) },
+ '/=' => sub { ($_[0] / $_[1])->copyto($_[0]) };
+
+method slice(@args) { confess("not implemented") }
+
+=head2 set
+
+ Set self to value. Also usable as overloaded .=
+
+ Parameters
+ ----------
+ value : AI::MXNet::NDArray or AI::MXNet::NDArray::CSR
+ or PDL/PDL::CCS::Nd/perl array ref in PDL constructor format
+ The value to set.
+
+ Examples
+ --------
+ >>> $src = mx->nd->sparse->zeros('raw_sparse', [3,3])
+ >>> $src->aspdl
+ [[ 0 0 0]
+ [ 0 0 0]
+ [ 0 0 0]]
+ >>> # AI::MXNet::NDArray::RowSparse with same storage type
+ >>> $x = mx->nd->ones('row_sparse', [3,3])
+ >>> $src .= $x
+ >>> $src->aspdl
+ [[ 1 1 1]
+ [ 1 1 1]
+ [ 1 1 1]]
+ >>> # assign NDArray to AI::MXNet::NDArray::RowSparse
+ >>> $x .= mx->nd->ones([3,3]) * 2
+ >>> $x->aspdl
+ [[ 2 2 2]
+ [ 2 2 2]
+ [ 2 2 2]]
+=cut
+
+method set(AcceptableInput $other, $reverse=)
+{
+ confess('Failed to assign to a readonly RowSparse') unless $self->writable;
+ if($other->isa('AI::MXNet::NDArray'))
+ {
+ if($self->handle ne $other->handle)
+ {
+ $other->copyto($self);
+ }
+ }
+ else
+ {
+ my $tmp = __PACKAGE__->array($other, ((not ref $other) ? ( pdl => $self->aspdl) : ()));
+ $tmp->copyto($self);
+ }
+}
+
+use overload '.=' => \&set;
+
+=head2 data
+
+ A deep copy NDArray of the data array of the AI::MXNet::NDArray::RowSparse.
+ This generates a deep copy of the data of the current `row_sparse` matrix.
+
+ Returns
+ -------
+ NDArray
+ This AI::MXNet::NDArray::RowSparse data array.
+=cut
+
+method data()
+{
+ return $self->_data;
+}
+
+=head2 indices
+
+ A deep copy NDArray of the indices array of the AI::MXNet::NDArray::RowSparse.
+ This generates a deep copy of the column indices of the current `row_sparse` matrix.
+
+ Returns
+ -------
+ NDArray
+ This AI::MXNet::NDArray::RowSparse indices array.
+=cut
+
+method indices()
+{
+ return $self->_aux_data(0);
+}
+
+=head2 data
+
+ A deep copy NDArray of the data array of the AI::MXNet::NDArray::RowSparse.
+ This generates a deep copy of the data of the current `row_sparse` matrix.
+
+ Returns
+ -------
+ NDArray
+ This AI::MXNet::NDArray::RowSparse data array.
+=cut
+
+=head2 tostype
+
+ Return a copy of the array with chosen storage type.
+
+ Returns
+ -------
+ NDArray or RowSparseNDArray
+ A copy of the array with the chosen storage stype
+=cut
+
+method tostype(Stype $stype)
+{
+ if($stype eq 'csr')
+ {
+ confess("cast_storage from row_sparse to csr is not supported");
+ }
+ return $self->cast_storage(stype => $stype);
+}
+
+
+=head2 copyto
+
+ Copies the value of this array to another array.
+
+ If $other is a AI::MXNet::NDArray or AI::MXNet::NDArray::RawSparse object, then $other->shape and
+ $self->shape should be the same. This function copies the value from
+ $self to $other.
+
+ If $other is a context, a new AI::MXNet::NDArray::RawSparse will be first created on
+ the target context, and the value of $self is copied.
+
+ Parameters
+ ----------
+ $other : AI::MXNet::NDArray or AI::MXNet::NDArray::RawSparse or AI::MXNet::Context
+ The destination array or context.
+
+ Returns
+ -------
+ AI::MXNet::NDArray or AI::MXNet::NDArray::RawSparse
+=cut
+
+method copyto(AI::MXNet::Context|AI::MXNet::NDArray $other)
+{
+ if($other->isa('AI::MXNet::Context'))
+ {
+ return $self->SUPER::copyto($other);
+ }
+ else
+ {
+ my $stype = $other->stype;
+ if($stype eq 'default' or $stype eq 'row_sparse')
+ {
+ return return $self->SUPER::copyto($other);
+ }
+ else
+ {
+ confess("copyto does not support destination NDArray stype $stype");
+ }
+ }
+}
+
+package AI::MXNet::NDArray::Sparse;
+
+# Prepare `source_array` so that it can be used to construct NDArray.
+# `source_array` is converted to a `pdl` if it's neither an `NDArray`
+# nor a `pdl`.
+
+method _prepare_src_array($source_array, Dtype $dtype)
+{
+ my $pdl_type = PDL::Type->new(DTYPE_MX_TO_PDL->{ $dtype });
+ if(not blessed($source_array))
+ {
+ $source_array = eval {
+ pdl($pdl_type, $source_array);
+ };
+ confess($@) if $@;
+ }
+ elsif($source_array->isa('AI::MXNet::NDArray'))
+ {
+ return $source_array;
+ }
+ $source_array = pdl($pdl_type, [@{ $source_array->unpdl } ? $source_array->unpdl->[0] : 0 ]) unless @{ $source_array->shape->unpdl };
+ return $source_array;
+}
+
+
+# Prepare the value of dtype if `dtype` is undef. If `src_array` is an NDArray, PDL
+# or PDL::CCS::Ne, return src_array->dtype. float32 is returned otherwise.
+
+method _prepare_default_dtype($src_array, $dtype)
+{
+ if(not defined $dtype)
+ {
+ if(blessed $src_array)
+ {
+ $dtype = $src_array->dtype;
+ }
+ else
+ {
+ $dtype = 'float32';
+ }
+ }
+ return $dtype;
+}
+
+use Data::Dumper;
+method _check_shape($s1, $s2)
+{
+ my ($ps1, $ps2) = map { (blessed($_) and $_->isa('AI::MXNet:NDArray')) ? pdl($_->shape) : blessed($_) ? $_ : pdl($_) } ($s1, $s2);
+ return 1 unless defined $s2;
+ ($ps1 == $ps2)->all
+ or
+ confess("Shape mismatch detected. " . Dumper(blessed ($s1) ? $s1->undpl : $s1 ) . " v.s. " . Dumper(blessed ($s2) ? $s2 : $s2));
+}
+
+method coo_matrix(@args)
+{
+ my ($data, $row, $col, $shape) = map { blessed $_ ? $_ : pdl($_) } @args;
+ my @which;
+ my $i = 0;
+ my $j = 0;
+ for (my $i = 0; $i < $row->nelem; $i++)
+ {
+ push @which, [$row->at($i), $col->at($i)];
+ }
+ return PDL::CCS::Nd->newFromWhich(
+ pdl(\@which), $data, pdims => $shape
+ )->xchg(0, 1);
+}
+
+=head2 csr_matrix
+
+ Creates a AI::MXNet::NDArray::CSR, an 2D array with compressed sparse row (CSR) format.
+
+ The AI::MXNet::NDArray::CSR can be instantiated in several ways:
+
+ - csr_matrix($arg1, Maybe[AI::MXNet::Context] :$ctx=, Maybe[Shape] :$shape, Maybe [Dtype] :$dtype=)
+ $ctx, $shape, $dtype are optional
+ $arg1 can be given in following variants
+
+ - to construct a AI::MXNet::NDArray::CSR with a dense 2D array $arg1
+ - $arg1 is in AI::MXNet::NDArray::array input format
+
+ - to construct a AI::MXNet::NDArray::CSR with a sparse 2D array $arg1
+ $arg1 is AI::MXNet::NDArray::CSR or PDL::CCS::Nd - A sparse matrix.
+ PDL::CCS::Nd is expected to be converted internally into CSR format
+ AI::MXNet injects 'tocsr' method into PDL and PDL::CCS::Nd modules for this purpose.
+
+ - to construct an empty AI::MXNet::NDArray::CSR with shape $arg1 = [$M, $N]
+ - $M - Number of rows in the matrix
+ - $N - Number of columns in the matrix
+
+ - to construct a AI::MXNet::NDArray::CSR based on the definition of compressed sparse row format
+ using three separate arrays,
+ where the column indices for row i are stored in ``indices[indptr[i]:indptr[i+1]]``
+ and their corresponding values are stored in ``data[indptr[i]:indptr[i+1]]``.
+ The column indices for a given row are expected to be **sorted in ascending order.**
+ Duplicate column entries for the same row are not allowed.
+ In this case $arg1 = [$data, $indices, $indptr]
+ $data, $indices, $indptr must be given in the AI::MXNet::NDArray::array input format
+ - $data - holds all the non-zero entries of the matrix in row-major order.
+ - $indices - stores the column index for each non-zero element in $data.
+ stores the column index for each non-zero element in $data.
+ - $indptr - stores the offset into $data of the first non-zero element number of each
+ row of the matrix.
+
+ to construct a AI::MXNet::NDArray::CSR based on the COOrdinate format
+ using three seperate arrays,
+ where ``row[i]`` is the row index of the element,
+ ``col[i]`` is the column index of the element
+ and ``data[i]`` is the data corresponding to the element. All the missing
+ elements in the input are taken to be zeroes.
+ In this case $arg1 = [$data, [$row, $col]]
+ $data, $row, $col must be given in the AI::MXNet::NDArray::array input format
+ $data - holds all the non-zero entries of the matrix in COO format.
+ $row - stores the row index for each non zero element in $data.
+ - **col** (*array_like*) - An object exposing the array interface, which
+ $col - stores the col index for each non zero element in $data.
+
+ Returns
+ -------
+ AI::MXNet::NDArray::CSR
+ A AI::MXNet::NDArray::CSR with the 'csr' storage representation.
+
+ Example
+ -------
+ >>> $a = mx->nd->sparse->csr_matrix([[1, 2, 3], [1, 0, 2], [0, 1, 2, 2, 3]], shape => [4, 3])
+ >>> $a->aspdl
+ [[ 0 1 0]
+ [ 2 0 0]
+ [ 0 0 0]
+ [ 0 0 3]]
+
+ See Also
+ --------
+ CSRNDArray : MXNet NDArray in compressed sparse row format.
+=cut
+method csr_matrix(
+ $arg1,
+ Maybe[Shape|PDL] :$shape=,
+ Maybe[AI::MXNet::Context] :$ctx=AI::MXNet::Context->current_ctx,
+ Maybe[Dtype] :$dtype=
+)
+{
+ if(not defined $arg1)
+ {
+ return __PACKAGE__->empty('csr', $shape, ctx => $ctx, (defined $dtype ? (dtype => $dtype) : ()));
+ }
+ # construct a csr matrix from (M, N) or (data, indices, indptr)
+ if(ref $arg1 eq 'ARRAY')
+ {
+ my $arg_len = @{ $arg1 };
+ if($arg_len == 2)
+ {
+ # construct a sparse csr matrix from
+ # scipy coo matrix if input format is coo
+ if(ref $arg1->[1] eq 'ARRAY' and @{ $arg1->[1] } == 2)
+ {
+ my $coo = __PACKAGE__->coo_matrix($arg1->[0], @{ $arg1->[1] }, $shape);
+ __PACKAGE__->_check_shape($coo->shape, $shape);
+ return __PACKAGE__->array($coo, ctx => $ctx, dtype => $dtype);
+ }
+ else
+ {
+ # empty matrix with shape
+ __PACKAGE__->_check_shape($arg1, $shape);
+ return __PACKAGE__->empty('csr', $arg1, ctx=>$ctx, dtype=>$dtype);
+ }
+ }
+ elsif($arg_len == 3)
+ {
+ # data, indices, indptr
+ return __PACKAGE__->_csr_matrix_from_definition(
+ @{ $arg1 }, shape => $shape,
+ ctx => $ctx, dtype => $dtype
+ );
+ }
+ else
+ {
+ confess("Unexpected length of input array: " . Dumper($arg1));
+ }
+ }
+ else
+ {
+ # construct a csr matrix from a sparse / dense one
+ if(blessed $arg1 and ($arg1->isa('AI::MXNet::NDArray::CSR') or $arg1->isa('PDL::CCS::Nd')))
+ {
+ # construct a csr matrix from scipy or CSRNDArray
+ __PACKAGE__->_check_shape($arg1->shape, $shape);
+ return __PACKAGE__->array($arg1, ctx => $ctx, dtype => $dtype);
+ }
+ elsif(blessed $arg1 and $arg1->isa('AI::MXNet::NDArray::RowSparse'))
+ {
+ confess("Unexpected input type: AI::MXNet::NDArray::RowSparse");
+ }
+ else
+ {
+ # construct a csr matrix from a dense one
+ # prepare default ctx and dtype since mx.nd.array doesn't use default values
+ # based on source_array
+ $dtype = __PACKAGE__->_prepare_default_dtype($arg1, $dtype);
+ # create dns array with provided dtype. ctx is not passed since copy across
+ # ctx requires dtype to be the same
+ my $dns = __PACKAGE__->array($arg1, dtype=>$dtype);
+ if(defined $ctx and $dns->context ne $ctx)
+ {
+ $dns = $dns->as_in_context($ctx);
+ }
+ __PACKAGE__->_check_shape($dns->shape, $shape);
+ return $dns->tostype('csr');
+ }
+ }
+}
+
+# Create a AI::MXNet::NDarray::CSR based on data, indices and indptr
+method _csr_matrix_from_definition(
+ $data, $indices, $indptr,
+ Maybe[Shape|PDL] :$shape=,
+ AI::MXNet::Context :$ctx=AI::MXNet::Context->current_ctx,
+ Maybe[Dtype] :$dtype=,
+ Maybe[Dtype] :$indices_type=STORAGE_AUX_TYPES->{'csr'}[0],
+ Maybe[Dtype] :$indptr_type=STORAGE_AUX_TYPES->{'csr'}[1]
+)
+{
+ $dtype = __PACKAGE__->_prepare_default_dtype($data, $dtype);
+ # prepare src array and types
+ $data = __PACKAGE__->_prepare_src_array($data, $dtype);
+ $indptr = __PACKAGE__->_prepare_src_array($indptr, $indptr_type);
+ $indices = __PACKAGE__->_prepare_src_array($indices, $indices_type);
+
+ if(not (blessed $data and $data->isa('AI::MXNet::NDArray')))
+ {
+ $data = __PACKAGE__->array($data, ctx => $ctx, dtype => $dtype);
+ }
+ if(not (blessed $indptr and $indptr->isa('AI::MXNet::NDArray')))
+ {
+ $indptr = __PACKAGE__->array($indptr, ctx => $ctx, dtype => $indptr_type);
+ }
+ if(not (blessed $indices and $indices->isa('AI::MXNet::NDArray')))
+ {
+ $indices = __PACKAGE__->array($indices, ctx => $ctx, dtype => $indices_type);
+ }
+ if(not defined $shape)
+ {
+ if($indices->shape->[0] == 0)
+ {
+ confess('invalid shape');
+ }
+ $shape = [@{ $indptr } - 1, $indices->max->asscalar + 1];
+ }
+ elsif(blessed $shape)
+ {
+ $shape = $shape->unpdl;
+ }
+ # verify shapes
+ my $aux_shapes = [$indptr->shape, $indices->shape];
+ if($data->ndim != 1 or $indptr->ndim != 1 or $indices->ndim != 1 or $indptr->shape->[0] == 0 or @{ $shape } != 2)
+ {
+ confess('invalid shape');
+ }
+ my $hdl = __PACKAGE__->_new_alloc_handle(
+ 'csr', $shape, $ctx, 0, $dtype,
+ [$indptr_type, $indices_type], $aux_shapes
+ );
+ my $result = AI::MXNet::NDArray::CSR->new(handle => $hdl);
+ check_call(AI::MXNetCAPI::NDArraySyncCopyFromNDArray($result->handle, $data->handle, -1));
+ check_call(AI::MXNetCAPI::NDArraySyncCopyFromNDArray($result->handle, $indptr->handle, 0));
+ check_call(AI::MXNetCAPI::NDArraySyncCopyFromNDArray($result->handle, $indices->handle, 1));
+ return $result;
+}
+
+=head2 row_sparse_array
+
+ Creates a AI::MXNet::NDArray::RowSparse, a multidimensional row sparse array with a set of
+ tensor slices at given indices.
+
+ The AI::MXNet::NDArray::RowSparse can be instantiated in several ways:
+
+ - row_sparse_array($arg1, Maybe[AI::MXNet::Context] :$ctx=, Maybe[Shape] :$shape, Maybe [Dtype] :$dtype=)
+ $ctx, $shape, $dtype are optional
+ $arg1 can be given in following variants
+
+ - to construct a AI::MXNet::NDArray::RowSparse with a dense array $arg1
+ - $arg1 is in AI::MXNet::NDArray::array input format
+
+ - to construct a AI::MXNet::NDArray::RowSparse with a sparse array $arg1
+ $arg1 is AI::MXNet::NDArray::RowSparse
+
+ - to construct an empty AI::MXNet::NDArray::RowSparse with shape $arg1 = [$D1, $D1, ...$DN]
+
+ - to construct a RowSparseNDArray based on the definition of row sparse format
+ using two separate arrays,
+ where the $indices stores the indices of the row slices with non-zeros,
+ while the values are stored in $data. The corresponding NDArray dense
+ represented by RowSparse rsp has
+ dense[rsp.indices[i], :, :, :, ...] = rsp.data[i, :, :, :, ...]
+ The row indices for are expected to be **sorted in ascending order.
+ $arg1 = [$data, $indices]
+ $data, $indices must be given in the AI::MXNet::NDArray::array input format
+
+ Returns
+ -------
+ AI::MXNet::NDArray::RowSparse
+ A AI::MXNet::NDArray::RowSparse with the 'row_sparse' storage representation.
+
+ Example
+ -------
+ >>> $a = mx->nd->sparse->row_sparse_array([[[1, 2], [3, 4]], [1, 4]], shape=>[6, 2])
+ >>> $a->aspdl
+ [[ 0 0]
+ [ 1 2]
+ [ 0 0]
+ [ 0 0]
+ [ 3 4]
+ [ 0 0]]
+=cut
+
+method row_sparse_array(
+ $arg1,
+ Maybe[Shape] :$shape=,
+ Maybe[AI::MXNet::Context] :$ctx=AI::MXNet::Context->current_ctx,
+ Maybe[Dtype] :$dtype=
+)
+{
+ if(not defined $arg1)
+ {
+ return __PACKAGE__->empty('row_sparse', $shape, ctx => $ctx, (defined $dtype ? (dtype => $dtype) : ()));
+ }
+ # construct a row sparse array from (D0, D1 ..) or (data, indices)
+ if(ref $arg1 eq 'ARRAY')
+ {
+ my $arg_len = @{ $arg1 };
+ if($arg_len < 2)
+ {
+ confess("Unexpected length of input array: $arg_len ");
+ }
+ elsif($arg_len > 2)
+ {
+ # empty ndarray with shape
+ __PACKAGE__->_check_shape($arg1, $shape);
+ return __PACKAGE__->empty('row_sparse', $arg1, ctx => $ctx, dtype => $dtype);
+ }
+ else
+ {
+ # len(arg1) = 2, is either shape or (data, indices)
+ if(not ref $arg1->[0] and not ref $arg1->[1])
+ {
+ # empty ndarray with shape
+ __PACKAGE__->_check_shape($arg1, $shape);
+ return __PACKAGE__->empty('row_sparse', $arg1, ctx => $ctx, dtype => $dtype);
+ }
+ else
+ {
+ # data, indices, indptr
+ return __PACKAGE__->_row_sparse_ndarray_from_definition(
+ @{ $arg1 }, shape => $shape, ctx => $ctx, dtype => $dtype
+ );
+ }
+ }
+ }
+ else
+ {
+ # construct a row sparse ndarray from a dense / sparse array
+ if(blessed $arg1 and $arg1->isa('AI::MXNet::NDArray::RowSparse'))
+ {
+ # construct a row sparse ndarray from RowSparseNDArray
+ __PACKAGE__->_check_shape($arg1->shape, $shape);
+ return __PACKAGE__->array($arg1, ctx => $ctx, dtype => $dtype);
+ }
+ elsif(blessed $arg1 and $arg1->isa('AI::MXNet::NDArray::CSR'))
+ {
+ confess("Unexpected input type: AI::MXNet::NDArray::CSR");
+ }
+ else
+ {
+ # construct a csr matrix from a dense one
+ # prepare default dtype since mx.nd.array doesn't use default values
+ # based on source_array
+ $dtype = __PACKAGE__->_prepare_default_dtype($arg1, $dtype);
+ # create dns array with provided dtype. ctx is not passed since copy across
+ # ctx requires dtype to be the same
+ my $dns = __PACKAGE__->array($arg1, dtype => $dtype);
+ if(defined $ctx and $dns->context ne $ctx)
+ {
+ $dns = $dns->as_in_context($ctx);
+ }
+ __PACKAGE__->_check_shape($dns->shape, $shape);
+ return $dns->tostype('row_sparse');
+ }
+ }
+}
+
+# Create a AI::MXNet::NDArray::RowSparse based on data and indices
+method _row_sparse_ndarray_from_definition(
+ $data, $indices,
+ Maybe[Shape] :$shape=,
+ AI::MXNet::Context :$ctx=AI::MXNet::Context->current_ctx,
+ Maybe[Dtype] :$dtype=,
+ Maybe[Dtype] :$indices_type=STORAGE_AUX_TYPES->{'row_sparse'}[0]
+)
+{
+ $dtype = __PACKAGE__->_prepare_default_dtype($data, $dtype);
+ # prepare src array and types
+ $data = __PACKAGE__->_prepare_src_array($data, $dtype);
+ $indices = __PACKAGE__->_prepare_src_array($indices, $indices_type);
+
+ if(not (blessed $data and $data->isa('AI::MXNet::NDArray')))
+ {
+ $data = __PACKAGE__->array($data, ctx => $ctx, dtype => $dtype);
+ }
+ if(not (blessed $indices and $indices->isa('AI::MXNet::NDArray')))
+ {
+ $indices = __PACKAGE__->array($indices, ctx => $ctx, dtype => $indices_type);
+ }
+ if(not defined $shape)
+ {
+ my $num_indices = $indices->shape->[0];
+ if($num_indices == 0)
+ {
+ confess('invalid shape');
+ }
+ my $dim0 = $indices->at($num_indices - 1)->asscalar + 1;
+ $shape = [$dim0, @{ $data->shape } [1..@{ $data->shape } - 1]];
+ }
+ # verify shapes
+ if($data->ndim != @{ $shape } or $indices->ndim != 1 or product(@{ $shape } [1..@{ $shape } - 1]) == 0)
+ {
+ confess("invalid shape");
+ }
+ my $handle = __PACKAGE__->_new_alloc_handle(
+ 'row_sparse', $shape, $ctx, 0, $dtype,
+ [$indices_type], [$indices->shape]
+ );
+ my $result = AI::MXNet::NDArray::RowSparse->new(handle => $handle);
+ check_call(AI::MXNetCAPI::NDArraySyncCopyFromNDArray($result->handle, $data->handle, -1));
+ check_call(AI::MXNetCAPI::NDArraySyncCopyFromNDArray($result->handle, $indices->handle, 0));
+ return $result
+}
+
+=head2 zeros
+
+ Return a new array of given shape and type, filled with zeros.
+
+ Parameters
+ ----------
+ $stype: string
+ The storage type of the empty array, such as 'row_sparse', 'csr', etc
+ shape : int or array ref of int
+ The shape of the empty array
+ :$ctx : AI::MXNet::Context, optional
+ An optional device context (default is the current default context)
+ :$dtype : Dtype, optional
+ An optional value type (default is `float32`)
+
+ Returns
+ -------
+ AI::MXNet::NDArray::RowSparse or AI::MXNet::NDArray::CSR
+ A created array
+ Examples
+ --------
+ >>> mx->nd->sparse->zeros('csr', [1,2])
+ <AI::MXNet::NDArray::CSR 1x2 @cpu(0)>
+ >>> mx->nd->sparse->zeros('row_sparse', [1,2], ctx=>mx->cpu(), dtype=>'float16')->aspdl
+ [[ 0 0]]
+=cut
+
+
+method zeros(
+ Stype $stype,
+ Shape $shape,
+ AI::MXNet::Context :$ctx=AI::MXNet::Context->current_ctx,
+ Maybe[Dtype] :$dtype='float32',
+ Maybe[AI::MXNet::NDArray] :$out=,
+ Maybe[Str] :$name=,
+ Maybe[Str] :$__layout__=
+)
+{
+ if($stype eq 'default')
+ {
+ return AI::MXNet::NDArray->zeros(
+ $shape, ctx => $ctx, dtype => $dtype, out => $out, name => $name, __layout__ => $__layout__
+ );
+ }
+ my $aux_types;
+ if($stype eq 'row_sparse' or $stype or 'csr')
+ {
+ $aux_types = STORAGE_AUX_TYPES->{ $stype };
+ }
+ else
+ {
+ confess("unknown storage type: $stype");
+ }
+ $out //= __PACKAGE__->_ndarray_cls(
+ __PACKAGE__->_new_alloc_handle(
+ $stype, $shape, $ctx, 1, $dtype, $aux_types)
+ );
+ return __PACKAGE__->_zeros(
+ shape => $shape, ctx => $ctx, dtype => $dtype, out => $out,
+ ($__layout__ ? (__layout__ => $__layout__) : ())
+ );
+}
+
+=head2 empty
+
+ Returns a new array of given shape and type, without initializing entries.
+
+ Parameters
+ ----------
+ stype: string
+ The storage type of the empty array, such as 'row_sparse', 'csr', etc
+ shape : int or array ref of int
+ The shape of the empty array.
+ ctx : Context, optional
+ An optional device context (default is the current default context).
+ dtype : Dtype, optional
+ An optional value type (default is `float32`).
+
+ Returns
+ -------
+ AI::MXNet::NDArray::CSR or AI::MXNet::NDArray::RowSparse
+ A created array.
+=cut
+
+method empty(
+ Stype $stype,
+ Shape $shape,
+ Maybe[AI::MXNet::Context] :$ctx=AI::MXNet::Context->current_ctx,
+ Maybe[Dtype] :$dtype='float32'
+)
+{
+ assert(defined $stype);
+ return __PACKAGE__->zeros($stype, $shape, ctx => $ctx, dtype => $dtype);
+}
+
+=head2 array
+
+ Creates a sparse array from any object exposing the array interface.
+
+ Parameters
+ ----------
+ $source_array : AI::MXNet::NDArray::RowSparse, AI::MXNet::NDArray::CSR or PDL::CCS::Nd
+ The source sparse array
+ :$ctx : Context, optional
+ The default context is $source_array->context if $source_array is an NDArray.
+ The current default context otherwise.
+ :$dtype : Dtype, optional
+ The data type of the output array. The default dtype is $source_array->dtype
+ if $source_array is an AI::MXNet::NDArray, PDL or PDL::CCS::Nd
+ 'float32' otherwise.
+
+ Returns
+ -------
+ AI::MXNet::NDArray::RowSparse or AI::MXNet::NDArray::CSR
+ An array with the same contents as the $source_array.
+
+ Examples
+ --------
+ >>> use PDL; use PDL::CCS::Nd
+ >>> $csr = zeros([100, 2])->tocsr
+ >>> mx->nd->sparse->array($csr)
+ <AI::MXNet::NDArray::CSR 2x100 @cpu(0)>
+ >>> mx->nd->sparse->array(mx->nd->sparse->zeros('csr', [3, 2]))
+ <AI::MXNet::NDArray::CSR 3x2 @cpu(0)>
+ >>> mx->nd->sparse->array(mx->nd->sparse->zeros('row_sparse', [3, 2]))
+ <AI::MXNet::NDArray::RowSparse 3x2 @cpu(0)>
+=cut
+
+method array(
+ AcceptableInput $source_array,
+ Maybe[AI::MXNet::Context] :$ctx=AI::MXNet::Context->current_ctx,
+ Maybe[Dtype] :$dtype='float32',
+ Maybe[PDL] :$pdl=
+)
+{
+ if(not blessed $source_array or $source_array->isa('PDL') or ($source_array->isa('AI::MXNet::NDArray') and $source_array->stype eq 'default'))
+ {
+ if(not ref $source_array)
+ {
+ $pdl .= $source_array;
+ $source_array = $pdl;
+ }
+ return __PACKAGE__->SUPER::array($source_array, ctx => $ctx, dtype => $dtype);
+ }
+ if($source_array->isa('AI::MXNet::NDArray'))
+ {
+ assert(
+ ($source_array->stype ne 'default'),
+ "Please use tostype to create AI::MXNet::NDarray::RowSparse or AI::MXNet::NDarray::CSR from an AI::MXNet::NDarray"
+ );
+ # prepare dtype and ctx based on source_array, if not provided
+ $dtype = __PACKAGE__->_prepare_default_dtype($source_array, $dtype);
+ # if both dtype and ctx are different from source_array, we cannot copy directly
+ my $arr;
+ if($source_array->dtype ne $dtype and $source_array->context ne $ctx)
+ {
+ $arr = __PACKAGE__->empty($source_array->stype, $source_array->shape, dtype => $dtype);
+ $arr .= $source_array;
+ $arr = $arr->as_in_context($ctx);
+ }
+ else
+ {
+ $arr = __PACKAGE__->empty($source_array->stype, $source_array->shape, dtype => $dtype, ctx => $ctx);
+ $arr .= $source_array;
+ }
+ return $arr;
+ }
+ elsif($source_array->isa('PDL::CCS::Nd'))
+ {
+ $dtype = __PACKAGE__->_prepare_default_dtype($source_array, $dtype);
+ return __PACKAGE__->csr_matrix(
+ [$source_array->data, $source_array->indices, $source_array->indptr],
+ shape => $source_array->shape , dtype => $dtype, ctx => $ctx
+ );
+ }
+}
+
+sub AUTOLOAD {
+ my $sub = $AI::MXNet::NDArray::Sparse::AUTOLOAD;
+ $sub =~ s/.*:://;
+ $sub = "_sparse_$sub";
+ shift;
+ return AI::MXNet::NDArray->$sub(@_);
+}
+
+1;
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Optimizer.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Optimizer.pm
index 2894b1e..b8ad043 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Optimizer.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Optimizer.pm
@@ -44,7 +44,7 @@ method register()
{
my $name = $self;
($name) = $name =~ /::(\w+)$/;
- { no strict 'refs'; *{__PACKAGE__."::$name"} = sub { $self }; }
+ { no strict 'refs'; *{__PACKAGE__."::$name"} = sub { shift; $self->new(@_) }; }
$name = lc $name;
if(exists $opt_registry{ $name })
{
@@ -267,6 +267,27 @@ method _get_wd(Index $index)
A very simple SGD optimizer with momentum and weight regularization.
+ If the storage types of weight and grad are both 'row_sparse', and 'lazy_update' is True,
+ **lazy updates** are applied by
+
+ for row in grad.indices:
+ rescaled_grad[row] = lr * rescale_grad * clip(grad[row], clip_gradient) + wd * weight[row]
+ state[row] = momentum[row] * state[row] + rescaled_grad[row]
+ weight[row] = weight[row] - state[row]
+
+ The sparse update only updates the momentum for the weights whose row_sparse
+ gradient indices appear in the current batch, rather than updating it for all
+ indices. Compared with the original update, it can provide large
+ improvements in model training throughput for some applications. However, it
+ provides slightly different semantics than the original update, and
+ may lead to different empirical results.
+
+ Otherwise, **standard updates** are applied by::
+
+ rescaled_grad = lr * rescale_grad * clip(grad, clip_gradient) + wd * weight
+ state = momentum * state + rescaled_grad
+ weight = weight - state
+
Parameters
----------
learning_rate : float, optional
@@ -293,15 +314,18 @@ method _get_wd(Index $index)
True makes internal 32-bit copy of the weights and applies gradients
in 32-bit precision even if actual weights used in the model have lower precision.
Turning this on can improve convergence and accuracy when training with float16.
+
+ lazy_update: Bool, optional, default true
=cut
package AI::MXNet::SGD;
use Mouse;
extends 'AI::MXNet::Optimizer';
-has 'kwargs' => (is => "rw", isa => "HashRef[Num]");
-has 'momentum' => (is => "rw", isa => "Num", default => 0);
+has 'kwargs' => (is => "rw", isa => "HashRef[Num]");
+has 'momentum' => (is => "rw", isa => "Num", default => 0);
has 'multi_precision' => (is => "ro", isa => "Bool", default => 0);
+has 'lazy_update' => (is => "ro", isa => "Bool", default => 1);
sub BUILD
{
@@ -321,12 +345,13 @@ method create_state(Index $index, AI::MXNet::NDArray $weight)
{
my $momentum;
my $weight_master_copy;
+ my $stype = $self->lazy_update ? $weight->stype : 'default';
if($self->multi_precision and $weight->dtype eq 'float16')
{
my $weight_master_copy = AI::MXNet::NDArray->array($weight, ctx => $weight->context, dtype => 'float32');
if($self->momentum != 0)
{
- $momentum = AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype => 'float32');
+ $momentum = AI::MXNet::NDArray->zeros($weight->shape, stype => $stype, ctx => $weight->context, dtype => 'float32');
}
return [$momentum, $weight_master_copy];
}
@@ -341,7 +366,7 @@ method create_state(Index $index, AI::MXNet::NDArray $weight)
}
if($self->momentum != 0)
{
- $momentum = AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype => $weight->dtype);
+ $momentum = AI::MXNet::NDArray->zeros($weight->shape, stype => $stype, ctx => $weight->context, dtype => $weight->dtype);
}
return $momentum;
}
@@ -353,9 +378,9 @@ method update(
Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
)
{
+ $self->_update_count($index);
my $lr = $self->_get_lr($index);
my $wd = $self->_get_wd($index);
- $self->_update_count($index);
my $kwargs = {
out => $weight,
lr => $lr,
@@ -398,6 +423,460 @@ method update(
__PACKAGE__->register;
+=head1 NAME
+
+ AI::MXNet::Signum - The Signum optimizer that takes the sign of gradient or momentum.
+=cut
+
+=head1 DESCRIPTION
+
+ The optimizer updates the weight by:
+
+ rescaled_grad = rescale_grad * clip(grad, clip_gradient) + wd * weight
+ state = momentum * state + (1-momentum)*rescaled_grad
+ weight = (1 - lr * wd_lh) * weight - lr * sign(state)
+
+ See the original paper at: https://jeremybernste.in/projects/amazon/signum.pdf
+
+ For details of the update algorithm see
+ :class:`~mxnet.ndarray.signsgd_update` and :class:`~mxnet.ndarray.signum_update`.
+
+ This optimizer accepts the following parameters in addition to those accepted
+ by :class:`.Optimizer`.
+
+ Parameters
+ ----------
+ momentum : float, optional
+ The momentum value.
+ wd_lh : float, optional
+ The amount of decoupled weight decay regularization, see details in the original paper at:\
+ https://arxiv.org/abs/1711.05101
+=cut
+
+package AI::MXNet::Signum;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'momentum' => (is => "rw", isa => "Num", default => 0.9);
+has 'wd_lh' => (is => "rw", isa => "Num", default => 0);
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+
+ my $momentum;
+ if($self->momentum != 0)
+ {
+ $momentum = AI::MXNet::NDArray->zeros(
+ $weight->shape,
+ ctx => $weight->context,
+ dtype=>$weight->dtype,
+ stype=>$weight->stype
+ );
+ }
+ return $momentum;
+}
+
+method update(
+ Index $index,
+ AI::MXNet::NDArray $weight,
+ AI::MXNet::NDArray $grad,
+ Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+ $self->_update_count($index);
+ my $lr = $self->_get_lr($index);
+ my $wd = $self->_get_wd($index);
+ my %kwargs = (
+ out => $weight,
+ lr => $lr,
+ wd => $wd,
+ rescale_grad => $self->rescale_grad,
+ );
+ if($self->momentum > 0)
+ {
+ $kwargs{momentum} = $self->momentum;
+ }
+ if($self->clip_gradient)
+ {
+ $kwargs{clip_gradient} = $self->clip_gradient;
+ }
+ if($self->wd_lh)
+ {
+ $kwargs{wd_lh} = $self->wd_lh;
+ }
+ if(defined $state)
+ {
+ AI::MXNet::NDArray->signum_update(
+ $weight, $grad, $state, %kwargs
+ );
+ }
+ else
+ {
+ AI::MXNet::NDArray->signsgd_update(
+ $weight, $grad, %kwargs
+ );
+ }
+}
+
+__PACKAGE__->register;
+
+=head1 NAME
+
+ AI::MXNet::FTML - The FTML optimizer.
+=cut
+
+=head1 DESCRIPTION
+
+ This class implements the optimizer described in
+ *FTML - Follow the Moving Leader in Deep Learning*,
+ available at http://proceedings.mlr.press/v70/zheng17a/zheng17a.pdf.
+
+ This optimizer accepts the following parameters in addition to those accepted
+ by AI::MXNet::Optimizer
+
+ Parameters
+ ----------
+ beta1 : float, optional
+ 0 < beta1 < 1. Generally close to 0.5.
+ beta2 : float, optional
+ 0 < beta2 < 1. Generally close to 1.
+ epsilon : float, optional
+ Small value to avoid division by 0.
+=cut
+
+package AI::MXNet::FTML;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'beta1' => (is => "rw", isa => "Num", default => 0.6);
+has 'beta2' => (is => "rw", isa => "Num", default => 0.999);
+has 'epsilon' => (is => "rw", isa => "Num", default => 1e-8);
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+ return [
+ AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # d_0
+ AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # v_0
+ AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # z_0
+ ];
+}
+
+method update(
+ Index $index,
+ AI::MXNet::NDArray $weight,
+ AI::MXNet::NDArray $grad,
+ Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+ my $lr = $self->_get_lr($index);
+ my $wd = $self->_get_wd($index);
+ my $t = $self->_update_count($index);
+ my %kwargs = (
+ out => $weight,
+ lr => $lr,
+ wd => $wd,
+ t => $t,
+ beta1 => $self->beta1,
+ beta2 => $self->beta2,
+ epsilon => $self->epsilon,
+ rescale_grad => $self->rescale_grad
+ );
+ if($self->clip_gradient)
+ {
+ $kwargs{clip_grad} = $self->clip_gradient;
+ }
+ AI::MXNet::NDArray->ftml_update($weight, $grad, @{ $state }, \%kwargs);
+}
+
+__PACKAGE__->register;
+
+=head1 NAME
+
+ AI::MXNet::LBSGD - The Large Batch SGD optimizer with momentum and weight decay.
+=cut
+
+=head1 DESCRIPTION
+
+ The optimizer updates the weight by::
+
+ state = momentum * state + lr * rescale_grad * clip(grad, clip_gradient) + wd * weight
+ weight = weight - state
+
+ Parameters
+ ----------
+ momentum : float, optional
+ The momentum value.
+ multi_precision: bool, optional
+ Flag to control the internal precision of the optimizer.
+ ``False`` results in using the same precision as the weights (default),
+ ``True`` makes internal 32-bit copy of the weights and applies gradients
+ in 32-bit precision even if actual weights used in the model have lower precision.`<
+ Turning this on can improve convergence and accuracy when training with float16.
+ warmup_strategy: string ('linear', 'power2', 'sqrt'. , 'lars' default : 'linear')
+ warmup_epochs: unsigned, default: 5
+ batch_scale: unsigned, default: 1 (same as batch size*numworkers)
+ updates_per_epoch: updates_per_epoch (default: 32, Default might not reflect true number batches per epoch. Used for warmup.)
+ begin_epoch: unsigned, default 0, starting epoch.
+=cut
+
+package AI::MXNet::LBSGD;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'momentum' => (is => 'rw', isa => 'Num', default => 0);
+has 'multi_precision' => (is => 'rw', isa => 'Bool', default => 0);
+has 'warmup_startegy' => (is => 'rw', isa => 'Str', default => 'linear');
+has 'warmup_epochs' => (is => 'rw', isa => 'Int', default => 5);
+has 'batch_scale' => (is => 'rw', isa => 'Num', default => 1);
+has 'updates_per_epoch' => (is => 'rw', isa => 'Int', default => 32);
+has 'begin_epoch' => (is => 'rw', isa => 'Int', default => 0);
+has 'num_epochs' => (is => 'rw', isa => 'Int', default => 60);
+has 'beta2' => (is => 'rw', isa => 'Num', default => 0.999);
+has 'epsilon' => (is => 'rw', isa => 'Num', default => 1e-8);
+has 'init_updates' => (is => 'rw', init_arg => undef);
+has [qw/lbmult
+ cumgrads
+ adaptive
+ init_updates
+ admult/] => (is => 'rw', init_arg => undef);
+
+sub BUILD
+{
+ my $self = shift;
+ AI::MXNet::Logging->info('Running Large-Batch SGD Algorithm');
+ AI::MXNet::Logging->info(
+ '(Batch_scale=%f, warmup_epochs=%d, warmup_strategy=%s, updates_per_epoch=%d)',
+ map { $self->$_ } qw/batch_scale warmup_epochs warmup_strategy updates_per_epoch/
+ );
+ $self->init_updates($self->begin_epoch * $self->updates_per_epoch);
+ $self->lbmult(1);
+ $self->cumgrads({});
+ $self->adaptive(0);
+ $self->admult(1);
+}
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+ return [
+ AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # d_0
+ AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # v_0
+ AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # z_0
+ ];
+ my $momentum;
+ my $weight_master_copy;
+ if($self->multi_precision and $weight->dtype eq 'float16')
+ {
+ $weight_master_copy = AI::MXNet::NDArray->array($weight, ctx=>$weight->context, dtype=>'float32');
+ if($self->momentum != 0)
+ {
+ $momentum = AI::MXNet::NDArray->zeros(
+ $weight->shape, ctx => $weight->context, dtype => 'float32',
+ stype => $weight->stype
+ );
+ }
+ return [$momentum, $weight_master_copy];
+ }
+ if($weight->dtype eq 'float16' and not $self->multi_precision)
+ {
+ AI::MXNet::Logging->warning(
+ "Accumulating with float16 in optimizer can lead to "
+ ."poor accuracy or slow convergence. "
+ ."Consider using multi_precision=True option of the "
+ ."LBSGD optimizer"
+ );
+ }
+ if($self->momentum != 0)
+ {
+ $momentum = AI::MXNet::NDArray->zeros(
+ $weight->shape, ctx => $weight->context, dtype => $weight->dtype,
+ stype => $weight->stype
+ );
+ }
+ return $momentum;
+}
+
+method _get_lbmult($nup)
+{
+ my $nwup = $self->warmup_epochs * $self->updates_per_epoch;
+ my $strategy = $self->warmup_strategy;
+ my $maxmult = $self->batch_scale;
+ my $mult;
+ if($nup >= $nwup)
+ {
+ $mult = $maxmult;
+ }
+ elsif($nwup <= 1)
+ {
+ $mult = 1;
+ }
+ else
+ {
+ if ($strategy eq 'linear')
+ {
+ $mult = 1 + ($maxmult - 1) * $nup / $nwup;
+ }
+ elsif($strategy eq 'power2')
+ {
+ $mult = 1 + ($maxmult-1) * ($nup*$nup)/($nwup*$nwup);
+ }
+ elsif($strategy eq 'sqrt')
+ {
+ $mult = 1 + ($maxmult - 1) * sqrt($nup / $nwup);
+ }
+ else
+ {
+ $mult = 1;
+ }
+ }
+ return $mult;
+}
+
+
+method _get_lars($weight, $g, $wd)
+{
+ my $weight2 = $self->_l2norm($weight);
+ my $grad2 = $self->_l2norm($g);
+ my $lars = sqrt($weight2 / ($grad2 + $wd * $weight2 + 1e-18));
+ if($lars < 0.01)
+ {
+ $lars = 0.01;
+ }
+ elsif($lars > 100)
+ {
+ $lars = 100;
+ }
+ return $lars;
+}
+
+method _l2norm($v)
+{
+ my $norm = AI::MXNet::NDArray->multiply($v, $v)->aspdl->sum;
+ return $norm;
+}
+
+method _reset_cum_gradient($index)
+{
+ $self->cumgrads->{$index}{cum_grad} = 0;
+}
+
+method _get_cum_gradient($index)
+{
+ if(exists $self->cumgrads->{$index})
+ {
+ return $self->cumgrads->{$index};
+ }
+ else
+ {
+ return {}
+ }
+}
+
+method _put_cum_gradient($index, $cgrad)
+{
+ $self->cumgrads->{$index} = $cgrad;
+}
+
+method _cumulate_gradient($grad, $index)
+{
+ my $cgrad = $self->_get_cum_gradient($index);
+ my ($num_cums, $cum_grad);
+ if(%{ $cgrad })
+ {
+ my $num_cums = $cgrad->{num_cums};
+ if($num_cums > 0)
+ {
+ $cum_grad = $cgrad->{cum_grad} + $grad;
+ $num_cums += 1;
+ }
+ else
+ {
+ $cum_grad = $grad;
+ $num_cums = $self->init_updates + 1;
+ }
+ }
+ else
+ {
+ $cum_grad = $grad;
+ $num_cums = $self->init_updates + 1;
+ }
+ $cgrad = {cum_grad => $cum_grad, num_cums => $num_cums};
+ $self->_put_cum_gradient($index, $cgrad);
+ return $cgrad;
+}
+
+
+
+method update(
+ Index $index,
+ AI::MXNet::NDArray $weight,
+ AI::MXNet::NDArray $grad,
+ Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+ my $lr = $self->_get_lr($index);
+ my $wd = $self->_get_wd($index);
+ my $t = $self->_update_count($index);
+ my $cgrad = $self->_cumulate_gradient($grad, $index);
+ if(($cgrad->{num_cums} % $self->batch_scale) == 0)
+ {
+ my $lbmult;
+ $grad = $cgrad->{cum_grad} / $self->batch_scale;
+ if($self->warmup_strategy eq 'lars')
+ {
+ $lbmult = $self->_get_lars($weight, $grad, $wd);
+ }
+ else
+ {
+ $lbmult = $self->_get_lbmult($cgrad->{num_cums});
+ }
+ $lr = $lr * $lbmult;
+ my %kwargs = (
+ out => $weight,
+ lr => $lr,
+ wd => $wd,
+ rescale_grad => $self->rescale_grad
+ );
+ if($self->clip_gradient)
+ {
+ $kwargs{clip_gradient} = $self->clip_gradient;
+ }
+ if($self->momentum > 0)
+ {
+ $kwargs{momentum} = $self->momentum;
+ }
+ my $use_multi_precision = ref($state) eq 'ARRAY';
+ if(not $use_multi_precision)
+ {
+ if(defined $state)
+ {
+ AI::MXNet::NDArray->sgd_mom_update($weight, $grad, $state, %kwargs);
+ }
+ else
+ {
+ AI::MXNet::NDArray->sgd_update($weight, $grad, %kwargs);
+ }
+ }
+ else
+ {
+ if(defined $state->[0])
+ {
+ AI::MXNet::NDArray->mp_sgd_mom_update($weight, $grad, @{ $state }, %kwargs);
+ }
+ else
+ {
+ AI::MXNet::NDArray->mp_sgd_update($weight, $grad, $state->[1], %kwargs);
+ }
+ }
+ $self->_reset_cum_gradient($index);
+ }
+ else
+ {
+ AI::MXNet::NDArray->sgd_update($weight, $grad, out => $weight, lr => 0, wd => $wd);
+ }
+}
+
+__PACKAGE__->register;
+
package AI::MXNet::DCASGD;
use Mouse;
use AI::MXNet::Base;
@@ -515,41 +994,83 @@ __PACKAGE__->register;
package AI::MXNet::NAG;
use Mouse;
-
extends 'AI::MXNet::SGD';
-method update(
- Index $index,
- AI::MXNet::NDArray $weight,
- AI::MXNet::NDArray $grad,
- AI::MXNet::NDArray|Undef $state
-)
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+ my $momentum;
+ my $weight_master_copy;
+ my $do_multi_precision = ($self->multi_precision and $weight->dtype eq 'float16');
+ if($do_multi_precision)
+ {
+ if($self->momentum != 0)
+ {
+ $momentum = AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype=>'float32');
+ }
+ $weight_master_copy = AI::MXNet::NDArray->array($weight, ctx=>$weight->context, dtype=>'float32');
+ return [$weight_master_copy, $momentum];
+ }
+ else
+ {
+ if($self->momentum != 0)
+ {
+ $momentum = AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype);
+ }
+ return $momentum;
+ }
+}
+
+method update($index, $weight, $grad, $state)
{
my $lr = $self->_get_lr($index);
my $wd = $self->_get_wd($index);
$self->_update_count($index);
- $grad = $grad * $self->rescale_grad;
- if($self->clip_gradient)
- {
- $grad = AI::MXNet::NDArray->clip(
- $grad,
- -$self->clip_gradient,
- $self->clip_gradient
- );
- }
- if($state)
+ my $use_multi_precision = (defined $state and not Scalar::Util::blessed($state) and ref($state eq 'ARRAY'));
+ if(not $use_multi_precision)
{
- my $mom = $state;
- $mom *= $self->momentum;
- $grad += $wd * $weight;
- $mom += $grad;
- $grad += $self->momentum * $mom;
- $weight += -$lr * $grad;
+ $grad *= $self->rescale_grad;
+ if(defined $self->clip_gradient)
+ {
+ $grad = AI::MXNet::NDArray->clip($grad, -$self->clip_gradient, $self->clip_gradient);
+ }
+ if($self->momentum == 0)
+ {
+ $weight += -$lr * ($grad + $wd * $weight);
+ }
+ else
+ {
+ my $mom = $state;
+ $mom *= $self->momentum;
+ $grad += $wd * $weight;
+ $mom += $grad;
+ $grad += $self->momentum * $mom;
+ $weight += -$lr * $grad;
+ }
}
else
{
- confess("momentum != 0") unless $self->momentum == 0;
- $weight += -$lr * ($grad + $wd * $weight);
+ my $grad32 = AI::MXNet::NDArray->array($grad, ctx=>$grad->context, dtype=>'float32');
+ $grad32 *= $self->rescale_grad;
+ if(defined $self->clip_gradient)
+ {
+ $grad32 = AI::MXNet::NDArray->clip($grad32, -$self->clip_gradient, $self->clip_gradient);
+ }
+ my $mom = $state->[1];
+ my $weight32 = $state->[0];
+ if($self->momentum == 0)
+ {
+ $weight32 += -$lr * ($grad32 + $wd * $weight32);
+ }
+ else
+ {
+ $mom *= $self->momentum;
+ $grad32 += $wd * $weight32;
+ $mom += $grad32;
+ $grad32 += $self->momentum * $mom;
+ $weight32 += -$lr * $grad32;
+ }
+ my $tmp = $weight32->astype($weight->dtype);
+ $tmp->copyto($weight);
}
}
@@ -557,12 +1078,16 @@ __PACKAGE__->register;
=head1 NAME
- AI::MXNet::SLGD - Stochastic Langevin Dynamics Updater to sample from a distribution.
+ AI::MXNet::SGLD - Stochastic Gradient Riemannian Langevin Dynamics.
=cut
=head1 DESCRIPTION
- Stochastic Langevin Dynamics Updater to sample from a distribution.
+ Stochastic Gradient Riemannian Langevin Dynamics.
+
+ This class implements the optimizer described in the paper *Stochastic Gradient
+ Riemannian Langevin Dynamics on the Probability Simplex*, available at
+ https://papers.nips.cc/paper/4883-stochastic-gradient-riemannian-langevin-dynamics-on-the-probability-simplex.pdf.
Parameters
----------
@@ -577,12 +1102,9 @@ __PACKAGE__->register;
clip_gradient : float, optional
clip gradient in range [-clip_gradient, clip_gradient]
-
- param_idx2name : dict of string/int to float, optional
- special treat weight decay in parameter ends with bias, gamma, and beta
=cut
-package AI::MXNet::SLGD;
+package AI::MXNet::SGLD;
use Mouse;
extends 'AI::MXNet::Optimizer';
@@ -615,8 +1137,9 @@ method update(
+
AI::MXNet::Random->normal(
0, sqrt($lr),
- $weight->shape,
- $weight->context
+ shape => $weight->shape,
+ ctx => $weight->context,
+ dtype => $weight->dtype
);
}
@@ -651,8 +1174,6 @@ __PACKAGE__->register;
Default value is set to 0.999.
epsilon : float, optional
Default value is set to 1e-8.
- decay_factor : float, optional
- Default value is set to 1 - 1e-8.
wd : float, optional
L2 regularization coefficient add to all the weights
@@ -672,7 +1193,7 @@ has '+learning_rate' => (default => 0.001);
has 'beta1' => (is => "rw", isa => "Num", default => 0.9);
has 'beta2' => (is => "rw", isa => "Num", default => 0.999);
has 'epsilon' => (is => "rw", isa => "Num", default => 1e-8);
-has 'decay_factor' => (is => "rw", isa => "Num", default => (1 - 1e-8));
+has 'lazy_update' => (is => 'rw', isa => 'Bool', default => 1);
sub BUILD
{
@@ -690,15 +1211,18 @@ sub BUILD
method create_state(Index $index, AI::MXNet::NDArray $weight)
{
+ my $stype = $self->lazy_update ? $weight->stype : 'default';
return [AI::MXNet::NDArray->zeros(
$weight->shape,
ctx => $weight->context,
- dtype => $weight->dtype
+ dtype => $weight->dtype,
+ stype => $stype
), # mean
AI::MXNet::NDArray->zeros(
$weight->shape,
ctx => $weight->context,
- dtype => $weight->dtype
+ dtype => $weight->dtype,
+ stype => $stype
) # variance
];
}
@@ -769,14 +1293,14 @@ use Mouse;
extends 'AI::MXNet::Optimizer';
-has 'float_stable_eps' => (is => "rw", isa => "Num", default => 1e-7);
-has '+learning_rate' => (default => 0.05);
+has 'eps' => (is => "rw", isa => "Num", default => 1e-7);
method create_state(Index $index, AI::MXNet::NDArray $weight)
{
return AI::MXNet::NDArray->zeros(
$weight->shape,
- ctx => $weight->context
+ ctx => $weight->context,
+ stype => $weight->stype
); # history
}
@@ -790,30 +1314,31 @@ method update(
my $lr = $self->_get_lr($index);
my $wd = $self->_get_wd($index);
$self->_update_count($index);
- $grad *= $self->rescale_grad;
- if($self->clip_gradient)
+ my $is_sparse = ($weight->stype eq 'row_sparse' and $grad->stype eq 'row_sparse') ? 1 : 0;
+ my $history = $state;
+ if($is_sparse)
{
- $grad = AI::MXNet::NDArray->clip(
- $grad,
- -$self->clip_gradient,
- $self->clip_gradient
+ my %kwargs = (
+ epsilon => $self->eps,
+ rescale_grad => $self->rescale_grad
);
+ if($self->clip_gradient)
+ {
+ $kwargs{clip_gradient} = $self->clip_gradient;
+ }
+ AI::MXNet::NDArray::Sparse->adagrad_update($weight, $grad, $history, { out=>$weight, lr=>$lr, wd=>$wd, %kwargs });
+ }
+ else
+ {
+ $grad *= $self->rescale_grad;
+ if(defined $self->clip_gradient)
+ {
+ $grad = AI::MXNet::NDArray->clip($grad, -$self->clip_gradient, $self->clip_gradient);
+ }
+ $history += $grad->square;
+ my $div = $grad / ($history + $self->eps)->sqrt;
+ $weight += ($div + $weight * $wd) * -$lr;
}
- my $history = $state;
- $history += ($grad * $grad);
- $weight += -$lr
- *
- (
- $grad
- /
- AI::MXNet::NDArray->sqrt(
- $history
- +
- $self->float_stable_eps
- )
- +
- $wd * $weight
- );
}
__PACKAGE__->register;
@@ -903,21 +1428,25 @@ method create_state(Index $index, AI::MXNet::NDArray $weight)
? (
AI::MXNet::NDArray->zeros(
$weight->shape,
- ctx => $weight->context
+ ctx => $weight->context,
+ stype => $weight->stype
), # n
AI::MXNet::NDArray->zeros(
$weight->shape,
- ctx => $weight->context
+ ctx => $weight->context,
+ stype => $weight->stype
), # g
AI::MXNet::NDArray->zeros(
$weight->shape,
- ctx => $weight->context
+ ctx => $weight->context,
+ stype => $weight->stype
)
) # delta
: (
AI::MXNet::NDArray->zeros(
$weight->shape,
- ctx => $weight->context
+ ctx => $weight->context,
+ stype => $weight->stype
), # n
)
];
@@ -1074,6 +1603,7 @@ __PACKAGE__->register;
package AI::MXNet::Ftrl;
+
=head1 NAME
AI::MXNet::Ftrl
@@ -1081,37 +1611,69 @@ package AI::MXNet::Ftrl;
=head1 DESCRIPTION
- Reference:Ad Click Prediction: a View from the Trenches
+ Referenced from *Ad Click Prediction: a View from the Trenches*, available at
+ http://dl.acm.org/citation.cfm?id=2488200.
+
+ eta :
+ .. math::
+ \\eta_{t,i} = \\frac{learningrate}{\\beta+\\sqrt{\\sum_{s=1}^tg_{s,i}^2}}
+
+ The optimizer updates the weight by::
+
+ rescaled_grad = clip(grad * rescale_grad, clip_gradient)
+ z += rescaled_grad - (sqrt(n + rescaled_grad**2) - sqrt(n)) * weight / learning_rate
+ n += rescaled_grad**2
+ w = (sign(z) * lamda1 - z) / ((beta + sqrt(n)) / learning_rate + wd) * (abs(z) > lamda1)
+
+ If the storage types of weight, state and grad are all ``row_sparse``, \
+ **sparse updates** are applied by::
+
+ for row in grad.indices:
+ rescaled_grad[row] = clip(grad[row] * rescale_grad, clip_gradient)
+ z[row] += rescaled_grad[row] - (sqrt(n[row] + rescaled_grad[row]**2) - sqrt(n[row])) * weight[row] / learning_rate
+ n[row] += rescaled_grad[row]**2
+ w[row] = (sign(z[row]) * lamda1 - z[row]) / ((beta + sqrt(n[row])) / learning_rate + wd) * (abs(z[row]) > lamda1)
+
+ The sparse update only updates the z and n for the weights whose row_sparse
+ gradient indices appear in the current batch, rather than updating it for all
+ indices. Compared with the original update, it can provide large
+ improvements in model training throughput for some applications. However, it
+ provides slightly different semantics than the original update, and
+ may lead to different empirical results.
+
+ For details of the update algorithm, see :class:`~mxnet.ndarray.ftrl_update`.
+
+ This optimizer accepts the following parameters in addition to those accepted
+ by :class:`.Optimizer`.
Parameters
----------
lamda1 : float, optional
L1 regularization coefficient.
-
learning_rate : float, optional
The initial learning rate.
-
beta : float, optional
Per-coordinate learning rate correlation parameter.
- eta_{t,i}=frac{learning_rate}{beta+sqrt{sum_{s=1^}tg_{s,i}^t}
=cut
use Mouse;
extends 'AI::MXNet::Optimizer';
has '+learning_rate' => (default => 0.1);
has 'beta' => (is => "ro", isa => "Num", default => 1);
-has 'lambda1' => (is => "ro", isa => "Num", default => 0.9);
+has 'lamda1' => (is => "ro", isa => "Num", default => 0.01);
method create_state(Index $index, AI::MXNet::NDArray $weight)
{
return [
AI::MXNet::NDArray->zeros(
$weight->shape,
- ctx => $weight->context
- ), # dn
+ ctx => $weight->context,
+ stype => $weight->stype
+ ), # z
AI::MXNet::NDArray->zeros(
$weight->shape,
- ctx => $weight->context
+ ctx => $weight->context,
+ stype => $weight->stype
) # n
];
}
@@ -1126,22 +1688,17 @@ method update(
$self->_update_count($index);
my $wd = $self->_get_wd($index);
my $lr = $self->_get_lr($index);
- $grad *= $self->rescale_grad;
+ my %kwargs = (lamda1 => $self->lamda1, beta => $self->beta, rescale_grad => $self->rescale_grad);
if($self->clip_gradient)
{
- $grad = AI::MXNet::NDArray->clip(
- $grad,
- -$self->clip_gradient,
- $self->clip_gradient
- );
+ $kwargs{clip_gradient} = $self->clip_gradient;
}
- my ($dn, $n) = @{ $state };
- $dn += $grad - (($n + $grad * $grad)->sqrt - $n->sqrt) * $weight / $lr;
- $n += $grad * $grad;
-
- $weight .= ($dn->sign * $self->lamda1 - $dn)
- /
- (($self->beta + $n->sqrt) / $lr + $wd) * ($dn->abs > $self->lamda1);
+ # accumulated g and delta initialization
+ my ($z, $n) = @{ $state };
+ AI::MXNet::NDArray->ftrl_update(
+ $weight, $grad, $z, $n,
+ { lr => $lr, wd => $wd, %kwargs, out => $weight }
+ );
}
__PACKAGE__->register;
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Symbol.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Symbol.pm
index 8fd885a..299aed7 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Symbol.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Symbol.pm
@@ -242,8 +242,14 @@ method call(@args)
return $s;
}
-method slice(Str|Index $index)
+method slice(@slices)
{
+ confess("No slices supplied") unless @slices;
+ if(@slices > 1)
+ {
+ return $self->SUPER::slice(@slices);
+ }
+ my $index = $slices[0];
## __getitem__ tie needs to die
if(not find_type_constraint('Index')->check($index))
{
@@ -778,6 +784,9 @@ method _get_ndarray_inputs(
:$type_dict : hash ref of str->Dtype
Input type map, name->dtype
+ :$type_dict : hash ref of str->Stype
+ Storage type map, name->stype (for sparse operations)
+
:$group2ctx : hash ref of string to AI::MXNet::Context
The mapping of the ctx_group attribute to the context assignment.
@@ -808,13 +817,14 @@ method simple_bind(
GradReq|ArrayRef[GradReq]|HashRef[GradReq] :$grad_req='write',
Maybe[HashRef[Shape]] :$shapes=,
Maybe[HashRef[Dtype]] :$type_dict=,
+ Maybe[HashRef[Stype]] :$stype_dict=,
Maybe[HashRef[AI::MXNet::Context]] :$group2ctx=,
Maybe[ArrayRef[Str]] :$shared_arg_names=,
Maybe[AI::MXNet::Executor] :$shared_exec=,
Maybe[HashRef[AI::MXNet::NDArray]] :$shared_buffer=
)
{
- my $num_provided_arg_types;
+ my $num_provided_arg_types = 0;
my @provided_arg_type_names;
my @provided_arg_type_data;
if(defined $type_dict)
@@ -826,6 +836,18 @@ method simple_bind(
}
$num_provided_arg_types = @provided_arg_type_names;
}
+ my $num_provided_arg_stypes = 0;
+ my @provided_arg_stype_names;
+ my @provided_arg_stype_data;
+ if(defined $stype_dict)
+ {
+ while(my ($k, $v) = each %{ $stype_dict })
+ {
+ push @provided_arg_stype_names, $k;
+ push @provided_arg_stype_data, STORAGE_TYPE_STR_TO_ID->{$v};
+ }
+ $num_provided_arg_stypes = @provided_arg_stype_names;
+ }
my @provided_arg_shape_data;
# argument shape index in sdata,
# e.g. [sdata[indptr[0]], sdata[indptr[1]]) is the shape of the first arg
@@ -923,6 +945,9 @@ method simple_bind(
$num_provided_arg_types,
\@provided_arg_type_names,
\@provided_arg_type_data,
+ $num_provided_arg_stypes,
+ \@provided_arg_stype_names,
+ \@provided_arg_stype_data,
scalar(@shared_arg_name_list),
\@shared_arg_name_list,
defined $shared_buffer ? \%shared_data : undef,
@@ -943,12 +968,12 @@ method simple_bind(
{
while(my ($k, $v) = each %{ $updated_shared_data })
{
- $shared_buffer->{$k} = AI::MXNet::NDArray->new(handle => $v);
+ $shared_buffer->{$k} = AI::MXNet::NDArray->_ndarray_cls($v);
}
}
- my @arg_arrays = map { AI::MXNet::NDArray->new(handle => $_) } @{ $in_arg_handles };
- my @grad_arrays = map { defined $_ ? AI::MXNet::NDArray->new(handle => $_) : undef } @{ $arg_grad_handles };
- my @aux_arrays = map { AI::MXNet::NDArray->new(handle => $_) } @{ $aux_state_handles };
+ my @arg_arrays = map { AI::MXNet::NDArray->_ndarray_cls($_) } @{ $in_arg_handles };
+ my @grad_arrays = map { defined $_ ? AI::MXNet::NDArray->_ndarray_cls($_) : undef } @{ $arg_grad_handles };
+ my @aux_arrays = map { AI::MXNet::NDArray->_ndarray_cls($_) } @{ $aux_state_handles };
my $executor = AI::MXNet::Executor->new(
handle => $exe_handle,
symbol => $self,
@@ -1237,6 +1262,7 @@ method Variable(
Maybe[Num] :$lr_mult=,
Maybe[Num] :$wd_mult=,
Maybe[Dtype] :$dtype=,
+ Maybe[Stype] :$stype=,
Maybe[Initializer] :$init=,
HashRef[Str] :$kwargs={},
Maybe[Str] :$__layout__=
@@ -1251,6 +1277,7 @@ method Variable(
$attr->{__dtype__} = DTYPE_STR_TO_MX->{ $dtype } if $dtype;
$attr->{__init__} = "$init" if defined $init;
$attr->{__layout__} = $__layout__ if defined $__layout__;
+ $attr->{__storage_type__} = STORAGE_TYPE_STR_TO_ID->{$stype} if defined $stype;
while(my ($k, $v) = each %{ $kwargs })
{
if($k =~ /^__/ and $k =~ /__$/)
@@ -1470,5 +1497,6 @@ sub _ufunc_helper
sub contrib { 'AI::MXNet::Contrib::Symbol' }
sub random { 'AI::MXNet::Symbol::Random' }
+sub sparse { 'AI::MXNet::Symbol::Sparse' }
1;
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Symbol/Base.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Symbol/Base.pm
index 8d3d069..2cb20b7 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Symbol/Base.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Symbol/Base.pm
@@ -155,7 +155,7 @@ func _make_atomic_symbol_function($handle, $name)
}
for my $key (keys %kwargs)
{
- $kwargs{ $key } = "(" .join(", ", @{ $kwargs{ $key } }) .")"
+ $kwargs{ $key } = "(" .join(", ", map { defined($_) ? $_ : 'None' } @{ $kwargs{ $key } }) .")"
if ref $kwargs{ $key } eq 'ARRAY';
}
while(my ($k, $v) = each %kwargs)
diff --git a/docker/install/perl.sh b/perl-package/AI-MXNet/lib/AI/MXNet/Symbol/Sparse.pm
old mode 100755
new mode 100644
similarity index 72%
copy from docker/install/perl.sh
copy to perl-package/AI-MXNet/lib/AI/MXNet/Symbol/Sparse.pm
index af49952..b81cba9
--- a/docker/install/perl.sh
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Symbol/Sparse.pm
@@ -1,5 +1,3 @@
-#!/usr/bin/env bash
-
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -17,6 +15,19 @@
# specific language governing permissions and limitations
# under the License.
-# install libraries for mxnet's perl package on ubuntu
-apt-get update && apt-get install -y libmouse-perl pdl cpanminus swig libgraphviz-perl
-cpanm -q Function::Parameters Hash::Ordered
+package AI::MXNet::Symbol::Sparse;
+use strict;
+use warnings;
+use Mouse;
+extends 'AI::MXNet::Symbol';
+
+sub AUTOLOAD {
+ my $sub = $AI::MXNet::Symbol::Sparse::AUTOLOAD;
+ $sub =~ s/.*:://;
+ shift;
+ my @args = @_;
+ $sub = "_sparse_$sub";
+ return AI::MXNet::Symbol->$sub(@args);
+}
+
+1;
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/TestUtils.pm b/perl-package/AI-MXNet/lib/AI/MXNet/TestUtils.pm
index af9b27d..37ef5b6 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/TestUtils.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/TestUtils.pm
@@ -19,14 +19,17 @@ package AI::MXNet::TestUtils;
use strict;
use warnings;
use PDL;
-use Carp;
+use Carp qw(confess);
use Scalar::Util qw(blessed);
+use List::Util qw(shuffle);
use AI::MXNet::Function::Parameters;
+use AI::MXNet::Base;
use Exporter;
use base qw(Exporter);
@AI::MXNet::TestUtils::EXPORT_OK = qw(same reldiff almost_equal GetMNIST_ubyte
- GetCifar10 pdl_maximum pdl_minimum mlp2 conv
- check_consistency zip assert enumerate same_array dies_like allclose);
+ GetCifar10 pdl_maximum pdl_minimum mlp2 conv dies_ok
+ check_consistency zip assert enumerate same_array dies_like allclose rand_shape_2d
+ rand_shape_3d rand_sparse_ndarray random_arrays rand_ndarray randint);
use constant default_numerical_threshold => 1e-6;
=head1 NAME
@@ -57,9 +60,9 @@ func same(PDL $a, PDL $b)
b : pdl
=cut
-func allclose(PDL $a, PDL $b)
+func allclose(PDL $a, PDL $b, Maybe[Num] $threshold=)
{
- return (($a - $b)->abs <= default_numerical_threshold)->all;
+ return (($a - $b)->abs <= ($threshold//default_numerical_threshold))->all;
}
=head2 reldiff
@@ -359,31 +362,6 @@ func check_consistency(
return $gt;
}
-sub zip
-{
- my ($sub, @arrays) = @_;
- my $len = @{ $arrays[0] };
- for (my $i = 0; $i < $len; $i++)
- {
- $sub->(map { $_->[$i] } @arrays);
- }
-}
-
-sub enumerate
-{
- my ($sub, @arrays) = @_;
- my $len = @{ $arrays[0] };
- zip($sub, [0..$len-1], @arrays);
-}
-
-sub assert
-{
- my ($input, $error_str) = @_;
- local($Carp::CarpLevel) = 1;
- Carp::confess($error_str//'AssertionError')
- unless $input;
-}
-
=head2 same_array
Check whether two NDArrays sharing the same memory block
@@ -431,4 +409,422 @@ func dies_like($code, $regexp)
}
}
+func random_arrays(@shapes)
+{
+ my @arrays = map { random(reverse(@$_))->float } @shapes;
+ if(@arrays > 1)
+ {
+ return @arrays;
+ }
+ else
+ {
+ return $arrays[0];
+ }
+}
+
+
+func _validate_csr_generation_inputs(
+ $num_rows, $num_cols, $density,
+ $distribution="uniform"
+)
+{
+ my $total_nnz = int($num_rows * $num_cols * $density);
+ if($density < 0 or $density > 1)
+ {
+ confess("density has to be between 0 and 1");
+ }
+ if($num_rows <= 0 or $num_cols <= 0)
+ {
+ confess("num_rows or num_cols should be greater than 0");
+ }
+ if($distribution eq "powerlaw")
+ {
+ if($total_nnz < 2 * $num_rows)
+ {
+ confess(
+ "not supported for this density: $density"
+ ." for this shape ($num_rows, $num_cols)"
+ ." Please keep :"
+ ." num_rows * num_cols * density >= 2 * num_rows"
+ );
+ }
+ }
+}
+
+# Shuffle CSR column indices per row
+# This allows validation of unordered column indices, which is not a requirement
+# for a valid CSR matrix
+
+func shuffle_csr_column_indices($csr)
+{
+ my $row_count = @{ $csr->indptr } - 1;
+ for my $i (0..$row_count-1)
+ {
+ my $start_index = $csr->indptr->[$i];
+ my $end_index = $csr->indptr->[$i + 1];
+ my @sublist = @{$csr->indices}[$start_index .. $end_index];
+ @sublist = shuffle(@sublist);
+ @{$csr->indices}[$start_index .. $end_index] = @sublist;
+ }
+}
+
+
+func _get_uniform_dataset_csr(
+ $num_rows, $num_cols, $density=0.1, $dtype='float32',
+ $data_init=, $shuffle_csr_indices=0)
+{
+ # Returns CSRNDArray with uniform distribution
+ # This generates a csr matrix with totalnnz unique randomly chosen numbers
+ # from num_rows*num_cols and arranges them in the 2d array in the
+ # following way:
+ # row_index = (random_number_generated / num_rows)
+ # col_index = random_number_generated - row_index * num_cols
+
+ _validate_csr_generation_inputs(
+ $num_rows, $num_cols, $density,
+ "uniform"
+ );
+ my $csr = rand_sparse($num_rows, $num_cols, $density, $dtype, "csr");
+ if(defined $data_init)
+ {
+ $csr->data->fill($data_init);
+ }
+ if($shuffle_csr_indices)
+ {
+ shuffle_csr_column_indices($csr);
+ }
+ return mx->nd->sparse->csr_matrix(
+ [$csr->data, $csr->indices, $csr->indptr],
+ shape => [$num_rows, $num_cols], dtype => $dtype
+ );
+}
+
+func _get_powerlaw_dataset_csr($num_rows, $num_cols, $density=0.1, $dtype='float32')
+{
+ # Returns CSRNDArray with powerlaw distribution
+ # with exponentially increasing number of non zeros in each row.
+ # Not supported for cases where total_nnz < 2*num_rows. This is because
+ # the algorithm first tries to ensure that there are rows with no zeros by
+ # putting non zeros at beginning of each row.
+
+ _validate_csr_generation_inputs($num_rows, $num_cols, $density,
+ "powerlaw");
+
+ my $total_nnz = int($num_rows * $num_cols * $density);
+
+ my $unused_nnz = $total_nnz;
+ my $output_arr = zeros($num_cols, $num_rows);
+ # Start with ones on each row so that no row is empty
+ for my $row (0..$num_rows-1)
+ {
+ $output_arr->slice(0, $row) .= 1 + rand(2);
+ $unused_nnz--;
+ if($unused_nnz <= 0)
+ {
+ return mx->nd->array($output_arr)->tostype("csr");
+ }
+ }
+ # Populate rest of matrix with 2^i items in ith row.
+ # if we have used all total nnz return the sparse matrix
+ # else if we reached max column size then fill up full columns until we use all nnz
+ my $col_max = 2;
+ for my $row (0..$num_rows-1)
+ {
+ my $col_limit = List::Util::min($num_cols, $col_max);
+ # In case col_limit reached assign same value to all elements, which is much faster
+ if($col_limit == $num_cols and $unused_nnz > $col_limit)
+ {
+ $output_arr->slice('X', $row) .= 1 + rand(2);
+ $unused_nnz = $unused_nnz - $col_limit + 1;
+ if($unused_nnz <= 0)
+ {
+ return mx->nd->array($output_arr)->tostype("csr");
+ }
+ }
+ else
+ {
+ for my $col_index (1..$col_limit-1)
+ {
+ $output_arr->slice($col_index, $row) .= 1 + rand(2);
+ $unused_nnz--;
+ if($unused_nnz <= 0)
+ {
+ return mx->nd->array($output_arr)->tostype("csr");
+ }
+ }
+ $col_max *= 2;
+ }
+ }
+
+ if($unused_nnz > 0)
+ {
+ warn $unused_nnz;
+ confess(
+ "not supported for this density: $density"
+ ." for this shape ($num_rows,$num_cols)"
+ );
+ }
+ else
+ {
+ return mx->nd->array($output_arr)->tostype("csr");
+ }
+}
+
+
+func assign_each($input, $function=)
+{
+ my $res = pdl($input);
+ if(defined $function)
+ {
+ return $function->($res);
+ }
+ return $res;
+}
+
+func assign_each2($input1, $input2, $function=)
+{
+ my $res = pdl($input1);
+ if(defined $function)
+ {
+ return $function->($res, pdl($input2));
+ }
+ return $res;
+}
+
+=head2 rand_sparse_ndarray
+
+ Generate a random sparse ndarray. Returns the ndarray, value(np) and indices(np)
+
+ Parameters
+ ----------
+ shape: list or tuple
+ stype: str, valid values: "csr" or "row_sparse"
+ density, optional: float, should be between 0 and 1
+ distribution, optional: str, valid values: "uniform" or "powerlaw"
+ dtype, optional: numpy.dtype, default value is None
+
+ Returns
+ -------
+ Result of type CSRNDArray or RowSparseNDArray
+
+ Examples
+ --------
+ Below is an example of the powerlaw distribution with csr as the stype.
+ It calculates the nnz using the shape and density.
+ It fills up the ndarray with exponentially increasing number of elements.
+ If there are enough unused_nnzs, n+1th row will have twice more nnzs compared to nth row.
+ else, remaining unused_nnzs will be used in n+1th row
+ If number of cols is too small and we have already reached column size it will fill up
+ all following columns in all followings rows until we reach the required density.
+
+ >>> csr_arr, _ = rand_sparse_ndarray(shape=(5, 16), stype="csr",
+ density=0.50, distribution="powerlaw")
+ >>> indptr = csr_arr.indptr.asnumpy()
+ >>> indices = csr_arr.indices.asnumpy()
+ >>> data = csr_arr.data.asnumpy()
+ >>> row2nnz = len(data[indptr[1]:indptr[2]])
+ >>> row3nnz = len(data[indptr[2]:indptr[3]])
+ >>> assert(row3nnz == 2*row2nnz)
+ >>> row4nnz = len(data[indptr[3]:indptr[4]])
+ >>> assert(row4nnz == 2*row3nnz)
+=cut
+
+func rand_sparse_ndarray(
+ $shape, $stype, :$density=rand, :$dtype='float32', :$distribution='uniform',
+ :$data_init=, :$rsp_indices=, :$modifier_func=,
+ :$shuffle_csr_indices=0
+)
+{
+ if($stype eq 'row_sparse')
+ {
+ assert (
+ ($distribution eq "uniform"),
+ "Distribution $distribution not supported for row_sparse"
+ );
+ # sample index
+ my $indices;
+ if(defined $rsp_indices)
+ {
+ $indices = $rsp_indices;
+ assert($indices->nelem <= $shape->[0]);
+ }
+ else
+ {
+ my $idx_sample = random($shape->[0]);
+ $indices = which($idx_sample < $density);
+ }
+ if($indices->shape(-1)->at(0) == 0)
+ {
+ my $result = mx->nd->zeros($shape, stype=>'row_sparse', dtype=>$dtype);
+ return ($result, [pdl([]), pdl([])]);
+ }
+ # generate random values
+ my $val = random(PDL::Type->new(DTYPE_MX_TO_PDL->{ $dtype }), reverse($indices->shape(-1)->at(0), @{ $shape }[1..@{ $shape }-1]));
+
+ # Allow caller to override or adjust random values
+ if(defined $data_init)
+ {
+ $val .= $data_init;
+ }
+ if(defined $modifier_func)
+ {
+ $val = assign_each($val, $modifier_func);
+ }
+ my $arr = mx->nd->sparse->row_sparse_array([$val, $indices], shape=>$shape, dtype=>$dtype);
+ return ($arr, [$val, $indices]);
+ }
+ elsif($stype eq 'csr')
+ {
+ assert(@{ $shape } == 2);
+ my $csr;
+ if($distribution eq "uniform")
+ {
+ $csr = _get_uniform_dataset_csr(
+ @{ $shape }, $density, $dtype,
+ $data_init, $shuffle_csr_indices
+ );
+ return ($csr, [$csr->indptr, $csr->indices, $csr->data]);
+ }
+ elsif($distribution eq "powerlaw")
+ {
+ $csr = _get_powerlaw_dataset_csr(@{ $shape }, $density, $dtype);
+ return ($csr, [$csr->indptr, $csr->indices, $csr->data]);
+ }
+ else
+ {
+ confess("Distribution not supported: $distribution");
+ }
+ }
+ else
+ {
+ confess("unknown storage type");
+ }
+}
+
+func rand_ndarray(
+ $shape, $stype, $density=rand, $dtype='float32',
+ $modifier_func=, $shuffle_csr_indices=0, $distribution='uniform'
+)
+{
+ my $arr;
+ if($stype eq 'default')
+ {
+ $arr = mx->nd->array(random_arrays($shape), dtype=>$dtype);
+ }
+ else
+ {
+ ($arr) = rand_sparse_ndarray(
+ $shape, $stype, density => $density, dtype => $dtype,
+ modifier_func => $modifier_func,
+ shuffle_csr_indices => $shuffle_csr_indices, distribution => $distribution
+ );
+ }
+ return $arr;
+}
+
+
+func create_sparse_array(
+ $shape, $stype, $data_init=, $rsp_indices=,
+ $dtype=, $modifier_func=, $density=0.5,
+ $shuffle_csr_indices=0
+)
+{
+ my $arr_data;
+ if($stype eq 'row_sparse')
+ {
+ my $arr_indices;
+ if(defined $rsp_indices)
+ {
+ $arr_indices = pdl($rsp_indices);
+ $arr_indices->inplace->qsort;
+ }
+ ($arr_data) = rand_sparse_ndarray(
+ $shape, $stype,
+ $density, $dtype,
+ $data_init,
+ $arr_indices,
+ $modifier_func
+ );
+ }
+ elsif($stype eq 'csr')
+ {
+ ($arr_data) = rand_sparse_ndarray(
+ $shape,
+ $stype,
+ $density, $dtype,
+ $data_init,
+ $modifier_func,
+ $shuffle_csr_indices
+ );
+ }
+ else
+ {
+ confess("Unknown storage type: $stype");
+ }
+ return $arr_data;
+}
+
+
+func create_sparse_array_zd(
+ $shape, $stype, $density, $data_init=,
+ $rsp_indices=, $dtype=, $modifier_func=,
+ $shuffle_csr_indices=0
+)
+{
+ if($stype eq 'row_sparse')
+ {
+ $density = 0;
+ if(defined $rsp_indices)
+ {
+ assert($rsp_indices->len <= $shape->[0]);
+ }
+ }
+ return create_sparse_array(
+ $shape, $stype,
+ $data_init,
+ $rsp_indices,
+ $dtype,
+ $modifier_func,
+ $density,
+ $shuffle_csr_indices
+ );
+}
+
+func rand_shape_2d($dim0=10, $dim1=10)
+{
+ [int(rand($dim0)+1), int(rand($dim1)+1)];
+}
+
+
+func rand_shape_3d($dim0=10, $dim1=10, $dim2=10)
+{
+ [int(rand($dim0)+1), int(rand($dim1)+1), int(rand($dim1)+1)];
+}
+
+
+func rand_shape_nd($num_dim, $dim=10)
+{
+ (random($num_dim)*$dim+1)->floor->unpdl;
+}
+
+func randint($low=0, $high=10)
+{
+ my $value = int(rand($high));
+ return $value < $low ? $low : $value;
+}
+
+sub dies_ok
+{
+ my $sub = shift;
+ eval { $sub->() };
+ if($@)
+ {
+ Test::More::ok(1);
+ }
+ else
+ {
+ Test::More::ok(0);
+ }
+}
+
1;
diff --git a/perl-package/AI-MXNet/lib/AI/MXNet/Types.pm b/perl-package/AI-MXNet/lib/AI/MXNet/Types.pm
index d48ccdd..835d36c 100644
--- a/perl-package/AI-MXNet/lib/AI/MXNet/Types.pm
+++ b/perl-package/AI-MXNet/lib/AI/MXNet/Types.pm
@@ -25,6 +25,7 @@ use base qw(Exporter);
class_type 'PDL';
class_type 'PDL::Matrix';
+class_type 'PDL::CCS::Nd';
class_type 'AI::MXNet::NDArray';
class_type 'AI::MXNet::Symbol';
class_type 'AI::MXNet::NDArray::Slice';
@@ -44,7 +45,10 @@ class_type 'AI::MXNet::Gluon::Block';
class_type 'AI::MXNet::Gluon::Data::Set';
class_type 'AI::MXNet::Gluon::RNN::HybridRecurrentCell';
class_type 'AI::MXNet::Symbol::NameManager';
-subtype "AcceptableInput" => as "Num|PDL|PDL::Matrix|AI::MXNet::NDArray|AI::MXNet::NDArray::Slice|ArrayRef";
+class_type 'AI::MXNet::NDarray::CSR';
+class_type 'AI::MXNet::NDArray::RowSparse';
+class_type 'PDL::CCS::Nd';
+subtype "AcceptableInput" => as "Num|PDL|PDL::Matrix|PDL::CCS::Nd|AI::MXNet::NDArray|AI::MXNet::NDArray::Slice|ArrayRef|AI::MXNet::NDarray::CSR|AI::MXNet::NDArray::RowSparse";
subtype "Index" => as "Int";
subtype "DimSize" => as "Int" => where { $_ >= 0 };
subtype "Dropout" => as "Num" => where { $_ >= 0 and $_ <= 1 };
@@ -52,10 +56,9 @@ subtype "Shape" => as "ArrayRef[DimSize]";
subtype "CudaKernelShape" => as "Shape" => where { @$_ == 3 };
subtype "WholeDim" => as "Str" => where { $_ eq 'X' };
subtype "Slice" => as "ArrayRef[Index]|WholeDim|Index" => where { ref $_ ? @$_ > 0 : 1 };
-subtype "Dtype" => as enum([qw[float32 float64 float16 uint8 int32]]);
+subtype "Dtype" => as enum([qw[float32 float64 float16 uint8 int8 int32 int64]]);
subtype "GluonClass" => as enum([qw[AI::MXNet::NDArray AI::MXNet::Symbol]]);
subtype "GluonInput" => as "AI::MXNet::NDArray|AI::MXNet::Symbol|ArrayRef[AI::MXNet::NDArray|AI::MXNet::Symbol]";
-subtype "ProfilerState" => as enum([qw[stop run]]);
subtype "GradReq" => as enum([qw[add write null]]);
subtype "KVStoreStr" => as enum([qw[local device dist dist_sync dist_async]]);
subtype "PoolType" => as enum([qw[max avg sum]]);
@@ -75,5 +78,8 @@ subtype "Activation" => as "AI::MXNet::Symbol|Str|CodeRef";
subtype "SymbolOrArrayOfSymbols" => as "AI::MXNet::Symbol|ArrayRef[AI::MXNet::Symbol]";
subtype "NameShapeOrDataDesc" => as "NameShape|AI::MXNet::DataDesc";
subtype "AdvancedSlice" => as "ArrayRef[ArrayRef|PDL|PDL::Matrix|AI::MXNet::NDArray]";
+subtype "InternalSlice" => as enum([qw[begin end step]]);
+subtype "Stype" => as enum([qw[default csr row_sparse]]);
+subtype "AuxTypes" => as "ArrayRef[Dtype]";
1;
diff --git a/perl-package/AI-MXNet/t/test_autograd.t b/perl-package/AI-MXNet/t/test_autograd.t
index b45d233..931c6d5 100644
--- a/perl-package/AI-MXNet/t/test_autograd.t
+++ b/perl-package/AI-MXNet/t/test_autograd.t
@@ -19,9 +19,10 @@ use strict;
use warnings;
use AI::MXNet qw(mx);
use AI::MXNet::AutoGrad qw(autograd);
-use AI::MXNet::TestUtils qw(same);
-use AI::MXNet::Base;
-use Test::More tests => 74;
+use AI::MXNet::TestUtils qw(same almost_equal rand_ndarray);
+use AI::MXNet::Base qw(:DEFAULT pones);
+use Test::More tests => 246;
+$ENV{MXNET_STORAGE_FALLBACK_LOG_VERBOSE} = 0;
sub autograd_assert
{
@@ -59,12 +60,11 @@ sub test_unary_func
};
my $uniform = mx->nd->uniform(shape=>[4, 5]);
$check_unary_func->($uniform);
- # sparse support
- #my $stypes = ['row_sparse', 'csr', 'default'];
- #for my $stype (@$stypes)
- #{
- # $check_unary_func->($uniform->tostype($stype));
- #}
+ my $stypes = ['row_sparse', 'csr', 'default'];
+ for my $stype (@$stypes)
+ {
+ $check_unary_func->($uniform->tostype($stype));
+ }
}
test_unary_func();
@@ -86,17 +86,16 @@ sub test_binary_func
my $uniform_x = mx->nd->uniform(shape=>[4, 5]);
my $uniform_y = mx->nd->uniform(shape=>[4, 5]);
$check_binary_func->($uniform_x, $uniform_y);
- # sparse support
- #my $stypes = ['row_sparse', 'csr', 'default'];
- #for my $stype_x (@$stypes)
- #{
- # for my $stype_y (@$stypes)
- # {
- # my $x = $uniform_x->tostype($stype_x);
- # my $y = $uniform_y->tostype($stype_y);
- # $check_binary_func->($x, $y);
- # }
- #}
+ my $stypes = ['row_sparse', 'csr', 'default'];
+ for my $stype_x (@$stypes)
+ {
+ for my $stype_y (@$stypes)
+ {
+ my $x = $uniform_x->tostype($stype_x);
+ my $y = $uniform_y->tostype($stype_y);
+ $check_binary_func->($x, $y);
+ }
+ }
}
test_binary_func();
@@ -270,11 +269,12 @@ sub test_attach_grad
};
my $zeros = mx->nd->zeros([10, 10]);
$check_attach_grad->($zeros);
- # sparse support
- #stypes = ['default', 'row_sparse', 'csr']
- #for stype in stypes:
- # x = zeros.tostype(stype)
- # check_attach_grad(x)
+ my @stypes = ('default', 'row_sparse', 'csr');
+ for my $stype (@stypes)
+ {
+ my $x = $zeros->tostype($stype);
+ $check_attach_grad->($x);
+ }
}
test_attach_grad();
@@ -368,3 +368,58 @@ sub test_gradient
}
test_gradient();
+
+sub test_grad_with_stype
+{
+ my $check_grad_with_stype = sub { my ($array_stype, $grad_stype, $expected_stype) = @_;
+ my $x = mx->nd->zeros([1, 1], stype=>$array_stype);
+ $x->attach_grad(stype=>$grad_stype);
+ # check grad attached
+ ok($x->grad->stype eq $expected_stype);
+ my $y = $x->detach();
+ # check array detached
+ ok($y->stype eq $array_stype);
+ };
+ my @stypes = ('default', 'csr', 'row_sparse');
+ for my $stype (@stypes)
+ {
+ # check the default stype of the gradient (same as the array stype)
+ $check_grad_with_stype->($stype, undef, $stype);
+ for my $grad_stype (@stypes)
+ {
+ # check the stype of the gradient when provided
+ $check_grad_with_stype->($stype, $grad_stype, $grad_stype);
+ }
+ }
+}
+
+test_grad_with_stype();
+
+sub test_sparse_dot_grad
+{
+ my $check_sparse_dot_grad = sub { my ($rhs) = @_;
+ my $lhs = rand_ndarray([2, 8], 'csr');
+ my $y;
+ mx->autograd->record(sub {
+ $y = mx->nd->dot($lhs, $rhs);
+ });
+ $y->backward();
+ my $grad = $rhs->grad;
+ my $grad_pdl = $lhs->aspdl->transpose x pones($rhs->shape->[1], $lhs->shape->[0]);
+ ok($grad->stype eq 'row_sparse');
+ ok(almost_equal($grad->aspdl, $grad_pdl));
+ };
+
+ # check grad with row_sparse weight
+ my $shape = [8, 3];
+ my $rsp = mx->nd->ones($shape)->tostype('row_sparse');
+ $rsp->attach_grad();
+ $check_sparse_dot_grad->($rsp);
+
+ # check grad with dense weight
+ my $dns = mx->nd->ones($shape);
+ $dns->attach_grad(stype=>'row_sparse');
+ $check_sparse_dot_grad->($dns);
+}
+
+test_sparse_dot_grad();
diff --git a/perl-package/AI-MXNet/t/test_init.t b/perl-package/AI-MXNet/t/test_init.t
index 0b4e89c..bf811f8 100644
--- a/perl-package/AI-MXNet/t/test_init.t
+++ b/perl-package/AI-MXNet/t/test_init.t
@@ -17,7 +17,7 @@
use strict;
use warnings;
-use Test::More tests => 4;
+use Test::More tests => 7;
use AI::MXNet qw(mx);
sub test_default_init
@@ -38,7 +38,7 @@ sub test_variable_init
my $mod = mx->mod->Module($sym);
$mod->bind(data_shapes=>[['data', [10,10]]]);
$mod->init_params();
- ok((((values %{ ($mod->get_params)[0] }))[0]->aspdl == 1)->all);
+ ok(((values %{ ($mod->get_params)[0] })[0]->aspdl == 1)->all);
}
sub test_aux_init
@@ -52,6 +52,26 @@ sub test_aux_init
ok((($mod->get_params)[1]->{bn_moving_mean}->aspdl == 0)->all);
}
+$ENV{MXNET_STORAGE_FALLBACK_LOG_VERBOSE} = 0;
+sub test_rsp_const_init
+{
+ my $check_rsp_const_init = sub { my ($init, $val) = @_;
+ my $shape = [10, 10];
+ my $x = mx->symbol->Variable("data", stype=>'csr');
+ my $weight = mx->symbol->Variable("weight", shape=>[$shape->[1], 2],
+ init=>$init, stype=>'row_sparse');
+ my $dot = mx->symbol->sparse->dot($x, $weight);
+ my $mod = mx->mod->Module($dot);
+ $mod->bind(data_shapes=>[['data', $shape]]);
+ $mod->init_params;
+ ok(((values %{ ($mod->get_params)[0] })[0]->aspdl == $val)->all);
+ };
+ $check_rsp_const_init->(mx->initializer->Constant(value=>2), 2);
+ $check_rsp_const_init->(mx->initializer->Zero(), 0);
+ $check_rsp_const_init->(mx->initializer->One(), 1);
+}
+
+test_rsp_const_init();
test_default_init();
test_variable_init();
test_aux_init();
diff --git a/perl-package/AI-MXNet/t/test_io.t b/perl-package/AI-MXNet/t/test_io.t
index e854749..557925b 100644
--- a/perl-package/AI-MXNet/t/test_io.t
+++ b/perl-package/AI-MXNet/t/test_io.t
@@ -16,8 +16,8 @@
# under the License.
use AI::MXNet qw(mx);
-use Test::More tests => 31;
-use AI::MXNet::TestUtils qw(same reldiff GetMNIST_ubyte GetCifar10);
+use Test::More tests => 36;
+use AI::MXNet::TestUtils qw(same reldiff almost_equal GetMNIST_ubyte GetCifar10 randint rand_sparse_ndarray dies_ok);
use PDL;
use PDL::Types;
use PDL::NiceSlice;
@@ -109,6 +109,49 @@ sub test_NDArrayIter()
}
}
+sub test_NDArrayIter_csr
+{
+ # creating toy data
+ my $num_rows = 20;
+ my $num_cols = 20;
+ my $batch_size = 6;
+ my $shape = [$num_rows, $num_cols];
+ my ($csr) = rand_sparse_ndarray($shape, 'csr', density => 0.5);
+ my $dns = $csr->aspdl;
+ dies_ok(sub { mx->io->NDArrayIter->new(data => $csr, label => $dns, batch_size => $batch_size) });
+
+ # AI::MXNet::NDArray::CSR with shuffle
+ my $csr_iter = mx->io->NDArrayIter(
+ data => Hash::Ordered->new(csr_data => $csr, dns_data => $dns),
+ label => $dns,
+ batch_size => $batch_size,
+ shuffle=>1, last_batch_handle=>'discard'
+ );
+ my $num_batch = 0;
+ for my $batch (@{ $csr_iter })
+ {
+ $num_batch += 1;
+ }
+
+ ok($num_batch == int($num_rows / $batch_size));
+
+ # make iterators
+ $csr_iter = mx->io->NDArrayIter(data => $dns, label => $dns, batch_size => $batch_size, last_batch_handle=>'discard');
+ my $begin = 0;
+ for my $batch (@{ $csr_iter })
+ {
+ my $expected = mx->nd->zeros([$batch_size, $num_cols])->aspdl;
+ my $end = $begin + $batch_size;
+ $expected->slice('X', [0, $batch_size-1]) .= $dns->slice('X', [$begin, $end-1]);
+ if($end > $num_rows)
+ {
+ $expected->slice('X', [0, $end - $num_rows - 1]) .= $dns->slice('X', [0, $end - $num_rows - 1]);
+ }
+ ok(almost_equal($batch->data->[0]->aspdl, $expected));
+ $begin += $batch_size;
+ }
+}
+
sub test_MNISTIter()
{
GetMNIST_ubyte();
@@ -146,5 +189,6 @@ sub test_MNISTIter()
}
test_NDArrayIter();
+test_NDArrayIter_csr();
test_MNISTIter();
test_Cifar10Rec();
diff --git a/perl-package/AI-MXNet/t/test_kvstore.t b/perl-package/AI-MXNet/t/test_kvstore.t
index 1d94bc9..5e3f49f 100644
--- a/perl-package/AI-MXNet/t/test_kvstore.t
+++ b/perl-package/AI-MXNet/t/test_kvstore.t
@@ -17,8 +17,10 @@
use strict;
use warnings;
-use Test::More tests => 38;
+use Test::More tests => 62;
use AI::MXNet qw(mx);
+use AI::MXNet::TestUtils qw(almost_equal randint zip rand_ndarray);
+use AI::MXNet::Base qw(pzeros);
my $shape = [4, 4];
my $keys = [5,7,9];
@@ -73,6 +75,93 @@ sub test_list_kv_pair
}
}
+sub test_row_sparse_pull
+{
+ my $kv = mx->kv->create();
+ $kv->init('e', mx->nd->ones($shape)->tostype('row_sparse'));
+
+ my $check_row_sparse_pull = sub { my ($kv, $count) = @_;
+ my $num_rows = $shape->[0];
+ my $vals = [];
+ my $row_ids = [];
+ my $all_row_ids = mx->nd->array([0..$num_rows-1])->aspdl;
+ for my $i (0..$count-1)
+ {
+ push @$vals, mx->nd->zeros($shape)->tostype('row_sparse');
+ my $row_id = [map { randint(0, $num_rows) } 1..$num_rows];
+ push @$row_ids, mx->nd->array($row_id)->reshape([2, int($num_rows/2)]);
+ }
+ my $row_ids_to_pull = @$row_ids == 1 ? $row_ids->[0] : $row_ids;
+ my $vals_to_pull = @$vals == 1 ? $vals->[0] : $vals;
+
+ $kv->row_sparse_pull('e', out=>$vals_to_pull, row_ids=>$row_ids_to_pull);
+ zip(sub {
+ my ($val, $row_id) = @_;
+ my $retained_val = $val->aspdl;
+ my %excluded_row_ids = map { $_ => 1 } @{ PDL::setops($all_row_ids, 'XOR', $row_id->aspdl)->unpdl };
+ for my $row (0..$num_rows-1)
+ {
+ my $expected_val = pzeros(@{ $retained_val->at($row)->shape->unpdl });
+ $expected_val += exists $excluded_row_ids{ $row } ? 0 : 1;
+ ok(almost_equal($retained_val->at($row), $expected_val));
+ }
+ }, $vals, $row_ids);
+ };
+ $check_row_sparse_pull->($kv, 1);
+ $check_row_sparse_pull->($kv, 4);
+}
+
+sub test_sparse_aggregator
+{
+ my $stype = 'row_sparse';
+ my $kv = mx->kv->create($stype);
+ $kv->init('a', mx->nd->zeros($shape, stype=>$stype));
+ $kv->init($keys, [map { mx->nd->zeros($shape, stype=>$stype) } 0..@$keys-1]);
+
+ # devices
+ my $num_devs = 4;
+ my $devs = [map { mx->cpu($_) } 0..$num_devs];
+
+ # single
+ my $vals = [map { rand_ndarray($shape, $stype)->copyto($devs->[$_]) } 0..$num_devs-1];
+ my $expected_sum = mx->nd->zeros($shape)->aspdl;
+ for my $v (@$vals)
+ {
+ $expected_sum += $v->aspdl;
+ }
+
+ # prepare row_ids
+ my $all_rows = mx->nd->array([0..$shape->[0]-1]);
+ $kv->push('a', $vals);
+ $kv->row_sparse_pull('a', out=>$vals, row_ids=>[($all_rows)x@$vals]);
+ my $result_sum = mx->nd->zeros($shape)->aspdl;
+ for my $v (@$vals)
+ {
+ $result_sum += $v->aspdl;
+ }
+ ok(almost_equal($result_sum, $expected_sum * $num_devs));
+
+ # list
+ $vals = [([map { rand_ndarray($shape, $stype)->copyto($devs->[$_]) } 0..$num_devs-1])x@$keys];
+ $expected_sum = mx->nd->zeros($shape)->aspdl;
+ for my $v (@{ $vals->[0] })
+ {
+ $expected_sum += $v->aspdl;
+ }
+
+ $kv->push($keys, $vals);
+ $kv->row_sparse_pull($keys, out=>$vals, row_ids=>[([($all_rows)x$num_devs])x@$vals]);
+ for my $vv (@$vals)
+ {
+ $result_sum = mx->nd->zeros($shape)->aspdl;
+ for my $v (@$vv)
+ {
+ $result_sum += $v->aspdl;
+ }
+ ok(almost_equal($result_sum, $expected_sum * $num_devs))
+ }
+}
+
sub test_aggregator
{
# aggregate value on muliple devices
@@ -169,3 +258,5 @@ test_single_kv_pair();
test_list_kv_pair();
test_aggregator();
test_updater();
+test_row_sparse_pull();
+test_sparse_aggregator();
diff --git a/perl-package/AI-MXNet/t/test_module.t b/perl-package/AI-MXNet/t/test_module.t
index c5f7410..c29c459 100644
--- a/perl-package/AI-MXNet/t/test_module.t
+++ b/perl-package/AI-MXNet/t/test_module.t
@@ -17,10 +17,11 @@
use strict;
use warnings;
-use Test::More tests => 257;
+use Test::More tests => 426;
use AI::MXNet qw(mx);
use AI::MXNet::Base;
-use AI::MXNet::TestUtils qw(almost_equal enumerate same_array dies_like);
+use AI::MXNet::TestUtils qw(almost_equal enumerate same_array dies_like rand_ndarray);
+$ENV{MXNET_STORAGE_FALLBACK_LOG_VERBOSE} = 0;
sub test_module_layout
{
@@ -351,7 +352,7 @@ sub test_module_input_grads
sub test_executor_group
{
- my $get_rnn_sym = sub { my ($num_layers, $num_words, $num_hidden, $num_embed, $seq_len) = @_;
+ my $get_rnn_sym = sub { my ($num_layers, $num_words, $num_hidden, $num_embed, $seq_len, $sparse_embedding) = @_;
my $stack = mx->rnn->SequentialRNNCell();
for my $i (0..$num_layers-1)
{
@@ -359,8 +360,20 @@ sub test_executor_group
}
my $data = mx->sym->Variable('data');
my $label = mx->sym->Variable('softmax_label');
- my $embed = mx->sym->Embedding(data=>$data, input_dim=>$num_words,
+ my $embed;
+ if($sparse_embedding)
+ {
+ my $embed_weight = mx->sym->Variable('embed_weight', stype=>'row_sparse');
+ $embed = mx->sym->contrib->SparseEmbedding(data=>$data, input_dim=>$num_words,
+ weight=>$embed_weight, output_dim=>$num_embed,
+ name=>'embed');
+
+ }
+ else
+ {
+ $embed = mx->sym->Embedding(data=>$data, input_dim=>$num_words,
output_dim=>$num_embed, name=>'embed');
+ }
$stack->reset();
my ($outputs, $states) = $stack->unroll($seq_len, inputs=>$embed, merge_outputs=>1);
@@ -373,7 +386,7 @@ sub test_executor_group
return $pred;
};
- my $test_shared_exec_group = sub { my ($exec_grp_shared, $exec_grp_created, $shared_arg_names, $extra_args) = @_;
+ my $test_shared_exec_group = sub { my ($exec_grp_shared, $exec_grp_created, $shared_arg_names, $extra_args, $check_grads) = @_;
# Test shared data arrays
for my $i (0..@{ $exec_grp_shared->execs }-1)
{
@@ -408,8 +421,11 @@ sub test_executor_group
# test shared argument gradients
for my $arg_name (@$shared_arg_names)
{
- ok(exists $exec_created->grad_dict->{$arg_name});
- ok(same_array($exec_shared->grad_dict->{$arg_name}, $exec_created->grad_dict->{$arg_name}));
+ if($check_grads)
+ {
+ ok(exists $exec_created->grad_dict->{$arg_name});
+ ok(same_array($exec_shared->grad_dict->{$arg_name}, $exec_created->grad_dict->{$arg_name}));
+ }
}
}
my $grad_req = $exec_grp_shared->grad_req;
@@ -419,52 +435,199 @@ sub test_executor_group
}
}
};
- my $contexts = [mx->cpu(0), mx->cpu(1)];
- my $workload = [(1) x scalar(@$contexts)];
- my $batch_size = 32;
- my $max_bucket_size = 80;
- my $num_words = 1000;
- my $num_hidden = 100;
- my $num_embed = 200;
- my $data_shapes = [['data', [$batch_size, $max_bucket_size]]];
- my $label_shapes = [['softmax_label', [$batch_size, $max_bucket_size]]];
-
- # generate an rnn sym with #layers=5
- my $sym = $get_rnn_sym->(3, $num_words, $num_hidden,
- $num_embed, $max_bucket_size);
- my $arg_names1 = $sym->list_arguments();
- my $input_names = ['data', 'softmax_label'];
- my $shared_arg_names = [grep { !/^(?:data|softmax_label)$/ } @$arg_names1];
- my $exec_group1 = AI::MXNet::DataParallelExecutorGroup->new(
- symbol=>$sym, contexts=>$contexts,
- workload=>$workload, data_shapes=>$data_shapes,
- label_shapes=>$label_shapes, param_names=>$shared_arg_names,
- for_training=>1, inputs_need_grad=>0
- );
- # shared_data_arrays should only have input "data" and "softmax_label" arrays
- for my $i (0..@{$contexts}-1)
+
+ for my $sparse_embedding (0, 1)
{
- ok(keys(%{$exec_group1->shared_data_arrays->[$i]}) == @$input_names);
- for my $name (@$input_names)
+ my $contexts = [mx->cpu(0), mx->cpu(1)];
+ my $workload = [(1) x scalar(@$contexts)];
+ my $batch_size = 32;
+ my $max_bucket_size = 80;
+ my $num_words = 1000;
+ my $num_hidden = 100;
+ my $num_embed = 200;
+ my $data_shapes = [['data', [$batch_size, $max_bucket_size]]];
+ my $label_shapes = [['softmax_label', [$batch_size, $max_bucket_size]]];
+
+ # generate an rnn sym with #layers=5
+ my $sym = $get_rnn_sym->(3, $num_words, $num_hidden,
+ $num_embed, $max_bucket_size, $sparse_embedding);
+ my $arg_names1 = $sym->list_arguments();
+ my $input_names = ['data', 'softmax_label'];
+ my $shared_arg_names = [grep { !/^(?:data|softmax_label)$/ } @$arg_names1];
+ my $exec_group1 = AI::MXNet::DataParallelExecutorGroup->new(
+ symbol=>$sym, contexts=>$contexts,
+ workload=>$workload, data_shapes=>$data_shapes,
+ label_shapes=>$label_shapes, param_names=>$shared_arg_names,
+ for_training=>1, inputs_need_grad=>0
+ );
+ # shared_data_arrays should only have input "data" and "softmax_label" arrays
+ for my $i (0..@{$contexts}-1)
{
- ok(exists $exec_group1->shared_data_arrays->[$i]->{$name});
+ ok(keys(%{$exec_group1->shared_data_arrays->[$i]}) == @$input_names);
+ for my $name (@$input_names)
+ {
+ ok(exists $exec_group1->shared_data_arrays->[$i]->{$name});
+ }
}
- }
- # generate an rnn sym with #layers=5
- $sym = $get_rnn_sym->(5, $num_words, $num_hidden,
- $num_embed, $max_bucket_size);
- my $arg_names2 = $sym->list_arguments();
- my $exec_group2 = AI::MXNet::DataParallelExecutorGroup->new(symbol=>$sym, contexts=>$contexts,
+ # generate an rnn sym with #layers=5
+ $sym = $get_rnn_sym->(5, $num_words, $num_hidden,
+ $num_embed, $max_bucket_size, $sparse_embedding);
+ my $arg_names2 = $sym->list_arguments();
+ my $exec_group2 = AI::MXNet::DataParallelExecutorGroup->new(symbol=>$sym, contexts=>$contexts,
workload=>$workload, data_shapes=>$data_shapes,
label_shapes=>$label_shapes, param_names=>$shared_arg_names,
for_training=>1, inputs_need_grad=>0,
shared_group=>$exec_group1);
- my %shared_arg_names = map { $_ => 1 } @$shared_arg_names;
- my $extra_args = [grep { not exists $shared_arg_names{$_} } @$arg_names2];
- $test_shared_exec_group->(
- $exec_group1, $exec_group2,
- $shared_arg_names, $extra_args
- );
+ my %shared_arg_names = map { $_ => 1 } @$shared_arg_names;
+ my $extra_args = [grep { not exists $shared_arg_names{$_} } @$arg_names2];
+ $test_shared_exec_group->(
+ $exec_group1, $exec_group2,
+ $shared_arg_names, $extra_args, not $sparse_embedding
+ );
+ }
+}
+
+sub test_factorization_machine_module
+{
+ mx->random->seed(11);
+ my $check_factorization_machine_module = sub { my ($optimizer, $num_epochs) = @_;
+ my $fm = sub { my ($factor_size, $feature_dim, $init) = @_;
+ my $x = mx->symbol->Variable("data", stype=>'csr');
+ my $v = mx->symbol->Variable("v", shape=>[$feature_dim, $factor_size],
+ init=>$init, stype=>'row_sparse');
+
+ my $w1_weight = mx->symbol->var('w1_weight', shape=>[$feature_dim, 1],
+ init=>$init, stype=>'row_sparse');
+ my $w1_bias = mx->symbol->var('w1_bias', shape=>[1]);
+ my $w1 = mx->symbol->broadcast_add(mx->symbol->dot($x, $w1_weight), $w1_bias);
+
+ my $v_s = mx->symbol->_square_sum(data=>$v, axis=>1, keepdims=>1);
+ my $x_s = mx->symbol->square(data=>$x);
+ my $bd_sum = mx->sym->dot($x_s, $v_s);
+
+ my $w2 = mx->symbol->dot($x, $v);
+ my $w2_squared = 0.5 * mx->symbol->square(data=>$w2);
+
+ my $w_all = mx->symbol->Concat($w1, $w2_squared, dim=>1);
+ my $sum1 = mx->symbol->sum(data=>$w_all, axis=>1, keepdims=>1);
+ my $sum2 = 0.5 * mx->symbol->negative($bd_sum);
+ my $model = mx->sym->elemwise_add($sum1, $sum2);
+
+ my $y = mx->symbol->Variable("label");
+ $model = mx->symbol->LinearRegressionOutput(data=>$model, label=>$y);
+ return $model
+ };
+
+ # model
+ my $init = mx->initializer->Normal(sigma=>0.01);
+ my $factor_size = 4;
+ my $feature_dim = 10000;
+ my $model = $fm->($factor_size, $feature_dim, $init);
+
+ # data iter
+ my $num_batches = 5;
+ my $batch_size = 64;
+ my $num_samples = $batch_size * $num_batches;
+ # generate some random csr data
+ my $csr_nd = rand_ndarray([$num_samples, $feature_dim], 'csr', 0.1);
+ my $label = mx->nd->ones([$num_samples,1]);
+ # the alternative is to use LibSVMIter
+ my $train_iter = mx->io->NDArrayIter(data=>$csr_nd,
+ label=>Hash::Ordered->new(label => $label),
+ batch_size=>$batch_size,
+ last_batch_handle=>'discard');
+ # create module
+ my $mod = mx->mod->Module(symbol=>$model, data_names=>['data'], label_names=>['label']);
+ # allocate memory by given the input data and lable shapes
+ $mod->bind(data_shapes=>$train_iter->provide_data, label_shapes=>$train_iter->provide_label);
+ # initialize parameters by uniform random numbers
+ $mod->init_params(initializer=>$init);
+ my $expected_accuracy;
+ if($optimizer eq 'sgd')
+ {
+ # use Sparse SGD with learning rate 0.1 to train
+ my $sgd = mx->optimizer->SGD(momentum=>0.1, clip_gradient=>5.0, learning_rate=>0.01,
+ rescale_grad=>1.0/$batch_size);
+ $mod->init_optimizer(optimizer=>$sgd);
+ $num_epochs //= 10;
+ $expected_accuracy = 0.02;
+ }
+ elsif($optimizer eq 'adam')
+ {
+ # use Sparse Adam to train
+ my $adam = mx->optimizer->Adam(clip_gradient=>5.0, learning_rate=>0.0005,
+ rescale_grad=>1.0/$batch_size);
+ $mod->init_optimizer(optimizer=>$adam);
+ $num_epochs //= 10;
+ $expected_accuracy = 0.05;
+ }
+ elsif($optimizer eq 'adagrad')
+ {
+ # use Sparse AdaGrad with learning rate 0.1 to train
+ my $adagrad = mx->optimizer->AdaGrad(clip_gradient=>5.0, learning_rate=>0.01,
+ rescale_grad=>1.0/$batch_size);
+ $mod->init_optimizer(optimizer=>$adagrad);
+ $num_epochs //= 20;
+ $expected_accuracy = 0.09;
+ }
+ else
+ {
+ die "Unsupported optimizer type $optimizer specified";
+ }
+ # use accuracy as the metric
+ my $metric = mx->metric->create('MSE');
+ # train 'num_epochs' epoch
+ for my $epoch (1..$num_epochs)
+ {
+ $train_iter->reset();
+ $metric->reset();
+ while(my $batch = <$train_iter>)
+ {
+ $mod->forward($batch, is_train=>1); # compute predictions
+ $mod->update_metric($metric, $batch->label); # accumulate prediction accuracy
+ $mod->backward(); # compute gradients
+ $mod->update(); # update parameters
+ }
+ }
+ if($num_epochs > 1)
+ {
+ ok(($metric->get)[1] < $expected_accuracy);
+ }
+ };
+
+ $check_factorization_machine_module->('sgd');
+ $check_factorization_machine_module->('adam');
+ $check_factorization_machine_module->('adagrad');
+}
+
+
+sub test_module_initializer
+{
+ my $regression_model = sub { my ($m) = @_;
+ my $x = mx->symbol->var("data", stype=>'csr');
+ my $v = mx->symbol->var("v", shape=>[$m, 1], init=>mx->init->Uniform(scale=>.1),
+ stype=>'row_sparse');
+ my $model = mx->symbol->dot(lhs=>$x, rhs=>$v);
+ my $y = mx->symbol->Variable("label");
+ $model = mx->symbol->LinearRegressionOutput(data=>$model, label=>$y, name=>"out");
+ return $model
+ };
+
+ my ($n, $m) = (128, 100);
+ my $model = $regression_model->($m);
+
+ my $data = mx->nd->zeros([$n, $m], stype=>'csr');
+ my $label = mx->nd->zeros([$n, 1]);
+ my $iterator = mx->io->NDArrayIter(data=>$data, label=>Hash::Ordered->new(label => $label),
+ batch_size=>$n, last_batch_handle=>'discard');
+
+ # create module
+ my $mod = mx->mod->Module(symbol=>$model, data_names=>['data'], label_names=>['label']);
+ $mod->bind(data_shapes=>$iterator->provide_data, label_shapes=>$iterator->provide_label);
+ $mod->init_params();
+ my $v = $mod->_arg_params->{v};
+ ok($v->stype eq 'row_sparse');
+ ok($v->aspdl->sum != 0);
}
sub test_module_set_params
@@ -637,3 +800,5 @@ test_save_load();
test_executor_group();
test_module_set_params();
test_forward_reshape();
+test_module_initializer();
+test_factorization_machine_module();
diff --git a/perl-package/AI-MXNet/t/test_optimizers.t b/perl-package/AI-MXNet/t/test_optimizers.t
index 6e1c0d1..07f362a 100644
--- a/perl-package/AI-MXNet/t/test_optimizers.t
+++ b/perl-package/AI-MXNet/t/test_optimizers.t
@@ -313,21 +313,368 @@ method update($index, $weight, $grad, $state)
}
}
+package PerlSparseSGD;
+# perl reference implemenation of sgd
+use Mouse;
+use AI::MXNet::TestUtils qw(almost_equal);
+extends 'AI::MXNet::Optimizer';
+has '+learning_rate' => (default => 0.01);
+has 'momentum' => (is => "ro", isa => "Num", default => 0);
+has 'multi_precision' => (is => 'ro', isa => 'Bool', default => 0);
+has 'lazy_update' => (is => 'ro', isa => 'Bool', default => 1);
+
+method create_state($index, $weight)
+{
+ if($self->momentum == 0)
+ {
+ return undef;
+ }
+ else
+ {
+ return mx->nd->zeros($weight->shape, ctx => $weight->context, dtype => $weight->dtype);
+ }
+}
+
+method update($index, $weight, $grad, $state)
+{
+ my $lr = $self->_get_lr($index);
+ my $wd = $self->_get_wd($index);
+ $self->_update_count($index);
+ my $num_rows = $weight->shape->[0];
+ if($self->momentum == 0)
+ {
+ # Update on a per row basis, skip all-zero rows
+ for my $row (0..$num_rows-1)
+ {
+ my $grad_row = $grad->at($row);
+ my $all_zeros = almost_equal($grad_row->aspdl, mx->nd->zeros($grad_row->shape, ctx => $grad_row->context, dtype => $grad_row->dtype)->aspdl);
+ if($all_zeros and $self->lazy_update)
+ {
+ next;
+ }
+ if(defined $self->clip_gradient)
+ {
+ $weight->at($row) .= (
+ (1 - $lr*$wd)*$weight->at($row) -
+ $lr * mx->nd->clip(
+ $grad->at($row)*$self->rescale_grad,
+ -$self->clip_gradient, $self->clip_gradient
+ )
+ );
+ }
+ else
+ {
+ $weight->at($row) .= (1 - $lr*$wd)*$weight->at($row) - $lr*$self->rescale_grad*$grad->at($row);
+ }
+ }
+ }
+ else
+ {
+ my $mom = $state;
+ for my $row (0..$num_rows-1)
+ {
+ my $grad_row = $grad->at($row);
+ my $all_zeros = almost_equal($grad_row->aspdl, mx->nd->zeros($grad_row->shape, ctx => $grad_row->context, dtype => $grad_row->dtype)->aspdl);
+ if($all_zeros and $self->lazy_update)
+ {
+ next;
+ }
+ if(defined $self->clip_gradient)
+ {
+ $mom->at($row) .= ($self->momentum*$mom->at($row) - $lr*$wd*$weight->at($row) -
+ $lr * mx->nd->clip($grad->at($row)*$self->rescale_grad, -$self->clip_gradient, $self->clip_gradient)
+ );
+ $weight->at($row) += $mom->at($row);
+ }
+ else
+ {
+ $mom->at($row) .= $self->momentum*$mom->at($row) - $lr*$wd*$weight->at($row) - $lr*$self->rescale_grad*$grad->at($row);
+ $weight->at($row) += $mom->at($row);
+ }
+ }
+ }
+
+}
+
+package PerlNAG;
+use Mouse;
+extends 'PerlSGD';
+
+method create_state($index, $weight)
+{
+ my $momentum;
+ my $weight_master_copy;
+ my $do_multi_precision = ($self->multi_precision and $weight->dtype eq 'float16');
+ if($do_multi_precision)
+ {
+ if($self->momentum != 0)
+ {
+ $momentum = mx->nd->zeros($weight->shape, ctx => $weight->context, dtype=>'float32');
+ }
+ $weight_master_copy = mx->nd->array($weight, ctx=>$weight->context, dtype=>'float32');
+ return [$weight_master_copy, $momentum];
+ }
+ else
+ {
+ if($self->momentum != 0)
+ {
+ $momentum = mx->nd->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype);
+ }
+ return $momentum;
+ }
+}
+
+method update($index, $weight, $grad, $state)
+{
+ my $lr = $self->_get_lr($index);
+ my $wd = $self->_get_wd($index);
+ $self->_update_count($index);
+ my $use_multi_precision = (defined $state and not Scalar::Util::blessed($state) and ref($state eq 'ARRAY'));
+ if(not $use_multi_precision)
+ {
+ $grad *= $self->rescale_grad;
+ if(defined $self->clip_gradient)
+ {
+ $grad = mx->nd->clip($grad, -$self->clip_gradient, $self->clip_gradient);
+ }
+ if($self->momentum == 0)
+ {
+ $weight += -$lr * ($grad + $wd * $weight);
+ }
+ else
+ {
+ my $mom = $state;
+ $mom *= $self->momentum;
+ $grad += $wd * $weight;
+ $mom += $grad;
+ $grad += $self->momentum * $mom;
+ $weight += -$lr * $grad;
+ }
+ }
+ else
+ {
+ my $grad32 = mx->nd->array($grad, ctx=>$grad->context, dtype=>'float32');
+ $grad32 *= $self->rescale_grad;
+ if(defined $self->clip_gradient)
+ {
+ $grad32 = mx->nd->clip($grad32, -$self->clip_gradient, $self->clip_gradient);
+ }
+ my $mom = $state->[1];
+ my $weight32 = $state->[0];
+ if($self->momentum == 0)
+ {
+ $weight32 += -$lr * ($grad32 + $wd * $weight32);
+ }
+ else
+ {
+ $mom *= $self->momentum;
+ $grad32 += $wd * $weight32;
+ $mom += $grad32;
+ $grad32 += $self->momentum * $mom;
+ $weight32 += -$lr * $grad32;
+ }
+ my $tmp = $weight32->astype($weight->dtype);
+ $tmp->copyto($weight);
+ }
+}
+
+package PerlFTML;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+has 'beta1' => (is => 'rw', default => 0.6);
+has 'beta2' => (is => 'rw', default => 0.999);
+has 'epsilon' => (is => 'rw', default => 1e-8);
+
+method create_state($index, $weight)
+{
+ return [mx->nd->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # d_0
+ mx->nd->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # v_0
+ mx->nd->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype)] # z_0
+}
+
+method update($index, $weight, $grad, $state)
+{
+ $self->_update_count($index);
+ my $lr = $self->_get_lr($index);
+ my $wd = $self->_get_wd($index);
+ my $t = $self->_index_update_count->{$index};
+
+ my $grad = $grad * $self->rescale_grad + $wd * $weight;
+ if(defined $self->clip_gradient)
+ {
+ $grad = mx->nd->clip($grad, -$self->clip_gradient, $self->clip_gradient);
+ }
+ # get previous states
+ my ($prev_d, $prev_v, $prev_z) = @{ $state };
+ # compute states
+ my $v_t = $self->beta2 * $prev_v + (1 - $self->beta2) * mx->nd->square($grad);
+ my $d_t = (1 - ($self->beta1**$t)) / $lr * (mx->nd->sqrt($v_t / (1 - ($self->beta2**$t))) + $self->epsilon);
+ my $sigma_t = $d_t - $self->beta1 * $prev_d;
+ my $z_t = $self->beta1 * $prev_z + (1 - $self->beta1) * $grad - $sigma_t * $weight;
+ # update weight
+ $weight .= - $z_t / $d_t;
+ # update states
+ $prev_d .= $d_t;
+ $prev_v .= $v_t;
+ $prev_z .= $z_t;
+}
+
+package PerlSignum;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+has 'wd_lh' => (is => 'rw', default => 0);
+has 'momentum' => (is => 'rw', default => 0.9);
+
+method create_state($index, $weight)
+{
+ if($self->momentum != 0)
+ {
+ return mx->nd->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype, stype=>$weight->stype);
+ }
+ return undef;
+}
+
+method update($index, $weight, $grad, $state)
+{
+ $self->_update_count($index);
+ my $lr = $self->_get_lr($index);
+ my $wd = $self->_get_wd($index);
+ if(defined $state)
+ {
+ my $mom = $state;
+ if(defined $self->clip_gradient)
+ {
+ $mom .= ($self->momentum*$mom - (1-$self->momentum)*($wd*$weight +
+ mx->nd->clip($grad*$self->rescale_grad, -$self->clip_gradient, $self->clip_gradient)));
+ }
+ else
+ {
+ $mom .= $self->momentum*$mom - (1-$self->momentum)*$wd*$weight - (1-$self->momentum)*$self->rescale_grad*$grad;
+ }
+ $weight .= (1 - $lr*$self->wd_lh)*$weight + $lr * mx->nd->sign($mom);
+ }
+ else
+ {
+ $weight .= (1 - $lr*($wd+$self->wd_lh))*$weight - $lr * mx->nd->sign($grad);
+ }
+}
+
+package PerlFtrl;
+use Mouse;
+use AI::MXNet::TestUtils qw(almost_equal);
+extends 'AI::MXNet::Optimizer';
+
+has 'lamda1' => (is => 'rw', default => 0.01);
+has '+learning_rate' => (default => 0.1);
+has 'beta' => (is => 'rw', default => 1);
+has 'sparse_update' => (is => 'rw', default => 0);
+
+method create_state($index, $weight)
+{
+ return [
+ mx->nd->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype), # dn
+ mx->nd->zeros($weight->shape, ctx => $weight->context, dtype=>$weight->dtype) # n
+ ];
+}
+
+method update($index, $weight, $grad, $state)
+{
+ $self->_update_count($index);
+ my $wd = $self->_get_wd($index);
+ my $lr = $self->_get_lr($index);
+ my $num_rows = $weight->shape->[0];
+
+ my ($dn, $n) = @$state;
+ for my $row (0..$num_rows-1)
+ {
+ my $grad_row = $grad->at($row);
+ my $all_zeros = almost_equal($grad_row->aspdl, mx->nd->zeros($grad_row->shape, ctx => $grad_row->context, dtype => $grad_row->dtype)->aspdl);
+ if($all_zeros and $self->sparse_update)
+ {
+ next;
+ }
+ $grad_row *= $self->rescale_grad;
+ if(defined $self->clip_gradient)
+ {
+ $grad_row .= mx->nd->clip($grad_row, -$self->clip_gradient, $self->clip_gradient);
+ }
+
+ #update dn, n
+ $dn->at($row) += $grad_row - (mx->nd->sqrt($n->at($row) + $grad_row * $grad_row) - mx->nd->sqrt($n->at($row))) * $weight->at($row) / $lr;
+ $n->at($row) += $grad_row * $grad_row;
+
+ # update weight
+ $weight->at($row) .= (mx->nd->sign($dn->at($row)) * $self->lamda1 - $dn->at($row)) /
+ (($self->beta + mx->nd->sqrt($n->at($row))) / $lr + $wd) * (mx->nd->abs($dn->at($row)) > $self->lamda1);
+ }
+}
+
+package PerlAdaGrad;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'eps' => (is => 'rw', default => 1e-7);
+method create_state($index, $weight)
+{
+ mx->nd->zeros($weight->shape, ctx => $weight->context, stype => $weight->stype);
+}
+
+method update($index, $weight, $grad, $state)
+{
+ $self->_update_count($index);
+ my $wd = $self->_get_wd($index);
+ my $lr = $self->_get_lr($index);
+ my $num_rows = $weight->shape->[0];
+ my $history = $state;
+ $grad *= $self->rescale_grad;
+ if(defined $self->clip_gradient)
+ {
+ $grad = mx->nd->clip($grad, -$self->clip_gradient, $self->clip_gradient);
+ }
+ $history += mx->nd->square($grad);
+ my $div = $grad / mx->nd->sqrt($history + $self->eps);
+ $weight += ($div + $weight * $wd) * -$lr;
+}
package main;
-use Test::More tests => 1314;
+use Carp;
+use Test::More tests => 7884;
use AI::MXNet::Base;
use PDL::NiceSlice;
-use AI::MXNet::TestUtils qw(same reldiff almost_equal);
+use AI::MXNet::TestUtils qw(same reldiff almost_equal rand_ndarray);
use AI::MXNet::Function::Parameters;
-func compare_optimizer($opt1, $opt2, $shape, $dtype)
+func compare_optimizer($opt1, $opt2, $shape, $dtype, $w_stype='default', $g_stype='default')
{
- my $w1 = mx->random->uniform({shape => $shape, dtype=>$dtype});
- my $g1 = mx->random->uniform({shape => $shape, dtype=>$dtype});
-
- my $w2 = $w1->copyto(mx->cpu());
- my $g2 = $g1->copyto(mx->cpu());
+ my ($w1, $w2, $g1, $g2);
+ if($w_stype eq 'default')
+ {
+ $w1 = mx->random->uniform({shape => $shape, ctx => mx->cpu, dtype=>$dtype});
+ $w2 = $w1->copyto(mx->cpu());
+ }
+ elsif($w_stype eq 'row_sparse' or $w_stype eq 'csr')
+ {
+ $w2 = rand_ndarray($shape, $w_stype, 1, $dtype);
+ $w1 = $w2->copyto(mx->cpu())->tostype('default');
+ }
+ else
+ {
+ Carp::confess("type not supported yet");
+ }
+ if($g_stype eq 'default')
+ {
+ $g2 = mx->random->uniform(shape=>$shape, ctx=>mx->cpu, dtype=>$dtype);
+ $g1 = $g2->copyto(mx->cpu);
+ }
+ elsif($g_stype eq 'row_sparse' or $g_stype eq 'csr')
+ {
+ $g2 = rand_ndarray($shape, $g_stype, rand(), $dtype);
+ $g1 = $g2->copyto(mx->cpu)->tostype('default');
+ }
+ else
+ {
+ Carp::confess("type not supported yet");
+ }
my $state1 = $opt1->create_state(0, $w1);
my $state2 = $opt2->create_state(0, $w2);
@@ -406,7 +753,6 @@ func test_rms()
}
}
-
sub test_sgd
{
mx->random->seed(0);
@@ -450,6 +796,202 @@ sub test_sgd
}
}
+sub test_sparse_sgd
+{
+ mx->random->seed(0);
+ my $opt1 = 'PerlSparseSGD';
+ my $opt2 = mx->optimizer->SGD;
+ my $shape = [3, 4, 5];
+ my @mom_options = ({}, {momentum => 0.9});
+ my @cg_options = ({}, {clip_gradient => 0.4}, {clip_gradient => 0.5});
+ my @rg_options = ({}, {rescale_grad => 0.14}, {rescale_grad => 0.8});
+ my @wd_options = ({}, {wd => 0.03}, {wd => 0.05}, {wd => 0.07});
+ my @mp_options = ({}, {multi_precision => 0}, {multi_precision => 1});
+ for my $dtype(qw/float32/)
+ {
+ for my $mom_option (@mom_options)
+ {
+ for my $cg_option (@cg_options)
+ {
+ for my $rg_option (@rg_options)
+ {
+ for my $wd_option (@wd_options)
+ {
+ for my $mp_option (@mp_options)
+ {
+ my %kwarg;
+ %kwarg = (%kwarg, %$mom_option);
+ %kwarg = (%kwarg, %$cg_option);
+ %kwarg = (%kwarg, %$rg_option);
+ %kwarg = (%kwarg, %$wd_option);
+ %kwarg = (%kwarg, %$mp_option);
+ compare_optimizer($opt1->new(%kwarg), $opt2->new(%kwarg), $shape, $dtype, 'row_sparse', 'row_sparse');
+ }
+ }
+ }
+ }
+ }
+ }
+}
+
+sub test_std_sparse_sgd
+{
+ mx->random->seed(0);
+ my $opt1 = 'PerlSparseSGD';
+ my $opt2 = mx->optimizer->SGD;
+ my $shape = [3, 4, 5];
+ my @mom_options = ({momentum => 0.9});
+ my @cg_options = ({}, {clip_gradient => 0.4}, {clip_gradient => 0.5});
+ my @rg_options = ({}, {rescale_grad => 0.14}, {rescale_grad => 0.8});
+ my @wd_options = ({}, {wd => 0.03}, {wd => 0.05}, {wd => 0.07});
+ for my $dtype(qw/float32/)
+ {
+ for my $mom_option (@mom_options)
+ {
+ for my $cg_option (@cg_options)
+ {
+ for my $rg_option (@rg_options)
+ {
+ for my $wd_option (@wd_options)
+ {
+ my %kwarg;
+ %kwarg = (%kwarg, %$mom_option);
+ %kwarg = (%kwarg, %$cg_option);
+ %kwarg = (%kwarg, %$rg_option);
+ %kwarg = (%kwarg, %$wd_option);
+ compare_optimizer($opt1->new(lazy_update => 0, %kwarg), $opt2->new(lazy_update => 0, %kwarg), $shape, $dtype, 'row_sparse', 'row_sparse');
+ }
+ }
+ }
+ }
+ }
+}
+
+sub test_nag
+{
+ mx->random->seed(0);
+ my $opt1 = 'PerlNAG';
+ my $opt2 = mx->optimizer->NAG;
+ my $shape = [3, 4, 5];
+ my @mom_options = ({}, {momentum => 0.9});
+ my @cg_options = ({}, {clip_gradient => 0.4}, {clip_gradient => 0.5});
+ my @rg_options = ({}, {rescale_grad => 0.14}, {rescale_grad => 0.8});
+ my @wd_options = ({}, {wd => 0.03}, {wd => 0.05}, {wd => 0.07});
+ my @mp_options = ({}, {multi_precision => 0}, {multi_precision => 1});
+ for my $dtype(qw/float16 float32 float64/)
+ {
+ for my $mom_option (@mom_options)
+ {
+ for my $cg_option (@cg_options)
+ {
+ for my $rg_option (@rg_options)
+ {
+ for my $wd_option (@wd_options)
+ {
+ for my $mp_option (@mp_options)
+ {
+ my %kwarg;
+ %kwarg = (%kwarg, %$mom_option);
+ %kwarg = (%kwarg, %$cg_option);
+ %kwarg = (%kwarg, %$rg_option);
+ %kwarg = (%kwarg, %$wd_option);
+ # %kwarg = (%kwarg, %$mp_option);
+ next if (
+ $dtype eq 'float16'
+ and
+ (not exists $kwarg{multi_precision} or not $kwarg{multi_precision})
+ );
+ compare_optimizer($opt1->new(%kwarg), $opt2->new(%kwarg), $shape, $dtype);
+ }
+ }
+ }
+ }
+ }
+ }
+}
+
+sub test_ftml
+{
+ mx->random->seed(0);
+ my $opt1 = 'PerlFTML';
+ my $opt2 = mx->optimizer->FTML;
+ my $shape = [3, 4, 5];
+ my @beta1_options = ({}, {beta1 => 0.5}, {beta1 => 0.7});
+ my @beta2_options = ({}, {beta1 => 0.8}, {beta1 => 0.9});
+ my @cg_options = ({}, {clip_gradient => 0.4}, {clip_gradient => 0.5});
+ my @rg_options = ({}, {rescale_grad => 0.14}, {rescale_grad => 0.8});
+ my @wd_options = ({}, {wd => 0.03}, {wd => 0.05}, {wd => 0.07});
+ for my $dtype(qw/float32/)
+ {
+ for my $beta1_option (@beta1_options)
+ {
+ for my $beta2_option (@beta2_options)
+ {
+ for my $rg_option (@rg_options)
+ {
+ for my $wd_option (@wd_options)
+ {
+ for my $cg_option (@cg_options)
+ {
+ my %kwarg;
+ %kwarg = (%kwarg, %$beta1_option);
+ %kwarg = (%kwarg, %$beta2_option);
+ %kwarg = (%kwarg, %$cg_option);
+ %kwarg = (%kwarg, %$rg_option);
+ %kwarg = (%kwarg, %$wd_option);
+ compare_optimizer($opt1->new(%kwarg), $opt2->new(%kwarg), $shape, $dtype);
+ }
+ }
+ }
+ }
+ }
+ }
+}
+
+sub test_signum
+{
+ mx->random->seed(0);
+ my $opt1 = 'PerlSignum';
+ my $opt2 = mx->optimizer->Signum;
+ my $shape = [3, 4, 5];
+ my @cg_options = ({}, {clip_gradient => 0.4}, {clip_gradient => 0.5});
+ my @rg_options = ({}, {rescale_grad => 0.14}, {rescale_grad => 0.8});
+ my @wd_options = ({}, {wd => 0.03}, {wd => 0.05}, {wd => 0.07});
+ my @wd_lh_options = ({}, {wd_lh => 0.015}, {wd_lh => 0.0});
+ my @mom_options = ({}, {momentum => 0.9});
+ my @lr_options = ({learning_rate => 0.05}, {learning_rate => 0.01});
+ for my $dtype (qw/float32 float64/)
+ {
+ for my $wd_lh_option (@wd_lh_options)
+ {
+ for my $mom_option (@mom_options)
+ {
+ for my $rg_option (@rg_options)
+ {
+ for my $wd_option (@wd_options)
+ {
+ for my $cg_option (@cg_options)
+ {
+ for my $lr_option (@lr_options)
+ {
+ my %kwarg;
+ %kwarg = (%kwarg, %$wd_lh_option);
+ %kwarg = (%kwarg, %$mom_option);
+ %kwarg = (%kwarg, %$lr_option);
+ %kwarg = (%kwarg, %$cg_option);
+ %kwarg = (%kwarg, %$rg_option);
+ %kwarg = (%kwarg, %$wd_option);
+ compare_optimizer($opt1->new(%kwarg), $opt2->new(%kwarg), $shape, $dtype);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+}
+
+
func test_lr_wd_mult()
{
my $data = mx->sym->Variable('data');
@@ -481,7 +1023,76 @@ func test_lr_wd_mult()
ok(!almost_equal($args1{fc2_weight}, $args2{fc2_weight}, 1e-1), "fc2_weight");
}
+sub test_ftrl
+{
+ mx->random->seed(0);
+ my $opt1 = 'PerlFtrl';
+ my $opt2 = mx->optimizer->Ftrl;
+ my $shape = [3, 4, 5];
+ my @kwargs = ({},
+ {clip_gradient => 0.5},
+ {clip_gradient => 0.4, rescale_grad => 0.14},
+ {rescale_grad => 0.8},
+ {clip_gradient => 0.5, wd => 0.07},
+ {clip_gradient => 0.4, rescale_grad => 0.14, wd => 0.03},
+ {rescale_grad => 0.8, wd => 0.05},
+ {rescale_grad => 0.8, wd => 0.05, lamda1 => 0.01},
+ {clip_gradient => 0.5, wd => 0.07, lamda1 => 1.0});
+ for my $kwarg (@kwargs)
+ {
+ compare_optimizer($opt1->new(%$kwarg), $opt2->new(%$kwarg), $shape, 'float32');
+ compare_optimizer($opt1->new(sparse_update=>1, %$kwarg), $opt2->new(%$kwarg), $shape,
+ 'float32', 'row_sparse', 'row_sparse');
+ }
+}
+
+sub test_adagrad
+{
+ mx->random->seed(0);
+ my $opt1 = 'PerlAdaGrad';
+ my $opt2 = mx->optimizer->AdaGrad;
+ my $shape = [3, 4, 5];
+ my @eps_options= ({}, {eps => 1e-9});
+ my @cg_options = ({}, {clip_gradient => 0.4}, {clip_gradient => 0.5});
+ my @rg_options = ({}, {rescale_grad => 0.14}, {rescale_grad => 0.8});
+ my @wd_options = ({}, {wd => 0});
+ for my $dtype(qw/float32/)
+ {
+ for my $eps_option (@eps_options)
+ {
+ for my $cg_option (@cg_options)
+ {
+ for my $rg_option (@rg_options)
+ {
+ for my $wd_option (@wd_options)
+ {
+ my %kwarg;
+ %kwarg = (%kwarg, %$eps_option);
+ %kwarg = (%kwarg, %$cg_option);
+ %kwarg = (%kwarg, %$rg_option);
+ %kwarg = (%kwarg, %$wd_option);
+ compare_optimizer($opt1->new(%kwarg), $opt2->new(%kwarg), $shape, $dtype);
+ if(($wd_option->{wd}//0) == 0)
+ {
+ compare_optimizer($opt1->new(%kwarg), $opt2->new(%kwarg), $shape, $dtype, 'row_sparse', 'row_sparse');
+ }
+ }
+ }
+ }
+ }
+ }
+}
+
test_adam();
test_rms();
test_sgd();
+test_std_sparse_sgd();
+test_sparse_sgd();
+test_nag();
+test_ftml();
+test_signum();
+test_ftrl();
+test_adagrad();
test_lr_wd_mult();
+
+
diff --git a/perl-package/AI-MXNet/t/test_sparse_ndarray.t b/perl-package/AI-MXNet/t/test_sparse_ndarray.t
new file mode 100644
index 0000000..f143346
--- /dev/null
+++ b/perl-package/AI-MXNet/t/test_sparse_ndarray.t
@@ -0,0 +1,1005 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+use strict;
+use warnings;
+use Scalar::Util qw(blessed);
+use Test::More 'no_plan';
+use AI::MXNet qw(mx);
+use AI::MXNet::TestUtils qw(zip assert enumerate same rand_shape_2d rand_shape_3d
+ rand_sparse_ndarray random_arrays almost_equal rand_ndarray randint allclose dies_ok);
+use AI::MXNet::Base qw(pones pzeros pdl product rand_sparse);
+$ENV{MXNET_STORAGE_FALLBACK_LOG_VERBOSE} = 0;
+
+
+sub sparse_nd_ones
+{
+ my ($shape, $stype) = @_;
+ return mx->nd->ones($shape)->tostype($stype);
+}
+
+sub test_sparse_nd_elemwise_add
+{
+ my $check_sparse_nd_elemwise_binary = sub {
+ my ($shapes, $stypes, $f, $g) = @_;
+ # generate inputs
+ my @nds;
+ enumerate(sub {
+ my ($i, $stype) = @_;
+ my $nd;
+ if($stype eq 'row_sparse')
+ {
+ ($nd) = rand_sparse_ndarray($shapes->[$i], $stype);
+ }
+ elsif($stype eq 'default')
+ {
+ $nd = mx->nd->array(random_arrays($shapes->[$i]), dtype => 'float32');
+ }
+ else
+ {
+ die;
+ }
+ push @nds, $nd;
+ }, $stypes);
+ # check result
+ my $test = $f->($nds[0], $nds[1]);
+ ok(almost_equal($test->aspdl, $g->($nds[0]->aspdl, $nds[1]->aspdl)));
+ };
+ my $num_repeats = 2;
+ my $g = sub { $_[0] + $_[1] };
+ my $op = sub { mx->nd->elemwise_add(@_) };
+ for my $i (0..$num_repeats)
+ {
+ my $shape = rand_shape_2d();
+ $shape = [$shape, $shape];
+ $check_sparse_nd_elemwise_binary->($shape, ['default', 'default'], $op, $g);
+ $check_sparse_nd_elemwise_binary->($shape, ['row_sparse', 'row_sparse'], $op, $g);
+ }
+}
+
+test_sparse_nd_elemwise_add();
+
+sub test_sparse_nd_copy
+{
+ my $check_sparse_nd_copy = sub { my ($from_stype, $to_stype, $shape) = @_;
+ my $from_nd = rand_ndarray($shape, $from_stype);
+ # copy to ctx
+ my $to_ctx = $from_nd->copyto(AI::MXNet::Context->current_ctx);
+ # copy to stype
+ my $to_nd = rand_ndarray($shape, $to_stype);
+ $from_nd->copyto($to_nd);
+ ok(($from_nd->aspdl != $to_ctx->aspdl)->abs->sum == 0);
+ ok(($from_nd->aspdl != $to_nd->aspdl)->abs->sum == 0);
+ };
+ my $shape = rand_shape_2d();
+ my $shape_3d = rand_shape_3d();
+ my @stypes = ('row_sparse', 'csr');
+ for my $stype (@stypes)
+ {
+ $check_sparse_nd_copy->($stype, 'default', $shape);
+ $check_sparse_nd_copy->('default', $stype, $shape);
+ }
+ $check_sparse_nd_copy->('row_sparse', 'row_sparse', $shape_3d);
+ $check_sparse_nd_copy->('row_sparse', 'default', $shape_3d);
+ $check_sparse_nd_copy->('default', 'row_sparse', $shape_3d);
+}
+
+test_sparse_nd_copy();
+
+sub test_sparse_nd_basic
+{
+ my $check_sparse_nd_basic_rsp = sub {
+ my $storage_type = 'row_sparse';
+ my $shape = rand_shape_2d();
+ my ($nd) = rand_sparse_ndarray($shape, $storage_type);
+ ok($nd->_num_aux == 1);
+ ok($nd->indices->dtype eq 'int64');
+ ok($nd->stype eq 'row_sparse');
+ };
+ $check_sparse_nd_basic_rsp->();
+}
+
+test_sparse_nd_basic();
+
+sub test_sparse_nd_setitem
+{
+ my $check_sparse_nd_setitem = sub { my ($stype, $shape, $dst) = @_;
+ my $x = mx->nd->zeros($shape, stype=>$stype);
+ $x .= $dst;
+ my $dst_nd = (blessed $dst and $dst->isa('PDL')) ? mx->nd->array($dst) : $dst;
+ ok(($x->aspdl == (ref $dst_nd ? $dst_nd->aspdl : $dst_nd))->all);
+ };
+
+ my $shape = rand_shape_2d();
+ for my $stype ('row_sparse', 'csr')
+ {
+ # ndarray assignment
+ $check_sparse_nd_setitem->($stype, $shape, rand_ndarray($shape, 'default'));
+ $check_sparse_nd_setitem->($stype, $shape, rand_ndarray($shape, $stype));
+ # numpy assignment
+ $check_sparse_nd_setitem->($stype, $shape, pones(reverse @{ $shape }));
+ }
+ # scalar assigned to row_sparse NDArray
+ $check_sparse_nd_setitem->('row_sparse', $shape, 2);
+}
+
+test_sparse_nd_setitem();
+
+sub test_sparse_nd_slice
+{
+ my $shape = [randint(2, 10), randint(2, 10)];
+ my $stype = 'csr';
+ my ($A) = rand_sparse_ndarray($shape, $stype);
+ my $A2 = $A->aspdl;
+ my $start = randint(0, $shape->[0] - 1);
+ my $end = randint($start + 1, $shape->[0]);
+ ok(same($A->slice([$start, $end])->aspdl, $A2->slice('X', [$start, $end])));
+ ok(same($A->slice([$start - $shape->[0], $end])->aspdl, $A2->slice('X', [$start, $end])));
+ ok(same($A->slice([$start, $shape->[0] - 1])->aspdl, $A2->slice('X', [$start, $shape->[0]-1])));
+ ok(same($A->slice([0, $end])->aspdl, $A2->slice('X', [0, $end])));
+
+ my $start_col = randint(0, $shape->[1] - 1);
+ my $end_col = randint($start_col + 1, $shape->[1]);
+ my $result = $A->slice(begin=>[$start, $start_col], end=>[$end, $end_col]);
+ my $result_dense = mx->nd->array($A2)->slice(begin=>[$start, $start_col], end=>[$end, $end_col]);
+ ok(same($result_dense->aspdl, $result->aspdl));
+
+ $A = mx->nd->sparse->zeros('csr', $shape);
+ $A2 = $A->aspdl;
+ ok(same($A->slice([$start, $end])->aspdl, $A2->slice('X', [$start, $end])));
+ $result = $A->slice(begin=>[$start, $start_col], end=>[$end, $end_col]);
+ $result_dense = mx->nd->array($A2)->slice(begin=>[$start, $start_col], end=>[$end, $end_col]);
+ ok(same($result_dense->aspdl, $result->aspdl));
+
+ my $check_slice_nd_csr_fallback = sub { my ($shape) = @_;
+ my $stype = 'csr';
+ my ($A) = rand_sparse_ndarray($shape, $stype);
+ my $A2 = $A->aspdl;
+ my $start = randint(0, $shape->[0] - 1);
+ my $end = randint($start + 1, $shape->[0]);
+
+ # non-trivial step should fallback to dense slice op
+ my $result = $A->slice(begin=>[$start], end=>[$end+1], step=>[2]);
+ my $result_dense = mx->nd->array($A2)->slice(begin=>[$start], end=>[$end + 1], step=>[2]);
+ ok(same($result_dense->aspdl, $result->aspdl));
+ };
+ $shape = [randint(2, 10), randint(1, 10)];
+ $check_slice_nd_csr_fallback->($shape);
+}
+
+test_sparse_nd_slice();
+
+sub test_sparse_nd_equal
+{
+ for my $stype ('row_sparse', 'csr')
+ {
+ my $shape = rand_shape_2d();
+ my $x = mx->nd->zeros($shape, stype=>$stype);
+ my $y = sparse_nd_ones($shape, $stype);
+ my $z = $x == $y;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ $z = 0 == $x;
+ ok(($z->aspdl == pones(reverse @{ $shape }))->all);
+ }
+}
+
+test_sparse_nd_equal();
+
+sub test_sparse_nd_not_equal
+{
+ for my $stype ('row_sparse', 'csr')
+ {
+ my $shape = rand_shape_2d();
+ my $x = mx->nd->zeros($shape, stype=>$stype);
+ my $y = sparse_nd_ones($shape, $stype);
+ my $z = $x != $y;
+ ok(($z->aspdl == pones(reverse @{ $shape }))->all);
+ $z = 0 != $x;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ }
+}
+
+test_sparse_nd_not_equal();
+
+sub test_sparse_nd_greater
+{
+ for my $stype ('row_sparse', 'csr')
+ {
+ my $shape = rand_shape_2d();
+ my $x = mx->nd->zeros($shape, stype=>$stype);
+ my $y = sparse_nd_ones($shape, $stype);
+ my $z = $x > $y;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ $z = $y > 0;
+ ok(($z->aspdl == pones(reverse @{ $shape }))->all);
+ $z = 0 > $y;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ }
+}
+
+test_sparse_nd_greater();
+
+sub test_sparse_nd_greater_equal
+{
+ for my $stype ('row_sparse', 'csr')
+ {
+ my $shape = rand_shape_2d();
+ my $x = mx->nd->zeros($shape, stype=>$stype);
+ my $y = sparse_nd_ones($shape, $stype);
+ my $z = $x >= $y;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ $z = $y >= 0;
+ ok(($z->aspdl == pones(reverse @{ $shape }))->all);
+ $z = 0 >= $y;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ $z = $y >= 1;
+ ok(($z->aspdl == pones(reverse @{ $shape }))->all);
+ }
+}
+
+test_sparse_nd_greater_equal();
+
+sub test_sparse_nd_lesser
+{
+ for my $stype ('row_sparse', 'csr')
+ {
+ my $shape = rand_shape_2d();
+ my $x = mx->nd->zeros($shape, stype=>$stype);
+ my $y = sparse_nd_ones($shape, $stype);
+ my $z = $y < $x;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ $z = 0 < $y;
+ ok(($z->aspdl == pones(reverse @{ $shape }))->all);
+ $z = $y < 0;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ }
+}
+
+test_sparse_nd_lesser();
+
+sub test_sparse_nd_lesser_equal
+{
+ for my $stype ('row_sparse', 'csr')
+ {
+ my $shape = rand_shape_2d();
+ my $x = mx->nd->zeros($shape, stype=>$stype);
+ my $y = sparse_nd_ones($shape, $stype);
+ my $z = $y <= $x;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ $z = 0 <= $y;
+ ok(($z->aspdl == pones(reverse @{ $shape }))->all);
+ $z = $y <= 0;
+ ok(($z->aspdl == pzeros(reverse @{ $shape }))->all);
+ $z = 1 <= $y;
+ ok(($z->aspdl == pones(reverse @{ $shape }))->all);
+ }
+}
+
+test_sparse_nd_lesser_equal();
+
+sub test_sparse_nd_binary
+{
+ my $N = 2;
+ my $check_binary = sub { my ($fn, $stype) = @_;
+ for (0 .. 2)
+ {
+ my $ndim = 2;
+ my $oshape = [map { randint(1, 6) } 1..$ndim];
+ my $bdim = 2;
+ my @lshape = @$oshape;
+ # one for broadcast op, another for elemwise op
+ my @rshape = @lshape[($ndim-$bdim)..@lshape-1];
+ for my $i (0..$bdim-1)
+ {
+ my $sep = mx->nd->random->uniform(0, 1)->asscalar;
+ if($sep < 0.33)
+ {
+ $lshape[$ndim-$i-1] = 1;
+ }
+ elsif($sep < 0.66)
+ {
+ $rshape[$bdim-$i-1] = 1;
+ }
+ }
+ my $lhs = mx->nd->random->uniform(0, 1, shape=>\@lshape)->aspdl;
+ my $rhs = mx->nd->random->uniform(0, 1, shape=>\@rshape)->aspdl;
+ my $lhs_nd = mx->nd->array($lhs)->tostype($stype);
+ my $rhs_nd = mx->nd->array($rhs)->tostype($stype);
+ ok(allclose($fn->($lhs, $rhs), $fn->($lhs_nd, $rhs_nd)->aspdl, 1e-4));
+ }
+ };
+ for my $stype ('row_sparse', 'csr')
+ {
+ $check_binary->(sub { $_[0] + $_[1] }, $stype);
+ $check_binary->(sub { $_[0] - $_[1] }, $stype);
+ $check_binary->(sub { $_[0] * $_[1] }, $stype);
+ $check_binary->(sub { $_[0] / $_[1] }, $stype);
+ $check_binary->(sub { $_[0] ** $_[1] }, $stype);
+ $check_binary->(sub { $_[0] > $_[1] }, $stype);
+ $check_binary->(sub { $_[0] < $_[1] }, $stype);
+ $check_binary->(sub { $_[0] >= $_[1] }, $stype);
+ $check_binary->(sub { $_[0] <= $_[1] }, $stype);
+ $check_binary->(sub { $_[0] == $_[1] }, $stype);
+ }
+}
+
+test_sparse_nd_binary();
+
+sub test_sparse_nd_binary_scalar_op
+{
+ my $N = 3;
+ my $check = sub { my ($fn, $stype) = @_;
+ for (1..$N)
+ {
+ my $ndim = 2;
+ my $shape = [map { randint(1, 6) } 1..$ndim];
+ my $npy = mx->nd->random->normal(0, 1, shape=>$shape)->aspdl;
+ my $nd = mx->nd->array($npy)->tostype($stype);
+ ok(allclose($fn->($npy), $fn->($nd)->aspdl, 1e-4));
+ }
+ };
+ for my $stype ('row_sparse', 'csr')
+ {
+ $check->(sub { 1 + $_[0] }, $stype);
+ $check->(sub { 1 - $_[0] }, $stype);
+ $check->(sub { 1 * $_[0] }, $stype);
+ $check->(sub { 1 / $_[0] }, $stype);
+ $check->(sub { 2 ** $_[0] }, $stype);
+ $check->(sub { 1 > $_[0] }, $stype);
+ $check->(sub { 0.5 > $_[0] }, $stype);
+ $check->(sub { 0.5 < $_[0] }, $stype);
+ $check->(sub { 0.5 >= $_[0] }, $stype);
+ $check->(sub { 0.5 <= $_[0] }, $stype);
+ $check->(sub { 0.5 == $_[0] }, $stype);
+ $check->(sub { $_[0] / 2 }, $stype);
+ }
+}
+
+test_sparse_nd_binary_scalar_op();
+
+sub test_sparse_nd_binary_iop
+{
+ my $N = 3;
+ my $check_binary = sub { my ($fn, $stype) = @_;
+ for (1..$N)
+ {
+ my $ndim = 2;
+ my $oshape = [map { randint(1, 6) } 1..$ndim];
+ my $lhs = mx->nd->random->uniform(0, 1, shape => $oshape)->aspdl;
+ my $rhs = mx->nd->random->uniform(0, 1, shape => $oshape)->aspdl;
+ my $lhs_nd = mx->nd->array($lhs)->tostype($stype);
+ my $rhs_nd = mx->nd->array($rhs)->tostype($stype);
+ ok(
+ allclose(
+ $fn->($lhs, $rhs),
+ $fn->($lhs_nd, $rhs_nd)->aspdl,
+ 1e-4
+ )
+ );
+ }
+ };
+
+ my $inplace_add = sub { my ($x, $y) = @_;
+ $x += $y;
+ return $x
+ };
+ my $inplace_mul = sub { my ($x, $y) = @_;
+ $x *= $y;
+ return $x
+ };
+ my @stypes = ('csr', 'row_sparse');
+ my @fns = ($inplace_add, $inplace_mul);
+ for my $stype (@stypes)
+ {
+ for my $fn (@fns)
+ {
+ $check_binary->($fn, $stype);
+ }
+ }
+}
+
+test_sparse_nd_binary_iop();
+
+sub test_sparse_nd_negate
+{
+ my $check_sparse_nd_negate = sub { my ($shape, $stype) = @_;
+ my $npy = mx->nd->random->uniform(-10, 10, shape => rand_shape_2d())->aspdl;
+ my $arr = mx->nd->array($npy)->tostype($stype);
+ ok(almost_equal($npy, $arr->aspdl));
+ ok(almost_equal(-$npy, (-$arr)->aspdl));
+
+ # a final check to make sure the negation (-) is not implemented
+ # as inplace operation, so the contents of arr does not change after
+ # we compute (-arr)
+ ok(almost_equal($npy, $arr->aspdl));
+ };
+ my $shape = rand_shape_2d();
+ my @stypes = ('csr', 'row_sparse');
+ for my $stype (@stypes)
+ {
+ $check_sparse_nd_negate->($shape, $stype);
+ }
+}
+
+test_sparse_nd_negate();
+
+sub test_sparse_nd_broadcast
+{
+ my $sample_num = 10; # TODO 1000
+ my $test_broadcast_to = sub { my ($stype) = @_;
+ for (1..$sample_num)
+ {
+ my $ndim = 2;
+ my $target_shape = [map { randint(1, 11) } 1..$ndim];
+ my $shape = \@{ $target_shape };
+ my $axis_flags = [map { randint(0, 2) } 1..$ndim];
+ my $axes = [];
+ enumerate(sub {
+ my ($axis, $flag) = @_;
+ if($flag)
+ {
+ $shape->[$axis] = 1;
+ }
+ }, $axis_flags);
+ my $dat = mx->nd->random->uniform(0, 1, shape => $shape)->aspdl - 0.5;
+ my $pdl_ret = $dat;
+ my $ndarray = mx->nd->array($dat)->tostype($stype);
+ my $ndarray_ret = $ndarray->broadcast_to($target_shape);
+ ok((pdl($ndarray_ret->shape) == pdl($target_shape))->all);
+ my $err = (($ndarray_ret->aspdl - $pdl_ret)**2)->avg;
+ ok($err < 1E-8);
+ }
+ };
+ my @stypes = ('csr', 'row_sparse');
+ for my $stype (@stypes)
+ {
+ $test_broadcast_to->($stype);
+ }
+}
+
+test_sparse_nd_broadcast();
+
+sub test_sparse_nd_transpose
+{
+ my $npy = mx->nd->random->uniform(-10, 10, shape => rand_shape_2d())->aspdl;
+ my @stypes = ('csr', 'row_sparse');
+ for my $stype (@stypes)
+ {
+ my $nd = mx->nd->array($npy)->tostype($stype);
+ ok(almost_equal($npy->transpose, ($nd->T)->aspdl));
+ }
+}
+
+test_sparse_nd_transpose();
+
+sub test_sparse_nd_storage_fallback
+{
+ my $check_output_fallback = sub { my ($shape) = @_;
+ my $ones = mx->nd->ones($shape);
+ my $out = mx->nd->zeros($shape, stype=>'csr');
+ mx->nd->broadcast_add($ones, $ones * 2, out=>$out);
+ ok(($out->aspdl - 3)->sum == 0);
+ };
+
+ my $check_input_fallback = sub { my ($shape) = @_;
+ my $ones = mx->nd->ones($shape);
+ my $out = mx->nd->broadcast_add($ones->tostype('csr'), $ones->tostype('row_sparse'));
+ ok(($out->aspdl - 2)->sum == 0);
+ };
+
+ my $check_fallback_with_temp_resource = sub { my ($shape) = @_;
+ my $ones = mx->nd->ones($shape);
+ my $out = mx->nd->sum($ones);
+ ok($out->asscalar == product(@{ $shape }));
+ };
+
+ my $shape = rand_shape_2d();
+ $check_output_fallback->($shape);
+ $check_input_fallback->($shape);
+ $check_fallback_with_temp_resource->($shape);
+}
+
+test_sparse_nd_storage_fallback();
+
+sub test_sparse_nd_astype
+{
+ my @stypes = ('row_sparse', 'csr');
+ for my $stype (@stypes)
+ {
+ my $x = mx->nd->zeros(rand_shape_2d(), stype => $stype, dtype => 'float32');
+ my $y = $x->astype('int32');
+ ok($y->dtype eq 'int32');
+ }
+}
+
+test_sparse_nd_astype();
+
+sub test_sparse_nd_storable
+{
+ my $repeat = 1;
+ my $dim0 = 40;
+ my $dim1 = 40;
+ my @stypes = ('row_sparse', 'csr');
+ my @densities = (0, 0.5);
+ my %stype = (row_sparse => 'AI::MXNet::NDArray::RowSparse', csr => 'AI::MXNet::NDArray::CSR');
+ for (1..$repeat)
+ {
+ my $shape = rand_shape_2d($dim0, $dim1);
+ for my $stype (@stypes)
+ {
+ for my $density (@densities)
+ {
+ my ($a) = rand_sparse_ndarray($shape, $stype, density => $density);
+ ok($a->isa($stype{$stype}));
+ my $data = Storable::freeze($a);
+ my $b = Storable::thaw($data);
+ ok($b->isa($stype{$stype}));
+ ok(same($a->aspdl, $b->aspdl));
+ }
+ }
+ }
+}
+
+test_sparse_nd_storable();
+
+sub test_sparse_nd_save_load
+{
+ my $repeat = 1;
+ my @stypes = ('default', 'row_sparse', 'csr');
+ my %stype = (default => 'AI::MXNet::NDArray', row_sparse => 'AI::MXNet::NDArray::RowSparse', csr => 'AI::MXNet::NDArray::CSR');
+ my $num_data = 20;
+ my @densities = (0, 0.5);
+ my $fname = 'tmp_list.bin';
+ for (1..$repeat)
+ {
+ my @data_list1;
+ for (1..$num_data)
+ {
+ my $stype = $stypes[randint(0, scalar(@stypes))];
+ my $shape = rand_shape_2d(40, 40);
+ my $density = $densities[randint(0, scalar(@densities))];
+ push @data_list1, rand_ndarray($shape, $stype, $density);
+ ok($data_list1[-1]->isa($stype{$stype}));
+ }
+ mx->nd->save($fname, \@data_list1);
+
+ my @data_list2 = @{ mx->nd->load($fname) };
+ ok(@data_list1 == @data_list2);
+ zip(sub {
+ my ($x, $y) = @_;
+ ok(same($x->aspdl, $y->aspdl));
+ }, \@data_list1, \@data_list2);
+
+ my %data_map1;
+ enumerate(sub {
+ my ($i, $x) = @_;
+ $data_map1{"ndarray xx $i"} = $x;
+ }, \@data_list1);
+ mx->nd->save($fname, \%data_map1);
+ my %data_map2 = %{ mx->nd->load($fname) };
+ ok(keys(%data_map1) == keys(%data_map2));
+ while(my ($k, $x) = each %data_map1)
+ {
+ my $y = $data_map2{$k};
+ ok(same($x->aspdl, $y->aspdl));
+ }
+ }
+ unlink $fname;
+}
+
+test_sparse_nd_save_load();
+
+sub test_create_csr
+{
+ my $check_create_csr_from_nd = sub { my ($shape, $density, $dtype) = @_;
+ my $matrix = rand_ndarray($shape, 'csr', $density);
+ # create data array with provided dtype and ctx
+ my $data = mx->nd->array($matrix->data->aspdl, dtype=>$dtype);
+ my $indptr = $matrix->indptr;
+ my $indices = $matrix->indices;
+ my $csr_created = mx->nd->sparse->csr_matrix([$data, $indices, $indptr], shape=>$shape);
+ ok($csr_created->stype eq 'csr');
+ ok(same($csr_created->data->aspdl, $data->aspdl));
+ ok(same($csr_created->indptr->aspdl, $indptr->aspdl));
+ ok(same($csr_created->indices->aspdl, $indices->aspdl));
+ # verify csr matrix dtype and ctx is consistent from the ones provided
+ ok($csr_created->dtype eq $dtype);
+ ok($csr_created->data->dtype eq $dtype);
+ ok($csr_created->context eq AI::MXNet::Context->current_ctx);
+ my $csr_copy = mx->nd->array($csr_created);
+ ok(same($csr_copy->aspdl, $csr_created->aspdl));
+ };
+
+ my $check_create_csr_from_coo = sub { my ($shape, $density, $dtype) = @_;
+ my $matrix = rand_ndarray($shape, 'csr', $density);
+ my $sp_csr = $matrix->aspdlccs;
+ my $sp_coo = $sp_csr->tocoo();
+ my $csr_created = mx->nd->sparse->csr_matrix([$sp_coo->data, [$sp_coo->row, $sp_coo->col]], shape=>$shape, dtype=>$dtype);
+ ok($csr_created->stype eq 'csr');
+ ok(same($csr_created->data->aspdl, $sp_csr->data));
+ ok(same($csr_created->indptr->aspdl, $sp_csr->indptr));
+ ok(same($csr_created->indices->aspdl, $sp_csr->indices));
+ my $csr_copy = mx->nd->array($csr_created);
+ ok(same($csr_copy->aspdl, $csr_created->aspdl));
+ # verify csr matrix dtype and ctx is consistent
+ ok($csr_created->dtype eq $dtype);
+ ok($csr_created->data->dtype eq $dtype);
+ ok($csr_created->context eq AI::MXNet::Context->current_ctx);
+ };
+
+ my $check_create_csr_from_pdlccs = sub { my ($shape, $density, $f) = @_;
+ my $assert_csr_almost_equal = sub { my ($nd, $sp) = @_;
+ ok(almost_equal($nd->data->aspdl, $sp->data));
+ ok(almost_equal($nd->indptr->aspdl, $sp->indptr));
+ ok(almost_equal($nd->indices->aspdl, $sp->indices));
+ my $sp_csr = $nd->aspdlccs;
+ ok(almost_equal($sp_csr->data, $sp->data));
+ ok(almost_equal($sp_csr->indptr, $sp->indptr));
+ ok(almost_equal($sp_csr->indices, $sp->indices));
+ ok($sp->dtype eq $sp_csr->dtype);
+ };
+
+ my $csr_sp = rand_sparse($shape->[0], $shape->[1], $density);
+ my $csr_nd = $f->($csr_sp);
+ ok(almost_equal($csr_nd->aspdl, $csr_sp->todense));
+ # non-canonical csr which contains duplicates and unsorted indices
+ my $indptr = pdl([0, 2, 3, 7]);
+ my $indices = pdl([0, 2, 2, 0, 1, 2, 1]);
+ my $data = pdl([1, 2, 3, 4, 5, 6, 1]);
+ my $non_canonical_csr = mx->nd->sparse->csr_matrix([$data, $indices, $indptr], shape=>[3, 3], dtype=>$csr_nd->dtype);
+ my $canonical_csr_nd = $f->($non_canonical_csr, dtype=>$csr_nd->dtype);
+ my $canonical_csr_sp = $non_canonical_csr->copy();
+ ok(almost_equal($canonical_csr_nd->aspdl, $canonical_csr_sp->aspdl));
+ };
+
+ my $dim0 = 20;
+ my $dim1 = 20;
+ my @densities = (0.5);
+ my $dtype = 'float64';
+ for my $density (@densities)
+ {
+ my $shape = [$dim0, $dim1];
+ $check_create_csr_from_nd->($shape, $density, $dtype);
+ $check_create_csr_from_coo->($shape, $density, $dtype);
+ $check_create_csr_from_pdlccs->($shape, $density, sub { mx->nd->sparse->array(@_) });
+ $check_create_csr_from_pdlccs->($shape, $density, sub { mx->nd->array(@_) });
+ }
+}
+
+test_create_csr();
+
+sub test_create_row_sparse
+{
+ my $dim0 = 50;
+ my $dim1 = 50;
+ my @densities = (0, 0.5, 1);
+ for my $density (@densities)
+ {
+ my $shape = rand_shape_2d($dim0, $dim1);
+ my $matrix = rand_ndarray($shape, 'row_sparse', $density);
+ my $data = $matrix->data;
+ my $indices = $matrix->indices;
+ my $rsp_created = mx->nd->sparse->row_sparse_array([$data, $indices], shape=>$shape);
+ ok($rsp_created->stype eq 'row_sparse');
+ ok(same($rsp_created->data->aspdl, $data->aspdl));
+ ok(same($rsp_created->indices->aspdl, $indices->aspdl));
+ my $rsp_copy = mx->nd->array($rsp_created);
+ ok(same($rsp_copy->aspdl, $rsp_created->aspdl));
+ }
+}
+
+test_create_row_sparse();
+
+sub test_create_sparse_nd_infer_shape
+{
+ my $check_create_csr_infer_shape = sub { my ($shape, $density, $dtype) = @_;
+ eval {
+ my $matrix = rand_ndarray($shape, 'csr', $density);
+ my $data = $matrix->data;
+ my $indptr = $matrix->indptr;
+ my $indices = $matrix->indices;
+ my $nd = mx->nd->sparse->csr_matrix([$data, $indices, $indptr], dtype=>$dtype);
+ my ($num_rows, $num_cols) = @{ $nd->shape };
+ ok($num_rows == @{ $indptr } - 1);
+ ok($indices->shape->[0] > 0);
+ ok(($num_cols <= $indices)->aspdl->sum == 0);
+ ok($nd->dtype eq $dtype);
+ };
+ };
+ my $check_create_rsp_infer_shape = sub { my ($shape, $density, $dtype) = @_;
+ eval {
+ my $array = rand_ndarray($shape, 'row_sparse', $density);
+ my $data = $array->data;
+ my $indices = $array->indices;
+ my $nd = mx->nd->sparse->row_sparse_array([$data, $indices], dtype=>$dtype);
+ my $inferred_shape = $nd->shape;
+ is_deeply([@{ $inferred_shape }[1..@{ $inferred_shape }-1]], [@{ $data->shape }[1..@{ $data->shape }-1]]);
+ ok($indices->ndim > 0);
+ ok($nd->dtype eq $dtype);
+ if($indices->shape->[0] > 0)
+ {
+ ok(($inferred_shape->[0] <= $indices)->aspdl->sum == 0);
+ }
+ };
+ };
+
+ my $dtype = 'int32';
+ my $shape = rand_shape_2d();
+ my $shape_3d = rand_shape_3d();
+ my @densities = (0, 0.5, 1);
+ for my $density (@densities)
+ {
+ $check_create_csr_infer_shape->($shape, $density, $dtype);
+ $check_create_rsp_infer_shape->($shape, $density, $dtype);
+ $check_create_rsp_infer_shape->($shape_3d, $density, $dtype);
+ }
+}
+
+test_create_sparse_nd_infer_shape();
+
+sub test_create_sparse_nd_from_dense
+{
+ my $check_create_from_dns = sub { my ($shape, $f, $dense_arr, $dtype, $default_dtype, $ctx) = @_;
+ my $arr = $f->($dense_arr, shape => $shape, dtype => $dtype, ctx => $ctx);
+ ok(same($arr->aspdl, pones(reverse @{ $shape })));
+ ok($arr->dtype eq $dtype);
+ ok($arr->context eq $ctx);
+ # verify the default dtype inferred from dense arr
+ my $arr2 = $f->($dense_arr);
+ ok($arr2->dtype eq $default_dtype);
+ ok($arr2->context eq AI::MXNet::Context->current_ctx);
+ };
+ my $shape = rand_shape_2d();
+ my $dtype = 'int32';
+ my $src_dtype = 'float64';
+ my $ctx = mx->cpu(1);
+ my @dense_arrs = (
+ mx->nd->ones($shape, dtype=>$src_dtype),
+ mx->nd->ones($shape, dtype=>$src_dtype)->aspdl
+ );
+ for my $f (sub { mx->nd->sparse->csr_matrix(@_) }, sub { mx->nd->sparse->row_sparse_array(@_) })
+ {
+ for my $dense_arr (@dense_arrs)
+ {
+ my $default_dtype = blessed($dense_arr) ? $dense_arr->dtype : 'float32';
+ $check_create_from_dns->($shape, $f, $dense_arr, $dtype, $default_dtype, $ctx);
+ }
+ }
+}
+
+test_create_sparse_nd_from_dense();
+
+sub test_create_sparse_nd_from_sparse
+{
+ my $check_create_from_sp = sub { my ($shape, $f, $sp_arr, $dtype, $src_dtype, $ctx) = @_;
+ my $arr = $f->($sp_arr, shape => $shape, dtype=>$dtype, ctx=>$ctx);
+ ok(same($arr->aspdl, pones(reverse @{ $shape })));
+ ok($arr->dtype eq $dtype);
+ ok($arr->context eq $ctx);
+ # verify the default dtype inferred from sparse arr
+ my $arr2 = $f->($sp_arr);
+ ok($arr2->dtype eq $src_dtype);
+ ok($arr2->context eq AI::MXNet::Context->current_ctx);
+ };
+
+ my $shape = rand_shape_2d();
+ my $src_dtype = 'float64';
+ my $dtype = 'int32';
+ my $ctx = mx->cpu(1);
+ my $ones = mx->nd->ones($shape, dtype=>$src_dtype);
+ my @csr_arrs = ($ones->tostype('csr'));
+ my @rsp_arrs = ($ones->tostype('row_sparse'));
+ push @csr_arrs, mx->nd->ones($shape, dtype=>$src_dtype)->aspdl->tocsr;
+ my $f_csr = sub { mx->nd->sparse->csr_matrix(@_) };
+ my $f_rsp = sub { mx->nd->sparse->row_sparse_array(@_) };
+ for my $sp_arr (@csr_arrs)
+ {
+ $check_create_from_sp->($shape, $f_csr, $sp_arr, $dtype, $src_dtype, $ctx);
+ }
+ for my $sp_arr (@rsp_arrs)
+ {
+ $check_create_from_sp->($shape, $f_rsp, $sp_arr, $dtype, $src_dtype, $ctx);
+ }
+}
+
+test_create_sparse_nd_from_sparse();
+
+sub test_create_sparse_nd_empty
+{
+ my $check_empty = sub { my ($shape, $stype) = @_;
+ my $arr = mx->nd->sparse->empty($stype, $shape);
+ ok($arr->stype eq $stype);
+ ok(same($arr->aspdl, pzeros(reverse(@{ $shape }))));
+ };
+
+ my $check_csr_empty = sub { my ($shape, $dtype, $ctx) = @_;
+ my $arr = mx->nd->sparse->csr_matrix(undef, shape => $shape, dtype => $dtype, ctx => $ctx);
+ ok($arr->stype eq 'csr');
+ ok($arr->dtype eq $dtype);
+ ok($arr->context eq $ctx);
+ ok(same($arr->aspdl, pzeros(reverse(@{ $shape }))));
+ # check the default value for dtype and ctx
+ $arr = mx->nd->sparse->csr_matrix(undef, shape => $shape);
+ ok($arr->dtype eq 'float32');
+ ok($arr->context eq AI::MXNet::Context->current_ctx);
+ };
+
+ my $check_rsp_empty = sub { my ($shape, $dtype, $ctx) = @_;
+ my $arr = mx->nd->sparse->row_sparse_array(undef, shape => $shape, dtype=>$dtype, ctx=>$ctx);
+ ok($arr->stype eq 'row_sparse');
+ ok($arr->dtype eq $dtype);
+ ok($arr->context eq $ctx);
+ ok(same($arr->aspdl, pzeros(reverse(@{ $shape }))));
+ # check the default value for dtype and ctx
+ $arr = mx->nd->sparse->row_sparse_array(undef, shape => $shape);
+ ok($arr->dtype eq 'float32');
+ ok($arr->context eq AI::MXNet::Context->current_ctx);
+ };
+
+ my @stypes = ('csr', 'row_sparse');
+ my $shape = rand_shape_2d();
+ my $shape_3d = rand_shape_3d();
+ my $dtype = 'int32';
+ my $ctx = mx->cpu(1);
+ for my $stype (@stypes)
+ {
+ $check_empty->($shape, $stype);
+ }
+ $check_csr_empty->($shape, $dtype, $ctx);
+ $check_rsp_empty->($shape, $dtype, $ctx);
+ $check_rsp_empty->($shape_3d, $dtype, $ctx);
+}
+
+test_create_sparse_nd_empty();
+
+sub test_synthetic_dataset_generator
+{
+ my $test_powerlaw_generator = sub { my ($csr_arr, $final_row) = @_;
+ my $indices = $csr_arr->indices->aspdl;
+ my $indptr = $csr_arr->indptr->aspdl;
+ for my $row (1..$final_row)
+ {
+ my $nextrow = $row + 1;
+ my $current_row_nnz = $indices->at($indptr->at($row) - 1) + 1;
+ my $next_row_nnz = $indices->at($indptr->at($nextrow) - 1) + 1;
+ ok($next_row_nnz == 2 * $current_row_nnz);
+ }
+ };
+
+ # Test if density is preserved
+ my ($csr_arr_cols) = rand_sparse_ndarray([32, 10000], "csr",
+ density=>0.01, distribution=>"powerlaw");
+
+ my ($csr_arr_small) = rand_sparse_ndarray([5, 5], "csr",
+ density=>0.5, distribution=>"powerlaw");
+
+ my ($csr_arr_big) = rand_sparse_ndarray([32, 1000000], "csr",
+ density=>0.4, distribution=>"powerlaw");
+
+ my ($csr_arr_square) = rand_sparse_ndarray([1600, 1600], "csr",
+ density=>0.5, distribution=>"powerlaw");
+ ok($csr_arr_cols->data->len == 3200);
+ $test_powerlaw_generator->($csr_arr_cols, 9);
+ $test_powerlaw_generator->($csr_arr_small, 1);
+ $test_powerlaw_generator->($csr_arr_big, 4);
+ $test_powerlaw_generator->($csr_arr_square, 6);
+}
+
+test_synthetic_dataset_generator();
+
+sub test_sparse_nd_fluent
+{
+ my $check_fluent_regular = sub { my ($stype, $func, $kwargs, $shape, $equal_nan) = @_;
+ $shape //= [5, 17];
+ my $data = mx->nd->random->uniform(shape=>$shape)->tostype($stype);
+ my $regular = AI::MXNet::NDArray::Base->$func($data, %$kwargs);
+ my $fluent = $data->$func(%$kwargs);
+ ok(almost_equal($regular->aspdl, $fluent->aspdl));
+ };
+
+ my @common_func = ('zeros_like', 'square');
+ my @rsp_func = ('round', 'rint', 'fix', 'floor', 'ceil', 'trunc',
+ 'abs', 'sign', 'sin', 'degrees', 'radians', 'expm1');
+ for my $func (@common_func)
+ {
+ $check_fluent_regular->('csr', $func, {});
+ }
+ for my $func (@common_func, @rsp_func)
+ {
+ $check_fluent_regular->('row_sparse', $func, {});
+ }
+
+ @rsp_func = ('arcsin', 'arctan', 'tan', 'sinh', 'tanh',
+ 'arcsinh', 'arctanh', 'log1p', 'sqrt', 'relu');
+ for my $func (@rsp_func)
+ {
+ $check_fluent_regular->('row_sparse', $func, {});
+ }
+
+ $check_fluent_regular->('csr', 'slice', {begin => [2, 5], end => [4, 7]});
+ $check_fluent_regular->('row_sparse', 'clip', {a_min => -0.25, a_max => 0.75});
+
+ for my $func ('sum', 'mean')
+ {
+ $check_fluent_regular->('csr', $func, {axis => 0});
+ }
+}
+
+test_sparse_nd_fluent();
+
+sub test_sparse_nd_exception
+{
+ my $a = mx->nd->ones([2,2]);
+ dies_ok(sub { mx->nd->sparse->retain($a, invalid_arg=>"garbage_value") });
+ dies_ok(sub { mx->nd->sparse->csr_matrix($a, shape=>[3,2]) });
+ dies_ok(sub { mx->nd->sparse->csr_matrix(pdl([2,2]), shape=>[3,2]) });
+ dies_ok(sub { mx->nd->sparse->row_sparse_array(pdl([2,2]), shape=>[3,2]) });
+ dies_ok(sub { mx->nd->sparse->zeros("invalid_stype", [2,2]) });
+}
+
+test_sparse_nd_exception();
+
+sub test_sparse_nd_check_format
+{
+ my $shape = rand_shape_2d();
+ my @stypes = ("csr", "row_sparse");
+ for my $stype (@stypes)
+ {
+ my ($arr) = rand_sparse_ndarray($shape, $stype);
+ $arr->check_format();
+ $arr = mx->nd->sparse->zeros($stype, $shape);
+ $arr->check_format();
+ }
+ # CSR format index pointer array should be less than the number of rows
+ $shape = [3, 4];
+ my $data_list = [7, 8, 9];
+ my $indices_list = [0, 2, 1];
+ my $indptr_list = [0, 5, 2, 3];
+ my $a = mx->nd->sparse->csr_matrix([$data_list, $indices_list, $indptr_list], shape=>$shape);
+ dies_ok(sub { $a->check_format });
+ # CSR format indices should be in ascending order per row
+ $indices_list = [2, 1, 1];
+ $indptr_list = [0, 2, 2, 3];
+ $a = mx->nd->sparse->csr_matrix([$data_list, $indices_list, $indptr_list], shape=>$shape);
+ dies_ok(sub { $a->check_format });
+ # CSR format indptr should end with value equal with size of indices
+ $indices_list = [1, 2, 1];
+ $indptr_list = [0, 2, 2, 4];
+ $a = mx->nd->sparse->csr_matrix([$data_list, $indices_list, $indptr_list], shape=>$shape);
+ dies_ok(sub { $a->check_format });
+ # CSR format indices should not be negative
+ $indices_list = [0, 2, 1];
+ $indptr_list = [0, -2, 2, 3];
+ $a = mx->nd->sparse->csr_matrix([$data_list, $indices_list, $indptr_list], shape=>$shape);
+ dies_ok(sub { $a->check_format });
+ # Row Sparse format indices should be less than the number of rows
+ $shape = [3, 2];
+ $data_list = [[1, 2], [3, 4]];
+ $indices_list = [1, 4];
+ $a = mx->nd->sparse->row_sparse_array([$data_list, $indices_list], shape=>$shape);
+ dies_ok(sub { $a->check_format });
+ # Row Sparse format indices should be in ascending order
+ $indices_list = [1, 0];
+ $a = mx->nd->sparse->row_sparse_array([$data_list, $indices_list], shape=>$shape);
+ dies_ok(sub { $a->check_format });
+ # Row Sparse format indices should not be negative
+ $indices_list = [1, -2];
+ $a = mx->nd->sparse->row_sparse_array([$data_list, $indices_list], shape=>$shape);
+ dies_ok(sub { $a->check_format });
+}
+
+test_sparse_nd_check_format();
diff --git a/perl-package/AI-MXNetCAPI/Changes b/perl-package/AI-MXNetCAPI/Changes
index e873680..30426a5 100644
--- a/perl-package/AI-MXNetCAPI/Changes
+++ b/perl-package/AI-MXNetCAPI/Changes
@@ -1,5 +1,8 @@
Revision history for Perl extension AI::MXNetCAPI
+1.2 Sun Mar 4 16:29:19 PST 2018
+ - Support for sparse tensors
+
1.1 Sun Oct 1 10:19:08 PDT 2017
- support for perl 5.14, Gluon, cuda kernels
diff --git a/perl-package/AI-MXNetCAPI/META.json b/perl-package/AI-MXNetCAPI/META.json
index 92864bc..e194db9 100644
--- a/perl-package/AI-MXNetCAPI/META.json
+++ b/perl-package/AI-MXNetCAPI/META.json
@@ -37,5 +37,5 @@
}
},
"release_status" : "stable",
- "version" : "1.1"
+ "version" : "1.2"
}
diff --git a/perl-package/AI-MXNetCAPI/META.yml b/perl-package/AI-MXNetCAPI/META.yml
index 99ffd07..fa0bd13 100644
--- a/perl-package/AI-MXNetCAPI/META.yml
+++ b/perl-package/AI-MXNetCAPI/META.yml
@@ -19,4 +19,4 @@ no_index:
- inc
requires:
Test::More: '0'
-version: '1.1'
+version: '1.2'
diff --git a/perl-package/AI-MXNetCAPI/README b/perl-package/AI-MXNetCAPI/README
index 411eb71..dbed0c2 100644
--- a/perl-package/AI-MXNetCAPI/README
+++ b/perl-package/AI-MXNetCAPI/README
@@ -1,4 +1,4 @@
-AI-MXNetCAPI version 1.1
+AI-MXNetCAPI version 1.2
=====================
Swig interface to MXNet c api.
diff --git a/perl-package/AI-MXNetCAPI/lib/AI/MXNetCAPI.pm b/perl-package/AI-MXNetCAPI/lib/AI/MXNetCAPI.pm
index bf79d9a..ef51539 100644
--- a/perl-package/AI-MXNetCAPI/lib/AI/MXNetCAPI.pm
+++ b/perl-package/AI-MXNetCAPI/lib/AI/MXNetCAPI.pm
@@ -18,7 +18,7 @@
package AI::MXNetCAPI;
use base qw(DynaLoader);
bootstrap AI::MXNetCAPI;
-our $VERSION = '1.1';
+our $VERSION = '1.2';
1;
__END__
diff --git a/perl-package/AI-MXNetCAPI/mxnet.i b/perl-package/AI-MXNetCAPI/mxnet.i
index a55ca13..08f1639 100644
--- a/perl-package/AI-MXNetCAPI/mxnet.i
+++ b/perl-package/AI-MXNetCAPI/mxnet.i
@@ -80,6 +80,19 @@ static void KVStore_callback(int index, NDArrayHandle recv, NDArrayHandle local,
}
}
+static void KVStoreStr_callback(const char *key, NDArrayHandle recv, NDArrayHandle local, void* callback)
+{
+ {
+ dSP;
+ PUSHMARK(SP);
+ XPUSHs(sv_2mortal(newSVpv(key, 0)));
+ XPUSHs(SWIG_NewPointerObj(SWIG_as_voidptr(recv), SWIGTYPE_p_MXNDArray, 0));
+ XPUSHs(SWIG_NewPointerObj(SWIG_as_voidptr(local), SWIGTYPE_p_MXNDArray, 0));
+ PUTBACK;
+ call_sv((SV*)callback, G_DISCARD);
+ }
+}
+
static void KVStoreServer_callback(int head, const char *body, void* callback)
{
{
@@ -339,6 +352,37 @@ int MXNDArrayCreateEx(const mx_uint *in,
int dtype,
NDArrayHandle *out);
/*!
+ * \brief create an empty sparse NDArray with specified shape and data type
+ * \param storage_type the storage type of the ndarray
+ * \param shape the pointer to the shape
+ * \param ndim the dimension of the shape
+ * \param dev_type device type, specify device we want to take
+ * \param dev_id the device id of the specific device
+ * \param delay_alloc whether to delay allocation until
+ * the narray is first mutated
+ * \param dtype data type of created array
+ * \param num_aux the number of aux data to support this ndarray
+ * \param aux_type data type of the aux data for the created array
+ * \param aux_ndims the dimension of the shapes of aux data
+ * \param aux_shape the shapes of aux data
+ * \param out the returning handle
+ * \return 0 when success, -1 when failure happens
+ */
+int MXNDArrayCreateSparseEx(int storage_type,
+ const mx_uint *in,
+ mx_uint ndim,
+ int dev_type,
+ int dev_id,
+ int delay_alloc,
+ int dtype,
+ mx_uint num_aux,
+ int *in,
+ mx_uint *in,
+ const mx_uint *in,
+ NDArrayHandle *out);
+
+
+/*!
* \brief create a NDArray handle that is loaded from raw bytes.
* \param buf the head of the raw bytes
* \param size size of the raw bytes
@@ -413,6 +457,23 @@ int MXNDArraySyncCopyToCPU(NDArrayHandle handle,
void *in,
size_t size);
/*!
+ * \brief Copy src.data() to dst.data() if i = -1, else dst.aux_data(i) if i >= 0
+ * This function blocks. Do not use it in performance critical code.
+ * \param handle_dst handle of a dst ndarray whose data/aux_data has been allocated
+ * \param handle_src handle of a src ndarray which has default storage type
+ * \param i dst data blob indicator
+ */
+int MXNDArraySyncCopyFromNDArray(NDArrayHandle handle_dst,
+ const NDArrayHandle handle_src,
+ const int i);
+
+/*!
+ * \brief check whether the NDArray format is valid
+ * \param full_check if `True`, rigorous check, O(N) operations
+ * Otherwise basic check, O(1) operations
+ */
+int MXNDArraySyncCheckFormat(NDArrayHandle handle, const bool full_check);
+/*!
* \brief Wait until all the pending writes with respect NDArray are finished.
* Always call this before read data out synchronizely.
* \param handle the NDArray handle
@@ -461,6 +522,11 @@ int MXNDArrayAt(NDArrayHandle handle,
mx_uint idx,
NDArrayHandle *out);
/*!
+ * \brief get the storage type of the array
+ */
+int MXNDArrayGetStorageType(NDArrayHandle handle,
+ int *out);
+/*!
* \brief Reshape the NDArray.
* \param handle the handle to the ndarray
* \param ndim number of dimensions of new shape
@@ -499,6 +565,32 @@ int MXNDArrayGetData(NDArrayHandle handle,
int MXNDArrayGetDType(NDArrayHandle handle,
int *out);
/*!
+ * \brief get the type of the ith aux data in NDArray
+ * \param handle the handle to the narray
+ * \param i the index of the aux data
+ * \param out_type pointer holder to get type of aux data
+ * \return 0 when success, -1 when failure happens
+ */
+int MXNDArrayGetAuxType(NDArrayHandle handle,
+ mx_uint i,
+ int *out);
+/*!
+ * \brief Get a deep copy of the ith aux data blob
+ * in the form of an NDArray of default storage type.
+ * This function blocks. Do not use it in performance critical code.
+ */
+int MXNDArrayGetAuxNDArray(NDArrayHandle handle,
+ mx_uint i,
+ NDArrayHandle *out);
+/*!
+ * \brief Get a deep copy of the data blob
+ * in the form of an NDArray of default storage type.
+ * This function blocks. Do not use it in performance critical code.
+ */
+int MXNDArrayGetDataNDArray(NDArrayHandle handle,
+ NDArrayHandle *out);
+
+/*!
* \brief get the context of the NDArray
* \param handle the handle to the ndarray
* \param out_dev_type the output device type
@@ -648,6 +740,29 @@ int MXImperativeInvoke(AtomicSymbolCreator in,
const char **keys,
const char **vals);
/*!
+ * \brief invoke a nnvm op and imperative function
+ * \param creator the op
+ * \param num_inputs number of input NDArrays
+ * \param inputs input NDArrays
+ * \param num_outputs number of output NDArrays
+ * \param outputs output NDArrays
+ * \param num_params number of keyword parameters
+ * \param param_keys keys for keyword parameters
+ * \param param_vals values for keyword parameters
+ * \param out_stypes output ndarrays' stypes
+ * \return 0 when success, -1 when failure happens
+ */
+int MXImperativeInvokeEx(AtomicSymbolCreator in,
+ int num_inputs,
+ NDArrayHandle *in,
+ int *out_size,
+ NDArrayHandle **out_array,
+ int num_params,
+ const char **keys,
+ const char **vals,
+ const int **out_stypes);
+
+/*!
* \brief set whether to record operator for autograd
* \param is_recording 1 when recording, 0 when not recording.
* \param prev returns the previous status before this set.
@@ -752,6 +867,22 @@ int MXInvokeCachedOp(CachedOpHandle handle,
NDArrayHandle *in,
int *out_size,
NDArrayHandle **out_array);
+/*!
+ * \brief invoke a cached op
+ * \param handle the handle to the cached op
+ * \param num_inputs number of input NDArrays
+ * \param inputs input NDArrays
+ * \param num_outputs number of output NDArrays
+ * \param outputs output NDArrays
+ * \param out_stypes output ndarrays' stypes
+ * \return 0 when success, -1 when failure happens
+ */
+int MXInvokeCachedOpEx(CachedOpHandle handle,
+ int num_inputs,
+ NDArrayHandle *in,
+ int *out_size,
+ NDArrayHandle **out_array,
+ const int** out_stypes);
//--------------------------------------------
// Part 3: symbolic configuration generation
//--------------------------------------------
@@ -1318,29 +1449,21 @@ int MXExecutorSimpleBind(SymbolHandle symbol_handle,
const mx_uint num_provided_arg_dtypes,
const char** in, // provided_arg_dtype_names,
const int* in, // provided_arg_dtypes,
-
-//--------------- sparse related variables, ignored for now
const mx_uint num_provided_arg_stypes,
- const char** provided_arg_stype_names,
- const int* provided_arg_stypes,
-//---------------
+ const char** in, // provided_arg_stype_names,
+ const int* in, // provided_arg_stypes,
const mx_uint num_shared_arg_names,
const char** in, // shared_arg_name_list,
-//------------
int* shared_buffer_len,
const char** shared_buffer_name_list,
NDArrayHandle* shared_buffer_handle_list,
const char*** updated_shared_buffer_name_list,
NDArrayHandle** updated_shared_buffer_handle_list,
-//------------------
-
mx_uint* num_in_args,
NDArrayHandle** in_args,
NDArrayHandle** arg_grads,
-//-----------------
mx_uint* num_aux_states,
NDArrayHandle** aux_states,
-//----------
ExecutorHandle shared_exec_handle,
ExecutorHandle* out
);
@@ -1480,6 +1603,19 @@ int MXKVStoreCreate(const char *type,
int MXKVStoreFree(KVStoreHandle handle);
/*!
+ * \brief Set parameters to use low-bit compressed gradients
+ * \param handle handle to the kvstore
+ * \param keys keys for compression parameters
+ * \param vals values for compression parameters
+ * \return 0 when success, -1 when failure happens
+ */
+int MXKVStoreSetGradientCompression(KVStoreHandle handle,
+ mx_uint num_params,
+ const char** keys,
+ const char** vals);
+
+
+/*!
* \brief Init a list of (key,value) pairs in kvstore, where each key is a string
* \param handle handle to the kvstore
* \param num the number of key-value pairs
@@ -1520,6 +1656,42 @@ int MXKVStorePullEx(KVStoreHandle handle,
NDArrayHandle* in,
int priority);
/*!
+ * \brief pull a list of (key, value) pairs from the kvstore, where each key is an integer.
+ * The NDArray pulled back will be in row_sparse storage with only the specified
+ * row_ids present based row_ids (others rows are zeros).
+ * \param handle handle to the kvstore
+ * \param num the number of key-value pairs
+ * \param keys the list of keys
+ * \param vals the list of values
+ * \param row_ids the list of row_id NDArrays
+ * \param priority the priority of the action
+ * \return 0 when success, -1 when failure happens
+ */
+int MXKVStorePullRowSparse(KVStoreHandle handle,
+ mx_uint num,
+ const int* in,
+ NDArrayHandle* in,
+ NDArrayHandle* in,
+ int priority);
+/*!
+ * \brief pull a list of (key, value) pairs from the kvstore, where each key is a string.
+ * The NDArray pulled back will be in row_sparse storage with only the specified
+ * row_ids present based row_ids (others rows are zeros).
+ * \param handle handle to the kvstore
+ * \param num the number of key-value pairs
+ * \param keys the list of keys
+ * \param vals the list of values
+ * \param row_ids the list of row_id NDArrays
+ * \param priority the priority of the action
+ * \return 0 when success, -1 when failure happens
+ */
+int MXKVStorePullRowSparseEx(KVStoreHandle handle,
+ mx_uint num,
+ const char** in,
+ NDArrayHandle* in,
+ NDArrayHandle* in,
+ int priority);
+/*!
* \brief user-defined updater for the kvstore
* It's this updater's responsibility to delete \a recv and \a local
* \param the key
@@ -1532,6 +1704,18 @@ typedef void (MXKVStoreUpdater)(int key,
NDArrayHandle local,
void *handle);
/*!
+ * \brief user-defined updater for the kvstore with string keys
+ * It's this updater's responsibility to delete \a recv and \a local
+ * \param the key
+ * \param recv the pushed value on this key
+ * \param local the value stored on local on this key
+ * \param handle The additional handle to the updater
+ */
+typedef void (MXKVStoreStrUpdater)(const char* key,
+ NDArrayHandle recv,
+ NDArrayHandle local,
+ void *handle);
+/*!
* \brief register an push updater
* \param handle handle to the KVStore
* \param updater udpater function
@@ -1542,6 +1726,18 @@ int MXKVStoreSetUpdater(KVStoreHandle handle,
MXKVStoreUpdater updater,
void *callback_handle);
/*!
+ * \brief register a push updater with int keys and one with string keys
+ * \param handle handle to the KVStore
+ * \param updater updater function with int keys
+ * \param str_updater updater function with string keys
+ * \param updater_handle The additional handle used to invoke the updater
+ * \return 0 when success, -1 when failure happens
+ */
+int MXKVStoreSetUpdaterEx(KVStoreHandle handle,
+ MXKVStoreUpdater updater,
+ MXKVStoreStrUpdater updater,
+ void *callback_handle);
+/*!
* \brief get the type of the kvstore
* \param handle handle to the KVStore
* \param type a string type
diff --git a/perl-package/AI-MXNetCAPI/mxnet_typemaps.i b/perl-package/AI-MXNetCAPI/mxnet_typemaps.i
index cfbcede..ae1c7c9 100644
--- a/perl-package/AI-MXNetCAPI/mxnet_typemaps.i
+++ b/perl-package/AI-MXNetCAPI/mxnet_typemaps.i
@@ -112,6 +112,32 @@
}
}
+%typemap(in,numinputs=0) (const int **out_stypes) (int* temp)
+{
+ temp = NULL;
+ $1 = &temp;
+}
+
+%typemap(argout) (const int **out_stypes)
+{
+ if(av_len((AV*)SvRV(ST(3))) == -1 && !result)
+ {
+ AV *myav;
+ SV **svs;
+ int i = 0;
+ svs = (SV **)safemalloc(*arg4*sizeof(SV *));
+ for (i = 0; i < *arg4 ; i++) {
+ svs[i] = newSViv((*$1)[i]);
+ sv_2mortal(svs[i]);
+ }
+ myav = av_make(*arg4, svs);
+ Safefree(svs);
+ $result = newRV_noinc((SV*)myav);
+ sv_2mortal($result);
+ argvi++;
+ }
+}
+
%typemap(in,numinputs=0) (nn_uint *out_size, const char ***out_array) (nn_uint temp_size, char** temp),
(mx_uint *out_size, const char ***out_array) (mx_uint temp_size, char** temp)
{
@@ -131,7 +157,7 @@
for (i = 0; i < *$1 ; i++) {
svs[i] = newSVpv((*$2)[i],0);
sv_2mortal(svs[i]);
- };
+ }
myav = av_make(*$1,svs);
Safefree(svs);
$result = newRV_noinc((SV*)myav);
@@ -157,7 +183,7 @@
for (i = 0; i < *$1*2 ; i++) {
svs[i] = newSVpv((*$2)[i],0);
sv_2mortal(svs[i]);
- };
+ }
myav = av_make(*$1*2,svs);
Safefree(svs);
$result = newRV_noinc((SV*)myav);
@@ -284,7 +310,7 @@
$1 = NULL;
}
}
-%typemap(freearg) (NDArrayHandle* in), (SymbolHandle* in) {
+%typemap(freearg) (NDArrayHandle* in), (SymbolHandle* in) {
Safefree($1);
}
@@ -935,14 +961,6 @@
}
}
-%typemap(in,numinputs=0) (const mx_uint num_provided_arg_stypes, const char** provided_arg_stype_names,
- const int* provided_arg_stypes)
-{
- $1 = 0;
- $2 = NULL;
- $3 = NULL;
-}
-
%typemap(in,numinputs=0) (mx_uint* num_aux_states,
NDArrayHandle** aux_states)
(mx_uint temp1,
@@ -1090,6 +1108,11 @@
$1 = KVStore_callback;
}
+%typemap(in,numinputs=0) (MXKVStoreStrUpdater* updater)
+{
+ $1 = KVStoreStr_callback;
+}
+
%typemap(in,numinputs=0) (MXKVStoreServerController* controller)
{
$1 = KVStoreServer_callback;
diff --git a/perl-package/AI-NNVMCAPI/Changes b/perl-package/AI-NNVMCAPI/Changes
index 480090d..8c944d9 100644
--- a/perl-package/AI-NNVMCAPI/Changes
+++ b/perl-package/AI-NNVMCAPI/Changes
@@ -1,5 +1,8 @@
Revision history for Perl extension AI::NNVMCAPI.
+1.2 Sun Mar 4 16:29:19 PST 2018
+ - Support for sparse tensors
+
1.1 Sun Sep 24 10:26:54 PDT 2017
- support for perl 5.14
diff --git a/perl-package/AI-NNVMCAPI/META.json b/perl-package/AI-NNVMCAPI/META.json
index e652043..4457e6f 100644
--- a/perl-package/AI-NNVMCAPI/META.json
+++ b/perl-package/AI-NNVMCAPI/META.json
@@ -37,5 +37,5 @@
}
},
"release_status" : "stable",
- "version" : "1.1"
+ "version" : "1.2"
}
diff --git a/perl-package/AI-NNVMCAPI/META.yml b/perl-package/AI-NNVMCAPI/META.yml
index 446c9df..e7c01f8 100644
--- a/perl-package/AI-NNVMCAPI/META.yml
+++ b/perl-package/AI-NNVMCAPI/META.yml
@@ -19,4 +19,4 @@ no_index:
- inc
requires:
Test::More: '0'
-version: '1.1'
+version: '1.2'
diff --git a/perl-package/AI-NNVMCAPI/README b/perl-package/AI-NNVMCAPI/README
index 856611b..cc7c2f1 100644
--- a/perl-package/AI-NNVMCAPI/README
+++ b/perl-package/AI-NNVMCAPI/README
@@ -1,4 +1,4 @@
-AI-NNVMCAPI version 1.1
+AI-NNVMCAPI version 1.2
=====================
Swig interface to MXNet c api.
diff --git a/perl-package/AI-NNVMCAPI/lib/AI/NNVMCAPI.pm b/perl-package/AI-NNVMCAPI/lib/AI/NNVMCAPI.pm
index e39eac5..81453cd 100644
--- a/perl-package/AI-NNVMCAPI/lib/AI/NNVMCAPI.pm
+++ b/perl-package/AI-NNVMCAPI/lib/AI/NNVMCAPI.pm
@@ -18,7 +18,7 @@
package AI::NNVMCAPI;
use base qw(DynaLoader);
bootstrap AI::NNVMCAPI;
-our $VERSION = '1.1';
+our $VERSION = '1.2';
1;
__END__
diff --git a/perl-package/README.md b/perl-package/README.md
index 3d3bc5b..93e34d1 100644
--- a/perl-package/README.md
+++ b/perl-package/README.md
@@ -1,5 +1,4 @@
-[](https://builds.apache.org/job/incubator-mxnet/job/master/)
-[](https://mxnet.incubator.apache.org/api/perl/index.html)
+[Perl API](https://mxnet.incubator.apache.org/api/perl/index.html)
[](../LICENSE)
You have found MXNet Perl Package! The MXNet Perl packages brings flexible and efficient GPU
@@ -11,7 +10,7 @@ computing and state-of-art deep learning to Perl.
Installation
---------
-* [MXNet Setup Document](https://mxnet.incubator.apache.org/get_started/ubuntu_setup.html)
+* [MXNet Setup Document](http://mxnet.incubator.apache.org/install/ubuntu_setup.html#install-the-mxnet-package-for-perl)
- Check this out for detailed documents, examples and installation guides.
License
diff --git a/tests/ci_build/install/ubuntu_install_perl.sh b/tests/ci_build/install/ubuntu_install_perl.sh
index 217f322..bb0fa9d 100755
--- a/tests/ci_build/install/ubuntu_install_perl.sh
+++ b/tests/ci_build/install/ubuntu_install_perl.sh
@@ -21,4 +21,4 @@ set -x
# install libraries for mxnet's perl package on ubuntu
apt-get update && apt-get install -y libmouse-perl pdl cpanminus swig libgraphviz-perl
-cpanm -q Function::Parameters Hash::Ordered
+cpanm -q Function::Parameters Hash::Ordered PDL::CCS
diff --git a/tests/travis/setup.sh b/tests/travis/setup.sh
index c6e3bf8..eec6c23 100755
--- a/tests/travis/setup.sh
+++ b/tests/travis/setup.sh
@@ -51,9 +51,9 @@ fi
if [ ${TASK} == "perl_test" ]; then
if [ ${TRAVIS_OS_NAME} == "linux" ]; then
- cpanm -q -L "${HOME}/perl5" Function::Parameters Hash::Ordered
+ cpanm -q -L "${HOME}/perl5" Function::Parameters Hash::Ordered PDL::CCS
else
sudo sh -c 'curl -L https://cpanmin.us | perl - App::cpanminus'
- sudo cpanm -q -n PDL Mouse Function::Parameters Hash::Ordered
+ sudo cpanm -q -n PDL Mouse Function::Parameters Hash::Ordered PDL::CCS
fi
fi
--
To stop receiving notification emails like this one, please contact
sergeykolychev@apache.org.