You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by ni...@apache.org on 2017/05/01 21:36:29 UTC
[1/2] incubator-systemml git commit: [SYSTEMML-769] Adding support
for native BLAS for Linux
Repository: incubator-systemml
Updated Branches:
refs/heads/gh-pages c5ff65305 -> 1efaa5520
[SYSTEMML-769] Adding support for native BLAS for Linux
- Both MKL and OpenBLAS show 2-3x performance benefits on conv2d
operators on Lenet.
- There are several OpenMP related issues on Mac, hence we have explicitly
disabled native support for Mac and Windows. Since we have already have
cmake setup in place, we can always support BLAS on Mac and Windows in
future when OpenMP related issues are resolved.
Closes #344.
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/4874c4f2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/4874c4f2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/4874c4f2
Branch: refs/heads/gh-pages
Commit: 4874c4f220a2d95521759b4053ebe6fe6e8c776a
Parents: c5ff653
Author: Niketan Pansare <np...@us.ibm.com>
Authored: Sun Apr 30 10:16:23 2017 -0800
Committer: Niketan Pansare <np...@us.ibm.com>
Committed: Sun Apr 30 11:16:23 2017 -0700
----------------------------------------------------------------------
native-backend.md | 204 ++++++++++++++++++++++++++++++++++++++++++
troubleshooting-guide.md | 6 +-
2 files changed, 209 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/4874c4f2/native-backend.md
----------------------------------------------------------------------
diff --git a/native-backend.md b/native-backend.md
new file mode 100644
index 0000000..86a1340
--- /dev/null
+++ b/native-backend.md
@@ -0,0 +1,204 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+# User Guide
+
+By default, SystemML implements all its matrix operations in Java.
+This simplifies deployment especially in a distributed environment.
+
+In some cases (such as deep learning), the user might want to use native BLAS
+rather than SystemML's internal Java library for performing single-node
+operations such matrix multiplication, convolution, etc.
+By default, SystemML will first attempt to use Intel MKL (if installed)
+and then OpenBLAS (if installed).
+If both Intel MKL and OpenBLAS are not available, SystemML
+falls back to its internal Java library.
+
+To force SystemML to use internal Java library rather than native BLAS,
+please set the configuration property `native.blas` to `false`.
+
+The current version of SystemML only supports BLAS on Linux machines.
+
+
+## Step 1: Install BLAS
+
+### Option 1: Install Intel MKL (recommended)
+
+Download and install the [community version of Intel MKL](https://software.intel.com/sites/campaigns/nest/).
+Intel requires you to first register your email address and then sends the download link to your email address
+with license key.
+
+* Linux users will have to extract the downloaded `.tgz` file, execute `install.sh` and follow the guided setup.
+
+### Option 2: Install OpenBLAS
+
+```bash
+# The default OpenBLAS (via yum/apt-get) uses its internal threading rather than OpenMP,
+# which can lead to performance degradation when using SystemML. So, instead we recommend that you
+# compile OpenBLAS from the source.
+# RedHat / CentOS: sudo yum install openblas
+# Ubuntu: sudo apt-get install openblas
+git clone https://github.com/xianyi/OpenBLAS.git
+cd OpenBLAS/
+make clean
+make USE_OPENMP=1
+sudo make install
+# After installation, you may also want to add `/opt/OpenBLAS/lib` to your LD_LIBRARY_PATH or `java.library.path`.
+```
+
+You can check if the OpenBLAS on you system is compiled with OpenMP or not using following commands:
+
+```bash
+$ ldconfig -p | grep libopenblas.so
+libopenblas.so (libc6,x86-64) => /opt/OpenBLAS/lib/libopenblas.so
+$ ldd /opt/OpenBLAS/lib/libopenblas.so | grep libgomp
+libgomp.so.1 => /lib64/libgomp.so.1
+```
+
+If you don't see any output after the second command, then OpenBLAS installed on your system is using its internal threading.
+In this case, we highly recommend that you reinstall OpenBLAS using the above commands.
+
+
+## Step 2: Install other dependencies
+
+```bash
+# Centos/RedHat
+sudo yum install gcc-c++
+# Ubuntu
+sudo apt-get install g++
+```
+
+We also depend on GNU OpenMP (gomp) which will be installed by GCC.
+To find the location of `gomp` on your system, please use the command `ldconfig -p | grep libgomp`.
+If gomp is available as `/lib64/libgomp.so.1` instead of `/lib64/libgomp.so`,
+please add a softlink to it:
+
+```bash
+sudo ln -s /lib64/libgomp.so.1 /lib64/libgomp.so
+```
+
+## Step 3: Provide the location of the native libraries
+
+1. Add the location of the native libraries (i.e. BLAS and other dependencies)
+to the environment variable `LD_LIBRARY_PATH` (on Linux).
+If you want to use SystemML with Spark, please add the following line to `spark-env.sh`
+
+ ```bash
+ export LD_LIBRARY_PATH=/path/to/blas-n-other-dependencies
+ # Or export SPARK_LIBRARY_PATH=/path/to/blas-n-other-dependencies
+ ```
+
+2. Alternatively, you can pass the location of the native libraries using command-line options:
+
+- Java: `-Djava.library.path=/path/to/blas-n-other-dependencies`
+- [Spark](http://spark.apache.org/docs/latest/configuration.html): `--driver-library-path`
+
+## Common issues on Linux
+
+1. Unable to load `gomp`
+
+First make sure if gomp is available on your system.
+
+ ```bash
+ ldconfig -p | grep libgomp
+ ```
+
+If the above command returns no results, then you may have to install `gcc`.
+On the other hand, if the above command only returns libgomp with major suffix (such as `so.1`),
+then please execute the below command:
+
+ ```bash
+ sudo ln -s /lib64/libgomp.so.1 /usr/lib64/libgomp.so
+ ```
+
+2. Unable to load `mkl_rt`
+
+By default, Intel MKL libraries will be installed in the location `/opt/intel/mkl/lib/intel64/`.
+Make sure that this path is accessible to Java as per instructions provided in the above section.
+
+3. Unable to load `openblas`
+
+By default, OpenBLAS libraries will be installed in the location `/opt/OpenBLAS/lib/`.
+Make sure that this path is accessible to Java as per instructions provided in the above section.
+
+# Developer Guide
+
+This section describes how to compile shared libraries in the folder `src/main/cpp/lib`.
+This is required when the developer makes changes to cpp directory or while validating the source package during the release process.
+
+To force SystemML to use OpenBLAS instead of Intel MKL if both are installed,
+please set the environment variable `SYSTEMML_BLAS` to `openblas`.
+This environment variable is used internally for testing and is not required for users.
+
+## Intro to CMake
+If you are familiar with cmake, skip this section.
+
+In a regular project with a Makefile, the compiled object files are placed in the same directory as the source.
+Sometimes we don't want to pollute the source tree. We might also want to have different binaries for different configurations. For instance, if we want to link a binary with separate libraries.
+CMake supports out of source tree builds. As an illustration, you can create a directory called "BUILD" and invoke cmake like so : `cmake <path/to/source>`. The makefile and other config files are placed in this "BUILD" directory. You can now say `make` and the compiled objects and binary files are created in this directory. You can then create another "BUILD2" directory and repeat the process.
+You can pass options to cmake as well. In this instance, it might be to specify whether to build with Intel MKL or OpenBLAS. This can be done from the command line with a "-D" appended to it, but more interestingly, it can also be done form a n-curses GUI which is invoked as `ccmake <path/to/source>`. (You may need to install this separately).
+Also, the C, C++ compilers and their flags are picked up by cmake when set in standard environment variables. These are respectively `CC`, `CXX`, `CFLAGS` & `CXFLAGS`. As an example, they may be specified as:
+
+ CXX=gcc-6 cmake ..
+
+For this project, I typically make a directory in the `cpp` folder (this folder) and name it the config I use. For instance, `INTEL` for Intel MKL and `OPENBLAS` for OpenBLAS.
+
+1. Install `g++`, OpenBLAS and MKL using the above instructions
+
+2. Set `JAVA_HOME` to JDK.
+
+ export JAVA_HOME=<path to JDK 1.8>
+
+3. Install cmake
+
+ ```bash
+ # Centos/RedHat
+ sudo yum install cmake3
+ # Ubuntu
+ sudo apt-get install cmake
+ ```
+
+4. Compile the libs using the below script.
+
+ ```bash
+ mkdir INTEL && cd INTEL
+ cmake -DUSE_INTEL_MKL=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CXX_FLAGS="-DUSE_GNU_THREADING -m64" ..
+ make install
+ cd ..
+ mkdir OPENBLAS && cd OPENBLAS
+ cmake -DUSE_OPEN_BLAS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CXX_FLAGS="-m64" ..
+ make install
+ cd ..
+ # The below script helps maintain this document as well as avoid accidental inclusion of non-standard dependencies.
+ ./check-dependency-linux-x86_64.sh
+ ```
+
+
+The generated library files are placed in src/main/cpp/lib. This location can be changed from the CMakeLists.txt file.
+
+The above script also validates whether additional dependencies have been added while compiling and warns the developer.
+The current set of dependencies other than MKL and OpenBLAS, are as follows:
+
+- GNU Standard C++ Library: `libstdc++.so.6`
+- GCC version 4.8 shared support library: `libgcc_s.so.1`
+- The GNU libc libraries: `libm.so.6, libdl.so.2, libc.so.6, libpthread.so.0`
+- GCC OpenMP v3.0 shared support library: `libgomp.so.1`
+- Additional OpenBLAS dependencies: Fortran runtime (`libgfortran.so.3`) and GCC `__float128` shared support library (`libquadmath.so.0`)
+
+If CMake cannot detect your OpenBLAS installation, set the `OpenBLAS_HOME` environment variable to the OpenBLAS Home.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/4874c4f2/troubleshooting-guide.md
----------------------------------------------------------------------
diff --git a/troubleshooting-guide.md b/troubleshooting-guide.md
index 4731f51..e80cad4 100644
--- a/troubleshooting-guide.md
+++ b/troubleshooting-guide.md
@@ -135,4 +135,8 @@ by setting the configuration `systemml.stats.extraGPU` and `systemml.stats.extra
Out-Of-Memory on executors is often caused due to side-effects of lazy evaluation and in-memory input data of Spark for large-scale problems.
Though we are constantly improving our optimizer to address this scenario, a quick hack to resolve this is reducing the number of cores allocated to the executor.
-We would highly appreciate if you file a bug report on our [issue tracker](https://issues.apache.org/jira/browse/SYSTEMML) if and when you encounter OOM.
\ No newline at end of file
+We would highly appreciate if you file a bug report on our [issue tracker](https://issues.apache.org/jira/browse/SYSTEMML) if and when you encounter OOM.
+
+## Native BLAS errors
+
+Please see [the user guide of native backend](http://apache.github.io/incubator-systemml/native-backend).
\ No newline at end of file
[2/2] incubator-systemml git commit: [SYSTEMML-540] Added the
documentation for convolution
Posted by ni...@apache.org.
[SYSTEMML-540] Added the documentation for convolution
Closes #464.
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/1efaa552
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/1efaa552
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/1efaa552
Branch: refs/heads/gh-pages
Commit: 1efaa552063a837ab7bb7f4a7e57e9efb088d7e8
Parents: 4874c4f
Author: Niketan Pansare <np...@us.ibm.com>
Authored: Sun Apr 30 12:39:12 2017 -0700
Committer: Niketan Pansare <np...@us.ibm.com>
Committed: Sun Apr 30 12:39:12 2017 -0700
----------------------------------------------------------------------
dml-language-reference.md | 32 +++++++++++++++++++
img/dml-language-reference/Conv2d.gif | Bin 0 -> 419930 bytes
img/dml-language-reference/Conv2d1.gif | Bin 0 -> 174964 bytes
.../Conv2d_backward_data.gif | Bin 0 -> 444715 bytes
.../Conv2d_backward_data1.gif | Bin 0 -> 427546 bytes
.../Conv2d_backward_data2.gif | Bin 0 -> 376145 bytes
6 files changed, 32 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/1efaa552/dml-language-reference.md
----------------------------------------------------------------------
diff --git a/dml-language-reference.md b/dml-language-reference.md
index 31f7d23..9273857 100644
--- a/dml-language-reference.md
+++ b/dml-language-reference.md
@@ -1494,6 +1494,38 @@ The following code snippet shows an example scenario of transforming a training
Note that the metadata generated during the training phase (located at `/user/ml/train_tf_metadata`) is used to apply the list of transformations (that were carried out on training data set) on the test data set. Since the second invocation of `transform()` does not really generate any new metadata data, the given metadata (`/user/ml/train_tf_metdata`) is copied to the target location (`/user/ml/test_tf_metdata`). Even though such a behavior creates redundant copies of transformation metadata, it is preferred as it allows the association of every data set with the corresponding transformation metadata.
+### Deep Learning Built-In Functions
+
+SystemML represent a tensor as a matrix stored in a row-major format,
+where first dimension of tensor and matrix are exactly the same. For example, a tensor (with all zeros)
+of shape [3, 2, 4, 5] can be instantiated by following DML statement:
+```sh
+A = matrix(0, rows=3, cols=2*4*5)
+```
+
+The images are assumed to be stored NCHW format, where N = batch size, C = #channels, H = height of image and W = width of image.
+Hence, the images are internally represented as a matrix with dimension (N, C * H * W).
+
+
+| Function name | Input matrices | Input Parameters | Notes |
+|------------------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|
+| conv2d | input, filter | stride=[stride_h, stride_w], padding=[pad_h, pad_w], input_shape=[batch_size, num_channels, height_image, width_image], filter_shape=[numFilters, numChannels, height_filter, width_filter] | Performs 2D convolution operation |
+| conv2d_backward_filter | input, dout | stride=[stride_h, stride_w], padding=[pad_h, pad_w], input_shape=[batch_size, num_channels, height_image, width_image], filter_shape=[numFilters, numChannels, height_filter, width_filter] | Computes the gradients wrt filter of 2D convolution |
+| conv2d_backward_data | filter, dout | stride=[stride_h, stride_w], padding=[pad_h, pad_w], input_shape=[batch_size, num_channels, height_image, width_image], filter_shape=[numFilters, numChannels, height_filter, width_filter] | Computes the gradients wrt input of 2D convolution |
+| max_pool | input | stride=[stride_h, stride_w], padding=[pad_h, pad_w], input_shape=[batch_size, num_channels, height_image, width_image], pool_size=[height_pool, width_pool] | Performs max pooling operation |
+| max_pool_backward | input, dout | stride=[stride_h, stride_w], padding=[pad_h, pad_w], input_shape=[batch_size, num_channels, height_image, width_image], pool_size=[height_pool, width_pool] | Computes the gradients wrt input of 2D maxpooling |
+
+
+Examples:
+
+| Function | Parameters | Visualization |
+|----------------------|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| conv2d | stride=[1,1] |  |
+| conv2d | stride=[2,2] |  |
+| conv2d_backward_data | stride=[1,1] |  |
+| conv2d_backward_data | stride=[2,2] |  |
+| conv2d_backward_data | stride=[2,2] and 2x2 filter |  |
+
### Other Built-In Functions
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/1efaa552/img/dml-language-reference/Conv2d.gif
----------------------------------------------------------------------
diff --git a/img/dml-language-reference/Conv2d.gif b/img/dml-language-reference/Conv2d.gif
new file mode 100644
index 0000000..b10a5cf
Binary files /dev/null and b/img/dml-language-reference/Conv2d.gif differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/1efaa552/img/dml-language-reference/Conv2d1.gif
----------------------------------------------------------------------
diff --git a/img/dml-language-reference/Conv2d1.gif b/img/dml-language-reference/Conv2d1.gif
new file mode 100644
index 0000000..fafc6fa
Binary files /dev/null and b/img/dml-language-reference/Conv2d1.gif differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/1efaa552/img/dml-language-reference/Conv2d_backward_data.gif
----------------------------------------------------------------------
diff --git a/img/dml-language-reference/Conv2d_backward_data.gif b/img/dml-language-reference/Conv2d_backward_data.gif
new file mode 100644
index 0000000..22c76e3
Binary files /dev/null and b/img/dml-language-reference/Conv2d_backward_data.gif differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/1efaa552/img/dml-language-reference/Conv2d_backward_data1.gif
----------------------------------------------------------------------
diff --git a/img/dml-language-reference/Conv2d_backward_data1.gif b/img/dml-language-reference/Conv2d_backward_data1.gif
new file mode 100644
index 0000000..fccec6a
Binary files /dev/null and b/img/dml-language-reference/Conv2d_backward_data1.gif differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/1efaa552/img/dml-language-reference/Conv2d_backward_data2.gif
----------------------------------------------------------------------
diff --git a/img/dml-language-reference/Conv2d_backward_data2.gif b/img/dml-language-reference/Conv2d_backward_data2.gif
new file mode 100644
index 0000000..f8273c2
Binary files /dev/null and b/img/dml-language-reference/Conv2d_backward_data2.gif differ