You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by de...@apache.org on 2016/06/01 19:50:37 UTC
[1/3] incubator-systemml git commit: [SYSTEMML-540] Initial
implementation of conv2d/pooling builtin function
Repository: incubator-systemml
Updated Branches:
refs/heads/gh-pages 9e715abcb -> 9e457e5bf
[SYSTEMML-540] Initial implementation of conv2d/pooling builtin function
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/686bf815
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/686bf815
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/686bf815
Branch: refs/heads/gh-pages
Commit: 686bf815e5e459511bba33909124b21ced965299
Parents: 9e715ab
Author: Niketan Pansare <np...@us.ibm.com>
Authored: Mon May 16 16:43:39 2016 -0800
Committer: Niketan Pansare <np...@us.ibm.com>
Committed: Mon May 16 17:45:39 2016 -0700
----------------------------------------------------------------------
devdocs/deep-learning.md | 122 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 122 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/686bf815/devdocs/deep-learning.md
----------------------------------------------------------------------
diff --git a/devdocs/deep-learning.md b/devdocs/deep-learning.md
new file mode 100644
index 0000000..46f2502
--- /dev/null
+++ b/devdocs/deep-learning.md
@@ -0,0 +1,122 @@
+# Initial prototype for Deep Learning
+
+## Representing tensor and images in SystemML
+
+In this prototype, we represent a tensor as a matrix stored in a row-major format,
+where first dimension of tensor and matrix are exactly the same. For example, a tensor (with all zeros)
+of shape [3, 2, 4, 5] can be instantiated by following DML statement:
+```sh
+A = matrix(0, rows=3, cols=2*4*5)
+```
+### Tensor functions:
+
+#### Element-wise arithmetic operators:
+Following operators work out-of-the box when both tensors X and Y have same shape:
+
+* Element-wise exponentiation: `X ^ Y`
+* Element-wise unary minus: `-X`
+* Element-wise integer division: `X %/% Y`
+* Element-wise modulus operation: `X %% Y`
+* Element-wise multiplication: `X * Y`
+* Element-wise division: `X / Y`
+* Element-wise addition: `X + Y`
+* Element-wise subtraction: `X - Y`
+
+SystemML does not support implicit broadcast for above tensor operations, however one can write a DML-bodied function to do so.
+For example: to perform the above operations with broadcasting on second dimensions, one can use the below `rep(Z, n)` function:
+``` python
+rep = function(matrix[double] Z, int C) return (matrix[double] ret) {
+ ret = Z
+ for(i in 2:C) {
+ ret = cbind(ret, Z)
+ }
+}
+```
+Using the above `rep(Z, n)` function, we can realize the element-wise arithmetic operation with broadcasting. Here are some examples:
+* X of shape [N, C, H, W] and Y of shape [1, C, H, W]: `X + Y` (Note: SystemML does implicit broadcasting in this case because of the way
+it represents the tensor)
+* X of shape [1, C, H, W] and Y of shape [N, C, H, W]: `X + Y` (Note: SystemML does implicit broadcasting in this case because of the way
+it represents the tensor)
+* X of shape [N, C, H, W] and Y of shape [N, 1, H, W]: `X + rep(Y, C)`
+* X of shape [N, C, H, W] and Y of shape [1, 1, H, W]: `X + rep(Y, C)`
+* X of shape [N, 1, H, W] and Y of shape [N, C, H, W]: `rep(X, C) + Y`
+* X of shape [1, 1, H, W] and Y of shape [N, C, H, W]: `rep(X, C) + Y`
+
+TODO: Map the NumPy tensor calls to DML expressions.
+
+## Representing images in SystemML
+
+The images are assumed to be stored NCHW format, where N = batch size, C = #channels, H = height of image and W = width of image.
+Hence, the images are internally represented as a matrix with dimension (N, C * H * W).
+
+## Convolution and Pooling built-in functions
+
+This prototype also contains initial implementation of forward/backward functions for 2D convolution and pooling:
+* `conv2d(x, w, ...)`
+* `conv2d_backward_filter(x, dout, ...)` and `conv2d_backward_data(w, dout, ...)`
+* `max_pool(x, ...)` and `max_pool_backward(x, dout, ...)`
+
+The required arguments for all above functions are:
+* stride=[stride_h, stride_w]
+* padding=[pad_h, pad_w]
+* input_shape=[numImages, numChannels, height_image, width_image]
+
+The additional required argument for conv2d/conv2d_backward_filter/conv2d_backward_data functions is:
+* filter_shape=[numFilters, numChannels, height_filter, width_filter]
+
+The additional required argument for max_pool/avg_pool functions is:
+* pool_size=[height_pool, width_pool]
+
+The results of these functions are consistent with Nvidia's CuDNN library.
+
+### Border mode:
+* To perform valid padding, use `padding = (input_shape-filter_shape)*(stride-1)/ 2`. (Hint: for stride length of 1, `padding = [0, 0]` performs valid padding).
+
+* To perform full padding, use `padding = ((stride-1)*input_shape + (stride+1)*filter_shape - 2*stride) / 2`. (Hint: for stride length of 1, `padding = [filter_h-1, filter_w-1]` performs full padding).
+
+* To perform same padding, use `padding = (input_shape*(stride-1) + filter_shape - stride)/2`. (Hint: for stride length of 1, `padding = [(filter_h-1)/2, (filter_w-1)/2]` performs same padding).
+
+### Explanation of backward functions for conv2d
+
+Consider one-channel 3 X 3 image =
+
+| x1 | x2 | x3 |
+|----|----|----|
+| x4 | x5 | x6 |
+| x7 | x8 | x9 |
+
+and one 2 X 2 filter:
+
+| w1 | w2 |
+|----|----|
+| w3 | w4 |
+
+Then, `conv2d(x, w, stride=[1, 1], padding=[0, 0], input_shape=[1, 1, 3, 3], filter_shape=[1, 1, 2, 2])` produces following tensor
+of shape `[1, 1, 2, 2]`, which is represented as `1 X 4` matrix in NCHW format:
+
+| `w1*x1 + w2*x2 + w3*x4 + w4*x5` | `w1*x2 + w2*x3 + w3*x5 + w4*x6` | `w1*x4 + w2*x5 + w3*x7 + w4*x8` | `w1*x5 + w2*x6 + w3*x8 + w4*x9` |
+|---------------------------------|---------------------------------|---------------------------------|---------------------------------|
+
+
+Let the error propagated from above layer is
+
+| y1 | y2 | y3 | y4 |
+|----|----|----|----|
+
+Then `conv2d_backward_filter(x, y, stride=[1, 1], padding=[0, 0], input_shape=[1, 1, 3, 3], filter_shape=[1, 1, 2, 2])` produces following
+updates for the filter:
+
+| `y1*x1 + y2*x2 + y3*x4 + y4*x5` | `y1*x2 + y2*x3 + y3*x5 + y4*x6` |
+|---------------------------------|---------------------------------|
+| `y1*x4 + y2*x5 + y3*x7 + y4*x8` | `y1*x5 + y2*x6 + y3*x8 + y4*x9` |
+
+Note: since the above update is a tensor of shape [1, 1, 2, 2], it will be represented as matrix of dimension [1, 4].
+
+Similarly, `conv2d_backward_data(w, y, stride=[1, 1], padding=[0, 0], input_shape=[1, 1, 3, 3], filter_shape=[1, 1, 2, 2])` produces following
+updates for the image:
+
+
+| `w1*y1` | `w2*y1 + w1*y2` | `w2*y2` |
+|-----------------|---------------------------------|-----------------|
+| `w3*y1 + w1*y3` | `w4*y1 + w3*y2 + w2*y3 + w1*y4` | `w4*y2 + w2*y4` |
+| `w3*y3` | `w4*y3 + w3*y4` | `w4*y4` |
[2/3] incubator-systemml git commit: [HOTFIX] Added license to
deep-learning.md
Posted by de...@apache.org.
[HOTFIX] Added license to deep-learning.md
Closes #161.
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/ee6727e6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/ee6727e6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/ee6727e6
Branch: refs/heads/gh-pages
Commit: ee6727e690af9b9aedb8b1fa1d74c00f28f302a2
Parents: 686bf81
Author: Glenn Weidner <gw...@us.ibm.com>
Authored: Tue May 17 08:47:40 2016 -0700
Committer: Mike Dusenberry <mw...@us.ibm.com>
Committed: Tue May 17 09:08:08 2016 -0700
----------------------------------------------------------------------
devdocs/deep-learning.md | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/ee6727e6/devdocs/deep-learning.md
----------------------------------------------------------------------
diff --git a/devdocs/deep-learning.md b/devdocs/deep-learning.md
index 46f2502..1fb951a 100644
--- a/devdocs/deep-learning.md
+++ b/devdocs/deep-learning.md
@@ -1,3 +1,22 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
# Initial prototype for Deep Learning
## Representing tensor and images in SystemML
[3/3] incubator-systemml git commit: Initial release process doc
Posted by de...@apache.org.
Initial release process doc
Closes #170.
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/9e457e5b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/9e457e5b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/9e457e5b
Branch: refs/heads/gh-pages
Commit: 9e457e5bfdb19eefaeff7ea7f55d6eea5d349cb8
Parents: ee6727e
Author: Deron Eriksson <de...@us.ibm.com>
Authored: Wed Jun 1 12:46:28 2016 -0700
Committer: Deron Eriksson <de...@us.ibm.com>
Committed: Wed Jun 1 12:46:28 2016 -0700
----------------------------------------------------------------------
release-process.md | 282 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 282 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/9e457e5b/release-process.md
----------------------------------------------------------------------
diff --git a/release-process.md b/release-process.md
new file mode 100644
index 0000000..ded33e9
--- /dev/null
+++ b/release-process.md
@@ -0,0 +1,282 @@
+---
+layout: global
+title: SystemML Release Process
+description: Description of the SystemML release process and validation.
+displayTitle: SystemML Release Process
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+* This will become a table of contents (this text will be scraped).
+{:toc}
+
+# Release Candidate Build and Deployment
+
+To be written.
+
+
+# Release Candidate Checklist
+
+## All Artifacts and Checksums Present
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+Verify that each expected artifact is present at [https://dist.apache.org/repos/dist/dev/incubator/systemml/](https://dist.apache.org/repos/dist/dev/incubator/systemml/) and that each artifact has accompanying
+checksums (such as .asc and .md5).
+
+
+## Release Candidate Build
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+The release candidate should build on Windows, OS X, and Linux. To do this cleanly,
+the following procedure can be performed.
+
+Clone the Apache SystemML GitHub repository
+to an empty location. Next, check out the release tag. Following
+this, build the distributions using Maven. This should be performed
+with an empty local Maven repository.
+
+Here is an example:
+
+ $ git clone https://github.com/apache/incubator-systemml.git
+ $ cd incubator-systemml
+ $ git tag -l
+ $ git checkout tags/0.10.0-incubating-rc1 -b 0.10.0-incubating-rc1
+ $ mvn -Dmaven.repo.local=$HOME/.m2/temp-repo clean package -P distribution
+
+
+## Test Suite Passes
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+The entire test suite should pass on Windows, OS X, and Linux.
+The test suite can be run using:
+
+ $ mvn clean verify
+
+
+## All Binaries Execute
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+Validate that all of the binary artifacts can execute, including those artifacts packaged
+in other artifacts (in the tar.gz and zip artifacts).
+
+The build artifacts should be downloaded from [https://dist.apache.org/repos/dist/dev/incubator/systemml/](https://dist.apache.org/repos/dist/dev/incubator/systemml/) and these artifacts should be tested, as in
+this OS X example.
+
+ # download artifacts
+ wget -r -nH -nd -np -R index.html* https://dist.apache.org/repos/dist/dev/incubator/systemml/0.10.0-incubating-rc1/
+
+ # verify standalone tar.gz works
+ tar -xvzf systemml-0.10.0-incubating-standalone.tar.gz
+ cd systemml-0.10.0-incubating-standalone
+ echo "print('hello world');" > hello.dml
+ ./runStandaloneSystemML.sh hello.dml
+ cd ..
+
+ # verify main jar works
+ mkdir lib
+ cp -R systemml-0.10.0-incubating-standalone/lib/* lib/
+ rm lib/systemml-0.10.0-incubating.jar
+ java -cp ./lib/*:systemml-0.10.0-incubating.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+
+ # verify standalone jar works
+ java -jar systemml-0.10.0-incubating-standalone.jar -s "print('hello world');"
+
+ # verify src works
+ tar -xvzf systemml-0.10.0-incubating-src.tar.gz
+ cd systemml-0.10.0-incubating-src
+ mvn clean package -P distribution
+ cd target/
+ java -cp ./lib/*:systemml-0.10.0-incubating.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+ java -cp ./lib/*:SystemML.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+ java -jar systemml-0.10.0-incubating-standalone.jar -s "print('hello world');"
+ cd ..
+ cd ..
+
+ # verify in-memory jar works
+ echo "import org.apache.sysml.api.jmlc.*;public class JMLCEx {public static void main(String[] args) throws Exception {Connection conn = new Connection();PreparedScript script = conn.prepareScript(\"print('hello world');\", new String[]{}, new String[]{}, false);script.executeScript();}}" > JMLCEx.java
+ javac -cp systemml-0.10.0-incubating-inmemory.jar JMLCEx.java
+ java -cp .:systemml-0.10.0-incubating-inmemory.jar JMLCEx
+
+ # verify distrib tar.gz works
+ tar -xvzf systemml-0.10.0-incubating.tar.gz
+ cd systemml-0.10.0-incubating
+ java -cp ../lib/*:SystemML.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+
+ # verify spark batch mode
+ export SPARK_HOME=/Users/deroneriksson/spark-1.5.1-bin-hadoop2.6
+ $SPARK_HOME/bin/spark-submit SystemML.jar -s "print('hello world');" -exec hybrid_spark
+
+ # verify hadoop batch mode
+ hadoop jar SystemML.jar -s "print('hello world');"
+
+
+Here is an example of doing a basic
+sanity check on OS X after building the artifacts manually.
+
+ # build distribution artifacts
+ mvn clean package -P distribution
+
+ cd target
+
+ # verify main jar works
+ java -cp ./lib/*:systemml-0.10.0-incubating.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+
+ # verify SystemML.jar works
+ java -cp ./lib/*:SystemML.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+
+ # verify standalone jar works
+ java -jar systemml-0.10.0-incubating-standalone.jar -s "print('hello world');"
+
+ # verify src works
+ tar -xvzf systemml-0.10.0-incubating-src.tar.gz
+ cd systemml-0.10.0-incubating-src
+ mvn clean package -P distribution
+ cd target/
+ java -cp ./lib/*:systemml-0.10.0-incubating.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+ java -cp ./lib/*:SystemML.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+ java -jar systemml-0.10.0-incubating-standalone.jar -s "print('hello world');"
+ cd ..
+ cd ..
+
+ # verify in-memory jar works
+ echo "import org.apache.sysml.api.jmlc.*;public class JMLCEx {public static void main(String[] args) throws Exception {Connection conn = new Connection();PreparedScript script = conn.prepareScript(\"print('hello world');\", new String[]{}, new String[]{}, false);script.executeScript();}}" > JMLCEx.java
+ javac -cp systemml-0.10.0-incubating-inmemory.jar JMLCEx.java
+ java -cp .:systemml-0.10.0-incubating-inmemory.jar JMLCEx
+
+ # verify standalone tar.gz works
+ tar -xvzf systemml-0.10.0-incubating-standalone.tar.gz
+ cd systemml-0.10.0-incubating-standalone
+ echo "print('hello world');" > hello.dml
+ ./runStandaloneSystemML.sh hello.dml
+ cd ..
+
+ # verify distrib tar.gz works
+ tar -xvzf systemml-0.10.0-incubating.tar.gz
+ cd systemml-0.10.0-incubating
+ java -cp ../lib/*:SystemML.jar org.apache.sysml.api.DMLScript -s "print('hello world');"
+
+ # verify spark batch mode
+ export SPARK_HOME=/Users/deroneriksson/spark-1.5.1-bin-hadoop2.6
+ $SPARK_HOME/bin/spark-submit SystemML.jar -s "print('hello world');" -exec hybrid_spark
+
+ # verify hadoop batch mode
+ hadoop jar SystemML.jar -s "print('hello world');"
+
+
+## Check LICENSE and NOTICE Files
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+Each artifact *must* contain LICENSE and NOTICE files. These files must reflect the
+contents of the artifacts. If the project dependencies (ie, libraries) have changed
+since the last release, the LICENSE and NOTICE files must be updated to reflect these
+changes.
+
+Each artifact *should* contain a DISCLAIMER file.
+
+For more information, see:
+
+1. <http://incubator.apache.org/guides/releasemanagement.html>
+2. <http://www.apache.org/dev/licensing-howto.html>
+
+
+## Src Artifact Builds and Tests Pass
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+The project should be built using the `src` (tar.gz and zip) artifacts.
+In addition, the test suite should be run using an `src` artifact and
+the tests should pass.
+
+ tar -xvzf systemml-0.10.0-incubating-src.tar.gz
+ cd systemml-0.10.0-incubating-src
+ mvn clean package -P distribution
+ mvn verify
+
+
+## Single-Node Standalone
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+The standalone tar.gz and zip artifacts contain `runStandaloneSystemML.sh` and `runStandaloneSystemML.bat`
+files. Verify that one or more algorithms can be run on a single node using these
+standalone distributions.
+
+Here is an example based on the [Quick Start Guide](http://apache.github.io/incubator-systemml/quick-start-guide.html)
+demonstrating the execution of an algorithm (on OS X).
+
+ $ tar -xvzf systemml-0.10.0-incubating-standalone.tar.gz
+ $ cd systemml-0.10.0-incubating-standalone
+ $ wget -P data/ http://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data
+ $ echo '{"rows": 306, "cols": 4, "format": "csv"}' > data/haberman.data.mtd
+ $ echo '1,1,1,2' > data/types.csv
+ $ echo '{"rows": 1, "cols": 4, "format": "csv"}' > data/types.csv.mtd
+ $ ./runStandaloneSystemML.sh scripts/algorithms/Univar-Stats.dml -nvargs X=data/haberman.data TYPES=data/types.csv STATS=data/univarOut.mtx CONSOLE_OUTPUT=TRUE
+
+
+## Single-Node Spark
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+Verify that SystemML runs algorithms on Spark locally.
+
+Here is an example of running the `Univar-Stats.dml` algorithm on random generated data.
+
+ $ tar -xvzf systemml-0.10.0-incubating.tar.gz
+ $ cd systemml-0.10.0-incubating
+ $ export SPARK_HOME=/Users/deroneriksson/spark-1.5.1-bin-hadoop2.6
+ $ $SPARK_HOME/bin/spark-submit SystemML.jar -f scripts/datagen/genRandData4Univariate.dml -exec hybrid_spark -args 1000000 100 10 1 2 3 4 uni.mtx
+ $ echo '1' > uni-types.csv
+ $ echo '{"rows": 1, "cols": 1, "format": "csv"}' > uni-types.csv.mtd
+ $ $SPARK_HOME/bin/spark-submit SystemML.jar -f scripts/algorithms/Univar-Stats.dml -exec hybrid_spark -nvargs X=uni.mtx TYPES=uni-types.csv STATS=uni-stats.txt CONSOLE_OUTPUT=TRUE
+
+
+## Single-Node Hadoop
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+Verify that SystemML runs algorithms on Hadoop locally.
+
+Based on the "Single-Node Spark" setup above, the `Univar-Stats.dml` algorithm could be run as follows:
+
+ $ hadoop jar SystemML.jar -f scripts/algorithms/Univar-Stats.dml -nvargs X=uni.mtx TYPES=uni-types.csv STATS=uni-stats.txt CONSOLE_OUTPUT=TRUE
+
+
+## Notebooks
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+Verify that SystemML can be executed from Jupyter and Zeppelin notebooks.
+For examples, see the [Spark MLContext Programming Guide](http://apache.github.io/incubator-systemml/spark-mlcontext-programming-guide.html).
+
+
+## Performance Suite
+
+<a href="#release-candidate-checklist">Up to Checklist</a>
+
+Verify that the performance suite located at scripts/perftest/ executes on Spark and Hadoop. Testing should
+include 80MB, 800MB, 8GB, and 80GB data sizes.
+
+
+
+