You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by ni...@apache.org on 2017/09/21 19:29:48 UTC
[1/4] systemml git commit: [SYSTEMML-1819] Added new Keras2DML
frontend
Repository: systemml
Updated Branches:
refs/heads/gh-pages dbe9a03b2 -> e3f0cf404
[SYSTEMML-1819] Added new Keras2DML frontend
This adds a new "Keras2DML" frontend to SystemML, built upon the
Caffe2DML infrastructure, that allows users to define (and even train)
models in Keras and then import them into SystemML for distributed
training and prediction.
More specifically, a new `Keras2DML` class accepts a Keras `Model`
object, converts it in the background to a Caffe model using the Python
Caffe API in a `keras2caffe.py` file, and then internally uses Caffe2DML
to transform the model into DML. From the user's perspective, no Caffe
interaction is necessary.
This API is still experimental at this point.
Closes #622.
Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/c95108c4
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/c95108c4
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/c95108c4
Branch: refs/heads/gh-pages
Commit: c95108c4db39be229fba1847987d8c4a58e0453a
Parents: dbe9a03
Author: FuturizeHandgun <19...@gmail.com>
Authored: Tue Sep 19 17:20:57 2017 -0700
Committer: Mike Dusenberry <mw...@us.ibm.com>
Committed: Tue Sep 19 17:20:57 2017 -0700
----------------------------------------------------------------------
beginners-guide-keras2dml.md | 62 +++++++++++++++++++++++++++++++++++++++
index.md | 2 +-
2 files changed, 63 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/systemml/blob/c95108c4/beginners-guide-keras2dml.md
----------------------------------------------------------------------
diff --git a/beginners-guide-keras2dml.md b/beginners-guide-keras2dml.md
new file mode 100644
index 0000000..fd2af87
--- /dev/null
+++ b/beginners-guide-keras2dml.md
@@ -0,0 +1,62 @@
+---
+layout: global
+title: Beginner's Guide for Keras2DML users
+description: Beginner's Guide for Keras2DML users
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+* This will become a table of contents (this text will be scraped).
+{:toc}
+
+<br/>
+
+## Introduction
+
+Keras2DML is an **experimental API** that converts a Keras specification to DML through the intermediate Caffe2DML module.
+It is designed to fit well into the mllearn framework and hence supports NumPy, Pandas as well as PySpark DataFrame.
+
+### Getting Started
+
+To create a Keras2DML object, one needs to create a Keras model through the Funcitonal API. please see the [Functional API.](https://keras.io/models/model/)
+This module utilizes the existing [Caffe2DML](beginners-guide-caffe2dml) backend to convert Keras models into DML. Keras models are
+parsed and translated into Caffe prototext and caffemodel files which are then piped into Caffe2DML. Thus one can follow the Caffe2DML
+documentation for further information.
+
+### Model Conversion
+
+Keras models are parsed based on their layer structure and corresponding weights and translated into the relative Caffe layer and weight
+configuration. Be aware that currently this is a translation into Caffe and there will be loss of information from keras models such as
+intializer information, and other layers which do not exist in Caffe.
+
+To create a Keras2DML object, simply pass the keras object to the Keras2DML constructor. It's also important to note that your models
+should be compiled so that the loss can be accessed for Caffe2DML
+
+```python
+from systemml.mllearn import Keras2DML
+import keras
+from keras.applications.resnet50 import preprocess_input, decode_predictions, ResNet50
+
+model = ResNet50(weights='imagenet',include_top=True,pooling='None',input_shape=(224,224,3))
+model.compile(optimizer='sgd', loss= 'categorical_crossentropy')
+
+resnet = Keras2DML(spark,model,input_shape=(3,224,224))
+resnet.summary()
+```
+
http://git-wip-us.apache.org/repos/asf/systemml/blob/c95108c4/index.md
----------------------------------------------------------------------
diff --git a/index.md b/index.md
index 96b6b2a..d1dded7 100644
--- a/index.md
+++ b/index.md
@@ -51,7 +51,7 @@ for running SystemML from Spark via Scala, Python, or Java.
machine in R-like and Python-like declarative languages.
* [JMLC](jmlc) - Java Machine Learning Connector.
* *Experimental* [Caffe2DML API](beginners-guide-caffe2dml) for Deep Learning.
-
+* *Experimental* [Keras2DML API](beginners-guide-keras2dml) for Deep Learning.
## Language Guides
* [Python API Reference](python-reference) - API Reference Guide for Python users.
[4/4] systemml git commit: [SYSTEMML-1927] New frame transformcolmap
builtin function
Posted by ni...@apache.org.
[SYSTEMML-1927] New frame transformcolmap builtin function
Our decision tree script requires dummy coded inputs of all categorical
attributes as well as a mapping matrix of their column positions. This
patch introduces a new frame builtin function that leverages the
transformencode meta data in order to automatically compute the mapping
matrix. Since this is a meta data operation, it is only implemented in
CP (but it requires the meta data frame to fit into memory).
Furthermore, this also includes a respective test case and the
documentation update.
This changes enables a future simplification of decision tree and random
forest, where we could do any pre-processing automatically inside the
script instead of requiring the user to do it manually.
Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/e3f0cf40
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/e3f0cf40
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/e3f0cf40
Branch: refs/heads/gh-pages
Commit: e3f0cf4041d6591632a0a7a7c5b6aa279203aeca
Parents: 27e06a5
Author: Matthias Boehm <mb...@gmail.com>
Authored: Thu Sep 21 00:55:04 2017 -0700
Committer: Matthias Boehm <mb...@gmail.com>
Committed: Thu Sep 21 11:54:11 2017 -0700
----------------------------------------------------------------------
dml-language-reference.md | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/systemml/blob/e3f0cf40/dml-language-reference.md
----------------------------------------------------------------------
diff --git a/dml-language-reference.md b/dml-language-reference.md
index d8ca07f..c402acc 100644
--- a/dml-language-reference.md
+++ b/dml-language-reference.md
@@ -1642,6 +1642,7 @@ Function | Description | Parameters | Example
transformencode() | Transforms a frame into a matrix using specification. <br/> Builds and applies frame metadata. | Input:<br/> target = <frame> <br/> spec = <json specification> <br/> Outputs: <matrix>, <frame>|[transformencode](dml-language-reference.html#transformencode)
transformdecode() | Transforms a matrix into a frame using specification. <br/> Valid only for specific transformation types. | Input:<br/> target = <matrix> <br/> spec = <json specification> <br/> meta = <frame> <br/> Output: <frame> |[transformdecode](dml-language-reference.html#transformdecode)
transformapply() | Transforms a frame into a matrix using specification. <br/> Applies existing frame metadata. | Input:<br/> target = <frame> <br/> spec = <json specification> <br/> meta = <frame> <br/> Output: <matrix> | [transformapply](dml-language-reference.html#transformapply)
+transformcolmap() | Obtains the column mapping of a transformed frame using the given specification. The input frame is assumed to be the meta data frame returned from a transformencode call. <br/> The output has a row per encoded input attribute, indicating the source column position, as well as the start and end positions in the encode output. | Input:<br/> target = <frame> <br/> spec = <json specification> <br/> Output: <matrix> |[transformcolmap](dml-language-reference.html#transformdecode)
The following table summarizes the supported transformations for <code>transformencode(), transformdecode(), transformapply()</code>. Note only recoding, dummy coding and pass-through are reversible, i.e., subject to <code>transformdecode()</code>, whereas binning, missing value imputation, and omit are not.
[3/4] systemml git commit: [SYSTEMML-1925] Fix consistency SystemML
configuration properties
Posted by ni...@apache.org.
[SYSTEMML-1925] Fix consistency SystemML configuration properties
This patch creates consistency in terms of a common prefix for all
configuration properties. Before we used the prefixes dml, systemml, or
none, which is now consolidated to sysml.
Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/27e06a56
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/27e06a56
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/27e06a56
Branch: refs/heads/gh-pages
Commit: 27e06a56d8549e7e85d5f02a84cb7eb761eb01a4
Parents: d74abab
Author: Matthias Boehm <mb...@gmail.com>
Authored: Wed Sep 20 22:17:50 2017 -0700
Committer: Matthias Boehm <mb...@gmail.com>
Committed: Thu Sep 21 11:54:09 2017 -0700
----------------------------------------------------------------------
beginners-guide-caffe2dml.md | 8 ++++----
standalone-guide.md | 24 ++++++++++++------------
2 files changed, 16 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/systemml/blob/27e06a56/beginners-guide-caffe2dml.md
----------------------------------------------------------------------
diff --git a/beginners-guide-caffe2dml.md b/beginners-guide-caffe2dml.md
index 7671c32..12b21d4 100644
--- a/beginners-guide-caffe2dml.md
+++ b/beginners-guide-caffe2dml.md
@@ -152,8 +152,8 @@ Iter:2000, validation loss:173.66147359346, validation accuracy:97.4897540983606
- Print the generated DML script along with classification report: `lenet.set(debug=True)`
- Print the heavy hitters instruction and the execution plan (advanced users): `lenet.setStatistics(True).setExplain(True)`
-- (Optional but recommended) Enable [native BLAS](http://apache.github.io/systemml/native-backend): `lenet.setConfigProperty("native.blas", "auto")`
-- Enable experimental feature such as codegen: `lenet.setConfigProperty("codegen.enabled", "true").setConfigProperty("codegen.plancache", "true")`
+- (Optional but recommended) Enable [native BLAS](http://apache.github.io/systemml/native-backend): `lenet.setConfigProperty("sysml.native.blas", "auto")`
+- Enable experimental feature such as codegen: `lenet.setConfigProperty("sysml.codegen.enabled", "true").setConfigProperty("sysml.codegen.plancache", "true")`
- Force GPU execution (please make sure the required jcuda dependency are included): lenet.setGPU(True).setForceGPU(True)
Unlike Caffe where default train and test algorithm is `minibatch`, you can specify the
@@ -242,13 +242,13 @@ If you are using OpenBLAS, please ensure that it was built with `USE_OPENMP` fla
For more detail see http://apache.github.io/systemml/native-backend
```python
-caffe2dmlObject.setConfigProperty("native.blas", "auto")
+caffe2dmlObject.setConfigProperty("sysml.native.blas", "auto")
```
- Turn on the experimental codegen feature. This should help reduce unnecessary allocation cost after every binary operation.
```python
-caffe2dmlObject.setConfigProperty("codegen.enabled", "true").setConfigProperty("codegen.plancache", "true")
+caffe2dmlObject.setConfigProperty("sysml.codegen.enabled", "true").setConfigProperty("sysml.codegen.plancache", "true")
```
- Tuned the [Garbage Collector](http://spark.apache.org/docs/latest/tuning.html#garbage-collection-tuning).
http://git-wip-us.apache.org/repos/asf/systemml/blob/27e06a56/standalone-guide.md
----------------------------------------------------------------------
diff --git a/standalone-guide.md b/standalone-guide.md
index a401c30..7116f25 100644
--- a/standalone-guide.md
+++ b/standalone-guide.md
@@ -324,18 +324,18 @@ The `l2-svm-predict.dml` algorithm is used on our test data sample to predict th
The console output should show the accuracy of the trained model in percent, i.e.:
15/09/01 01:32:51 INFO api.DMLScript: BEGIN DML run 09/01/2015 01:32:51
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating localtmpdir with value /tmp/systemml
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating scratch with value scratch_space
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating optlevel with value 2
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating numreducers with value 10
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating jvmreuse with value false
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating defaultblocksize with value 1000
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating dml.yarn.appmaster with value false
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating dml.yarn.appmaster.mem with value 2048
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating dml.yarn.mapreduce.mem with value 2048
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating dml.yarn.app.queue with value default
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating cp.parallel.ops with value true
- 15/09/01 01:32:51 INFO conf.DMLConfig: Updating cp.parallel.io with value true
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.localtmpdir with value /tmp/systemml
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.scratch with value scratch_space
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.optlevel with value 2
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.numreducers with value 10
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.jvmreuse with value false
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.defaultblocksize with value 1000
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.yarn.appmaster with value false
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.yarn.appmaster.mem with value 2048
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.yarn.mapreduce.mem with value 2048
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.yarn.app.queue with value default
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.parallel.ops with value true
+ 15/09/01 01:32:51 INFO conf.DMLConfig: Updating sysml.parallel.io with value true
Accuracy (%): 74.14965986394557
15/09/01 01:32:52 INFO api.DMLScript: SystemML Statistics:
Total execution time: 0.130 sec.
[2/4] systemml git commit: [SYSTEMML-1923] Support sinh,
cosh and tanh as built-in functions
Posted by ni...@apache.org.
[SYSTEMML-1923] Support sinh, cosh and tanh as built-in functions
- Added sinh, cosh and tanh builtin functions to CP, Spark and GPU backend.
- Added these functions to Python DSL.
- Also, updated the DML language reference and Python reference documentation.
Closes #668.
Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/d74abab1
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/d74abab1
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/d74abab1
Branch: refs/heads/gh-pages
Commit: d74abab185bb60f4a3c7b88f145a5523031d9179
Parents: c95108c
Author: Niketan Pansare <np...@us.ibm.com>
Authored: Thu Sep 21 10:06:04 2017 -0700
Committer: Niketan Pansare <np...@us.ibm.com>
Committed: Thu Sep 21 10:07:15 2017 -0700
----------------------------------------------------------------------
dml-language-reference.md | 2 +-
python-reference.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/systemml/blob/d74abab1/dml-language-reference.md
----------------------------------------------------------------------
diff --git a/dml-language-reference.md b/dml-language-reference.md
index bd66a42..d8ca07f 100644
--- a/dml-language-reference.md
+++ b/dml-language-reference.md
@@ -822,7 +822,7 @@ is same as
Function | Description | Parameters | Example
-------- | ----------- | ---------- | -------
exp(), log(), abs(), sqrt(), round(), floor(), ceil() | Apply mathematical function on input (cell wise if input is matrix) | Input: (<matrix>), or (<scalar>) <br/> Output: <matrix>, or <scalar> | sqrt(X) <br/> log(X,y) <br/> round(X) <br/> floor(X) <br/> ceil(X)
-sin(), cos(), tan(), asin(), acos(), atan() | Apply trigonometric function on input (cell wise if input is matrix) | Input: (<matrix>), or (<scalar>) <br/> Output: <matrix>, or <scalar> | sin(X)
+sin(), cos(), tan(), sinh(), cosh(), tanh(), asin(), acos(), atan() | Apply trigonometric function on input (cell wise if input is matrix) | Input: (<matrix>), or (<scalar>) <br/> Output: <matrix>, or <scalar> | sin(X)
sign() | Returns a matrix representing the signs of the input matrix elements, where 1 represents positive, 0 represents zero, and -1 represents negative | Input : (A <matrix>) <br/> Output : <matrix> | <span style="white-space: nowrap;">A = matrix("-5 0 3 -3",</span> rows=2, cols=2) <br/>signA = sign(A)<br/>Matrix signA: [[-1, 0], [1, -1]]
http://git-wip-us.apache.org/repos/asf/systemml/blob/d74abab1/python-reference.md
----------------------------------------------------------------------
diff --git a/python-reference.md b/python-reference.md
index 119c1d0..4fd78fe 100644
--- a/python-reference.md
+++ b/python-reference.md
@@ -191,7 +191,7 @@ In addition to the above mentioned operators, following functions are supported.
| argmax(self, axis=None) | Returns the indices of the maximum values along an axis. | axis : int, optional (only axis=1, i.e. rowIndexMax is supported in this version) |
| cumsum(self, axis=None) | Returns the indices of the maximum values along an axis. | axis : int, optional (only axis=0, i.e. cumsum along the rows is supported in this version) |
-- Global statistical built-In functions: exp, log, abs, sqrt, round, floor, ceil, sin, cos, tan, asin, acos, atan, sign, solve
+- Global statistical built-In functions: exp, log, abs, sqrt, round, floor, ceil, sin, cos, tan, sinh, cosh, tanh, asin, acos, atan, sign, solve
| | Description | Parameters |
|------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|