You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by re...@apache.org on 2017/12/21 19:42:44 UTC
systemml-website git commit: Add 1.0.0 release notes,
and updated roadmap
Repository: systemml-website
Updated Branches:
refs/heads/master 733cbd741 -> 2c1e7dfca
Add 1.0.0 release notes, and updated roadmap
Project: http://git-wip-us.apache.org/repos/asf/systemml-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml-website/commit/2c1e7dfc
Tree: http://git-wip-us.apache.org/repos/asf/systemml-website/tree/2c1e7dfc
Diff: http://git-wip-us.apache.org/repos/asf/systemml-website/diff/2c1e7dfc
Branch: refs/heads/master
Commit: 2c1e7dfca536dc2ae885ac527950cfebf07b84c9
Parents: 733cbd7
Author: Berthold Reinwald <re...@us.ibm.com>
Authored: Thu Dec 21 11:42:24 2017 -0800
Committer: Berthold Reinwald <re...@us.ibm.com>
Committed: Thu Dec 21 11:42:24 2017 -0800
----------------------------------------------------------------------
.../systemml-release-notes-1.0.0.md | 94 ++++++++++++
_src/roadmap.html | 143 +++++++++++++++----
2 files changed, 209 insertions(+), 28 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/systemml-website/blob/2c1e7dfc/_src/release-notes/systemml-release-notes-1.0.0.md
----------------------------------------------------------------------
diff --git a/_src/release-notes/systemml-release-notes-1.0.0.md b/_src/release-notes/systemml-release-notes-1.0.0.md
new file mode 100644
index 0000000..6e0ad0d
--- /dev/null
+++ b/_src/release-notes/systemml-release-notes-1.0.0.md
@@ -0,0 +1,94 @@
+---
+layout: page
+title: SystemML 1.0.0 Release Notes
+description: Project Release Notes
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<section class="full-stripe full-stripe--subpage-header clear-header">
+ <div class="ml-container ml-container--horizontally-center">
+ <div class="col col-12 content-group content-group--center-content content-group--center-align">
+ <h1>{{ site.data.project.name }} 1.0.0 Release Notes</h1>
+ </div>
+ </div>
+</section>
+
+<section class="full-stripe full-stripe--alternate">
+ <div class="ml-container">
+ <div class="col col-12 content-group content-group--medium-bottom-margin" markdown="1">
+
+The Apache SystemML 1.0.0 release was approved on December 12, 2017. The release includes enhancements, features, and additions as listed below.
+
+### API Enhancements
+- JMLC/MLContext parameter passing w/ and w/o meta data.
+
+### Deep Learning
+- Improved CPU convolution function performance, including sparsity.
+- `Caffe2DML`.
+- Enhanced/additional NN layers, e.g. conv2d transpose and depthwise convolution
+- NN architecture summary table.
+- [Experimental] `Keras2DML`.
+
+### New Scripts / Algorithms
+- Autoencoder.
+- Enhanced PCA.
+
+### Features & new Functions
+- `sinh`, `cosh`, and `tanh`.
+- `transformcolmap`.
+- n-ary `rbind`/`cbind`.
+- `order` with multiple order-by columns.
+
+### Compiler
+- Improved rewrites, e.g. merging of statement block sequences.
+- IPA effectiveness.
+- Automatic vectorization of indexing pairs.
+- Instruction generation for memory efficiency.
+- Instruction code organization.
+- [Experimental] Code generation, code gen optimizer, and multi-threaded codegen operators.
+
+### Robustness & Performance
+- ParFor/HOP Memory budgets for Spark cluster configurations.
+- JMLC prepared scripts (thread affinity for outputs and configs, script cloning, configuration management).
+- Sparse-dense binary cell wise operations.
+
+### [Experimental] GPU
+- GPU conv2d and memory estimates.
+- Additional kernels,e.g. right indexing.
+- Single precision backend.
+
+### Additional Packages
+- SystemML Lite artifact, a minimumm-size uber JAR for embeddability (w/o Hadoop or Spark dependencies).
+
+### Environment
+- Compression on by default.
+- Exploitation of native BLAS libraries.
+- JDK 8 (dropped JDK 7).
+- jCUDA for Windows & Linux (x86_64, ppc64le) included.
+- Fine-grained runtime statistics.
+- Refactored configurations parameters to have sysml prefix
+- Performance test suite.
+
+### Other
+- Refreshed/new examples and notebooks (DML examples), tutorial.
+
+### [JIRA release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12338328)
+- Features, Improvements, Bug fixes, etc.
http://git-wip-us.apache.org/repos/asf/systemml-website/blob/2c1e7dfc/_src/roadmap.html
----------------------------------------------------------------------
diff --git a/_src/roadmap.html b/_src/roadmap.html
index b0a2681..45ebca9 100644
--- a/_src/roadmap.html
+++ b/_src/roadmap.html
@@ -36,19 +36,75 @@ limitations under the License.
<section class="full-stripe full-stripe--alternate">
<div class="ml-container ml-container--narrow content-group">
<div class="col col-12">
- <h2>Planned for Future SystemML 1.0</h2>
+ <h2>Planned for Future SystemML 1.1</h2>
<ul>
- <li>Compression (additional operations)</li>
- <li>Experimental Features:
+ <li>Algorithms & builtin functions
<ul>
- <li>Deep Learning</li>
- <li>Single GPU support</li>
- <li>Native BLAS support</li>
- <li>Code Generation</li>
+ <li>NN layers-based factorization machines with regression & classification capabilities</li>
+ <li>NN optimization test suite with well known optimization test functions</li>
+ <li>Model selection & hyper parameter tuning</li>
+ <li>Additional distribution functions, e.g. weibull, gamma</li>
+ <li>Generalization of operations, such as xor, and other operations</li>
+ </ul>
+ </li>
+ <li>Enhanced Deep Learning support
+ <ul>
+ <li>Coherent sparse operations on CPU/GPU</li>
+ <li>Coherent single-precision support on CPU/GPU</li>
+ <li>Distributed DL operations</li>
+ </ul>
+ </li>
+ <li>GPU Support
+ <ul>
+ <li>Full compiler integration (cost-based, automatic placement)</li>
+ <li>Multi GPUs</li>
+ <li>Distributed GPUs</li>
+ </ul>
+ </li>
+ <li>Code generation
+ <ul>
+ <li>Deep learning operationss</li>
+ <li>Heterogeneous HW, incl GPUs</li>
</ul>
</li>
- <li>Rigorous Performance and Scalability Testing (Bug Fixes)</li>
- <li>Remove Deprecated APIs/Functions</li>
+ <li>Compressed Linear Algebra
+ <ul>
+ <li>Matrix-matrix multiplications</li>
+ <li>Deep learning operations</li>
+ <li>Ultra-sparse datasets</li>
+ </ul>
+ </li>
+ <li>Misc Runtime
+ <ul>
+ <li>Large dense matrix blocks > 16GB</li>
+ <li>NUMA-awareness (thread pools, matrix partitioning)</li>
+ <li>Unified memory management (ops, bufferpool, RDDs/broadcasts)</li>
+ <li>Support additional external formats such as feather format for matrices and frames</li>
+ <li>Parfor support for broadcasts</li>
+ <li>Extended support for multi-threaded operations</li>
+ <li>Boolean matrices</li>
+ </ul>
+ </li>
+ <li>Misc Compiler
+ <ul>
+ <li>Support single-output UDFs in expressions</li>
+ <li>Consolidate replicated compilation chain (e.g., diff APIs)</li>
+ <li>Holistic sum-product optimization and operator fusion</li>
+ <li>Extended sparsity estimators</li>
+ <li>Rewrites and compiler improvements for mini-batching including prefetching</li>
+ <li>Parfor optimizer support for shared reads</li>
+ <li>SPOOF compiler improvement</li>
+ </ul>
+ </li>
+ <li>APIs
+ <ul>
+ <li>Python Binding for JMLC API</li>
+ <li>Consistency Python/Java APIs</li>
+ </ul>
+
+ </li>
+
+
</ul>
</div>
@@ -72,6 +128,40 @@ limitations under the License.
<h2>Current Release</h2>
<ul>
<li>
+ <strong>SystemML 1.0.0 (<a href="release-notes/systemml-release-notes-1.0.0.html">released</a> in December, 2017)
+ <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12338328">details</a></strong>
+ <ul>
+ <li>Enhanced Deep Learning support with enhanced NN layers and functions, Caffe2DML, and operator implementation</li>
+ <li>Native BLAS support</li>
+ <li>Additional algorithms: autoencoder, enhanced PCA</li>
+ <li>Enhanced rewrites, IPA, vectorization, and instruction generation</li>
+ <li>Enhanced JMLC API, e.g. prepared scripts with thread affinity for outputs and configs, script cloning, configuration management</li>
+ <li>SystemML Lite artifact</li>
+ <li>Compression on by default</li>
+ </ul>
+ </li>
+ <li>
+ Experimental Features
+ <ul>
+ <li>Keras2DML.</li>
+ <li>Enhanced code generation, code gen optimizer, and multi-threaded codegen operators/li>
+ <li>Enhanced GPU support</li>
+ </ul>
+ </li>
+ <li>
+ Removals
+ <ul>
+ <li>Dropped JDK 7 support</li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+
+ <div class="col col-12">
+ <h2>Prior Releases</h2>
+
+ <ul>
+ <li>
<strong>SystemML 0.15.0 (<a href="release-notes/systemml-release-notes-0.15.0.html">released</a> in September, 2017)
<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12341587">details</a></strong>
<ul>
@@ -94,10 +184,6 @@ limitations under the License.
</ul>
</li>
</ul>
- </div>
-
- <div class="col col-12">
- <h2>Prior Releases</h2>
<ul>
<li>
@@ -111,8 +197,9 @@ limitations under the License.
<li>
Experimental Features
<ul>
- <li>New Code Generation capabilities for automatic operator fusion (basic code generator, compiler integration, runtime integration, in-memory source code compilation, extended explain tool, support for right indexing and replace in cellwise and row aggregate templates, support for row, column, or no aggregation in rowwise template).
- Note code generation provides significant performance gains with fewer read/write intermediates, reduced scans of inputs and intermediates, and enhanced sparsity exploitation. To enable this feature, set codegen.enabled property to true in SystemML-config.xml file.</li>
+ <li>New Code Generation capabilities for automatic operator fusion (basic code generator, compiler integration, runtime integration, in-memory source code compilation, extended explain tool, support for right indexing and replace in cellwise and
+ row aggregate templates, support for row, column, or no aggregation in rowwise template). Note code generation provides significant performance gains with fewer read/write intermediates, reduced scans of inputs and intermediates, and enhanced
+ sparsity exploitation. To enable this feature, set codegen.enabled property to true in SystemML-config.xml file.</li>
<li>New instructions and operators for GPU support (relu_maxpooling, conv2d_bias_add, bias_multiply)</li>
</ul>
</li>
@@ -131,30 +218,30 @@ limitations under the License.
<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12339247">details</a></strong>
<ul>
<li>Updated build for Spark 2.1.0</li>
- <li>New simplification rewrites for stratstats</li>
- <li>New fused operator tack+* in CP and Spark</li>
+ <li>New simplification rewrites for stratstats</li>
+ <li>New fused operator tack+* in CP and Spark</li>
<li>New dmlFromResource capability in Python (equivalent to Scala)</li>
- <li>Add input float support to MLContext</li>
+ <li>Add input float support to MLContext</li>
</ul>
</li>
<li>
Documentation Enhancements
<ul>
<li>Deploy versioned documentation to main project website</li>
- <li>Add python mlcontext example to engine dev guide</li>
- <li>Add MLContext info functionality to docs</li>
- <li>Update DML Language Reference for write description parameter</li>
+ <li>Add python mlcontext example to engine dev guide</li>
+ <li>Add MLContext info functionality to docs</li>
+ <li>Update DML Language Reference for write description parameter</li>
</ul>
</li>
<li>
Deprecations, Removals
<ul>
<li>Deprecate old MLContext API</li>
- <li>Deprecate parfor perftesttool</li>
- <li>Deprecate SQLContext methods</li>
- <li>Replace deprecated Accumulator with AccumulatorV2</li>
- <li>Replace append with cbind for matrices</li>
- <li>Migrate Vector and LabeledPoint classes from mllib to ml</li>
+ <li>Deprecate parfor perftesttool</li>
+ <li>Deprecate SQLContext methods</li>
+ <li>Replace deprecated Accumulator with AccumulatorV2</li>
+ <li>Replace append with cbind for matrices</li>
+ <li>Migrate Vector and LabeledPoint classes from mllib to ml</li>
</ul>
</li>
<li>
@@ -173,8 +260,8 @@ limitations under the License.
<li>Support pip install of new python package</li>
<li>Allow NumPy arrays, Pandas DataFrame and SciPy matrices as input to MLContext</li>
<li>Improve SystemML Python DSL for NumPy</li>
- <li>Updated build for Spark 1.6.0</li>
- <li>DML utility script to shuffle input dataset</li>
+ <li>Updated build for Spark 1.6.0</li>
+ <li>DML utility script to shuffle input dataset</li>
</ul>
</li>
<li>