You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by Deron Eriksson <de...@gmail.com> on 2016/06/13 20:29:06 UTC
Release notes for 0.10.0
Hi,
We need to put together a page of release notes for the 0.10.0 release
(similar to http://systemml.apache.org/0.9.0-incubating/release_notes.html).
Could SystemML committers and contributors please respond to this thread
with what they feel are the major updates/improvements in this release?
Thanks!
Deron
Re: Release notes for 0.10.0
Posted by Deron Eriksson <de...@gmail.com>.
Matthias, thank you for all the hard work preparing the 0.10.0 release
notes. I've updated the website with the release notes at:
http://systemml.apache.org/0.10.0-incubating/release_notes.html
Deron
On Tue, Jun 14, 2016 at 12:39 AM, Matthias Boehm <mb...@us.ibm.com> wrote:
> the sub-bullet "c - fixes backwards compatibility MRv1" also referred to
> (1) guava dependency conflicts (for MRv1, the classpath has a very old
> version), and (2) a couple of runtime changes such as SYSTEMML-513 (task
> handling for multiple output committer which overwrites the default file
> output committer).
>
> Regards,
> Matthias
>
> [image: Inactive hide details for Berthold Reinwald---06/14/2016 12:18:09
> AM---this color Regards,]Berthold Reinwald---06/14/2016 12:18:09
> AM---this color Regards,
>
> From: Berthold Reinwald/Almaden/IBM@IBMUS
> To: dev@systemml.incubator.apache.org
> Date: 06/14/2016 12:18 AM
> Subject: Re: Release notes for 0.10.0
> ------------------------------
>
>
>
> this color
>
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinwald@us.ibm.com
>
>
>
> From: Matthias Boehm/Almaden/IBM@IBMUS
> To: dev@systemml.incubator.apache.org
> Date: 06/13/2016 11:25 PM
> Subject: Re: Release notes for 0.10.0
>
>
>
> ok, here is a first draft of the release notes - please feel free to
> extend or prune this:
>
> a) [SYSTEMML-377] Different internal implementation types of spark matrix
> blocks
> * Supported internal formats: MCSR (default), CSR, COO
> * Automatic MCSR->CSR on Spark read/caching (for memory efficiency)
> * Automatic MCSR->CSR on sparse update-in-place (avoid serialization)
>
> b) [SYSTEMML-554] Enhanced SystemML Frame support for JMLC API / CP
> * SystemML frame data type, deeply integrated into compiler and runtime
> * New builtin functions: transform, transformapply, transformencode,
> transformdecode, transformmeta
> * Supported operations: read/write, left/right indexing, casting, append,
> transform/transformapply
>
> c) Framework compatibility / configuration
> * [SYSTEMML-418] Version-specific Spark memory budgets (>=1.6, legacy)
> * [SYSTEMML-158] Updated deprecated Hadoop properties
> * [SYSTEMML-476] Version-specific MR configuration handling (MRv2, MRv1)
> * Fixes for backwards compatibility to MRv1 (?? shouldn't this be in -158
> ??)
> * New pass-through mapred/mapreduce configurations through SystemML-config
> * [SYSTEMML-584/585] New thread-local configuration handling (compiler/dml
> config)
>
> d) Deep learning support (experimental)
> * [SYSTMEML-618] New DML-bodied function library for deep learning
> * [SYSTEMML-540] New builtin singlenode operations: conv2d, maxpooling,
> im2col, col2im, rotate
> * New lenet-train DML script (staging)
>
> e) API/script usability
> * [SYSTEMML-607/604/611] Parser error handling
> * [SYSTEMML-506/508/544/577/649/651] Extended MLContext/JMLC APIs
> * [SYSTEMML-625/626/632] Improved source statement handling (e.g.,
> imports, absolute paths)
> * [SYSTEMML-617/631/654] Improved name space handling
> * [SYSTEMML-240] Extended stats outputs for Spark
> collect/broadcast/parallelize
> * [SYSTEMML-495] SystemML configuration handling
> * [SYSTEMML-209] Include algorithms in systemml jar
> * [SYSTEMML-647/648] Deprecated castAsScalar, ppred
> * [SYSTEMML-477] JSON meta data handling
> * [SYSTEMML-294] Print matrix builtin function
> * [SYSTEMML-296/676/670] Improved PyDML syntax: slicing, rand, cdf, elif
> * [SYSTEMML-675] Support for negative for/parfor loop increments
>
> f) New fused physical operators
> * [SYSTEMML-488] Fused wdivmm w/ 4 operands
> * [SYSTEMML-510] Fused wdivmm/wcemm w/ eps term
>
> g) Various performance features
> * [SYSTEMML-427/512] Extended IPA (propagate scalar variables)
> * [SYSTEMML-282] Extended update-in-place support for parfor intermediates
>
> * [SYSTEMML-552/399] Performance parallel binary/text readers (sort
> sparse/nnz handling)
> * [SYSTEMML-552/641] Cache-conscious operations: sparse-dense
> wdivmm/wsloss, sparse-dense/sparse-sparse mm, dense-dense skinny rhs mm
> * [SYSTEMML-641] Tuned special cases for block matrix multiplication:
> e.g., mm w/ skinny rhs, colwise parallelization wide rhs
> * [SYSTEMML-396/400] New/extended multithreaded operations:
> cumsum/cummin/cummax/cumprod, transpose, and rand
> * [SYSTEMML-510/694] New simplification rewrites: "pushdown
> unaryagg-transpose", "simplify transpose-aggbin-binary chains", "reorder
> minus-mmult", "canonicalize matmult-add-scalar", improved constant folding
> (all unary)
> * [SYSTEMML-653] Asynchronous bufferpool cleanup of evicted files / nio
> file eviction
> - MR iqm/quantile/median (qsort num reducers, qpick buffer size)
>
> h) DML script updates
> * [SYSTEMML-536] New KNN algorithm (still staging)
> * [SYSTEMML-534] Optional console output univariate statistics
> * [SYSTMEML-494] GLM compiler warnings
> * Robustness input/output handling L2SVM, MSVM, and Naive Bayes
> * Random data generator for ALS
>
> i) Various fixes
> - Dozens of fixes for diverse issues, fix pack for 0.9 release
>
> j) Build, documentation, examples
> * [SYSTEMML-551] Enhanced jmlc javadoc
> * [SYSTEMML-484] Build javadoc jar
> * [SYSTEMML-468] Contributing to SystemML doc
> * [SYSTEMML-517/524] DML language ref updates
> * [SYSTEMML-498] Trouble shooting guide
> * SystemML Jupyter/Zeppelin notebook examples
>
>
> Regards,
> Matthias
>
> Deron Eriksson ---06/13/2016 01:29:16 PM---Hi, We need to put together a
> page of release notes for the 0.10.0 release
>
> From: Deron Eriksson <de...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 06/13/2016 01:29 PM
> Subject: Release notes for 0.10.0
>
>
>
> Hi,
>
> We need to put together a page of release notes for the 0.10.0 release
> (similar to http://systemml.apache.org/0.9.0-incubating/release_notes.html
> ).
> Could SystemML committers and contributors please respond to this thread
> with what they feel are the major updates/improvements in this release?
>
> Thanks!
> Deron
>
>
>
>
>
>
>
>
Re: Release notes for 0.10.0
Posted by Matthias Boehm <mb...@us.ibm.com>.
the sub-bullet "c - fixes backwards compatibility MRv1" also referred to
(1) guava dependency conflicts (for MRv1, the classpath has a very old
version), and (2) a couple of runtime changes such as SYSTEMML-513 (task
handling for multiple output committer which overwrites the default file
output committer).
Regards,
Matthias
From: Berthold Reinwald/Almaden/IBM@IBMUS
To: dev@systemml.incubator.apache.org
Date: 06/14/2016 12:18 AM
Subject: Re: Release notes for 0.10.0
this color
Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinwald@us.ibm.com
From: Matthias Boehm/Almaden/IBM@IBMUS
To: dev@systemml.incubator.apache.org
Date: 06/13/2016 11:25 PM
Subject: Re: Release notes for 0.10.0
ok, here is a first draft of the release notes - please feel free to
extend or prune this:
a) [SYSTEMML-377] Different internal implementation types of spark matrix
blocks
* Supported internal formats: MCSR (default), CSR, COO
* Automatic MCSR->CSR on Spark read/caching (for memory efficiency)
* Automatic MCSR->CSR on sparse update-in-place (avoid serialization)
b) [SYSTEMML-554] Enhanced SystemML Frame support for JMLC API / CP
* SystemML frame data type, deeply integrated into compiler and runtime
* New builtin functions: transform, transformapply, transformencode,
transformdecode, transformmeta
* Supported operations: read/write, left/right indexing, casting, append,
transform/transformapply
c) Framework compatibility / configuration
* [SYSTEMML-418] Version-specific Spark memory budgets (>=1.6, legacy)
* [SYSTEMML-158] Updated deprecated Hadoop properties
* [SYSTEMML-476] Version-specific MR configuration handling (MRv2, MRv1)
* Fixes for backwards compatibility to MRv1 (?? shouldn't this be in -158
??)
* New pass-through mapred/mapreduce configurations through SystemML-config
* [SYSTEMML-584/585] New thread-local configuration handling (compiler/dml
config)
d) Deep learning support (experimental)
* [SYSTMEML-618] New DML-bodied function library for deep learning
* [SYSTEMML-540] New builtin singlenode operations: conv2d, maxpooling,
im2col, col2im, rotate
* New lenet-train DML script (staging)
e) API/script usability
* [SYSTEMML-607/604/611] Parser error handling
* [SYSTEMML-506/508/544/577/649/651] Extended MLContext/JMLC APIs
* [SYSTEMML-625/626/632] Improved source statement handling (e.g.,
imports, absolute paths)
* [SYSTEMML-617/631/654] Improved name space handling
* [SYSTEMML-240] Extended stats outputs for Spark
collect/broadcast/parallelize
* [SYSTEMML-495] SystemML configuration handling
* [SYSTEMML-209] Include algorithms in systemml jar
* [SYSTEMML-647/648] Deprecated castAsScalar, ppred
* [SYSTEMML-477] JSON meta data handling
* [SYSTEMML-294] Print matrix builtin function
* [SYSTEMML-296/676/670] Improved PyDML syntax: slicing, rand, cdf, elif
* [SYSTEMML-675] Support for negative for/parfor loop increments
f) New fused physical operators
* [SYSTEMML-488] Fused wdivmm w/ 4 operands
* [SYSTEMML-510] Fused wdivmm/wcemm w/ eps term
g) Various performance features
* [SYSTEMML-427/512] Extended IPA (propagate scalar variables)
* [SYSTEMML-282] Extended update-in-place support for parfor intermediates
* [SYSTEMML-552/399] Performance parallel binary/text readers (sort
sparse/nnz handling)
* [SYSTEMML-552/641] Cache-conscious operations: sparse-dense
wdivmm/wsloss, sparse-dense/sparse-sparse mm, dense-dense skinny rhs mm
* [SYSTEMML-641] Tuned special cases for block matrix multiplication:
e.g., mm w/ skinny rhs, colwise parallelization wide rhs
* [SYSTEMML-396/400] New/extended multithreaded operations:
cumsum/cummin/cummax/cumprod, transpose, and rand
* [SYSTEMML-510/694] New simplification rewrites: "pushdown
unaryagg-transpose", "simplify transpose-aggbin-binary chains", "reorder
minus-mmult", "canonicalize matmult-add-scalar", improved constant folding
(all unary)
* [SYSTEMML-653] Asynchronous bufferpool cleanup of evicted files / nio
file eviction
- MR iqm/quantile/median (qsort num reducers, qpick buffer size)
h) DML script updates
* [SYSTEMML-536] New KNN algorithm (still staging)
* [SYSTEMML-534] Optional console output univariate statistics
* [SYSTMEML-494] GLM compiler warnings
* Robustness input/output handling L2SVM, MSVM, and Naive Bayes
* Random data generator for ALS
i) Various fixes
- Dozens of fixes for diverse issues, fix pack for 0.9 release
j) Build, documentation, examples
* [SYSTEMML-551] Enhanced jmlc javadoc
* [SYSTEMML-484] Build javadoc jar
* [SYSTEMML-468] Contributing to SystemML doc
* [SYSTEMML-517/524] DML language ref updates
* [SYSTEMML-498] Trouble shooting guide
* SystemML Jupyter/Zeppelin notebook examples
Regards,
Matthias
Deron Eriksson ---06/13/2016 01:29:16 PM---Hi, We need to put together a
page of release notes for the 0.10.0 release
From: Deron Eriksson <de...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 06/13/2016 01:29 PM
Subject: Release notes for 0.10.0
Hi,
We need to put together a page of release notes for the 0.10.0 release
(similar to http://systemml.apache.org/0.9.0-incubating/release_notes.html
).
Could SystemML committers and contributors please respond to this thread
with what they feel are the major updates/improvements in this release?
Thanks!
Deron
Re: Release notes for 0.10.0
Posted by Berthold Reinwald <re...@us.ibm.com>.
this color
Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinwald@us.ibm.com
From: Matthias Boehm/Almaden/IBM@IBMUS
To: dev@systemml.incubator.apache.org
Date: 06/13/2016 11:25 PM
Subject: Re: Release notes for 0.10.0
ok, here is a first draft of the release notes - please feel free to
extend or prune this:
a) [SYSTEMML-377] Different internal implementation types of spark matrix
blocks
* Supported internal formats: MCSR (default), CSR, COO
* Automatic MCSR->CSR on Spark read/caching (for memory efficiency)
* Automatic MCSR->CSR on sparse update-in-place (avoid serialization)
b) [SYSTEMML-554] Enhanced SystemML Frame support for JMLC API / CP
* SystemML frame data type, deeply integrated into compiler and runtime
* New builtin functions: transform, transformapply, transformencode,
transformdecode, transformmeta
* Supported operations: read/write, left/right indexing, casting, append,
transform/transformapply
c) Framework compatibility / configuration
* [SYSTEMML-418] Version-specific Spark memory budgets (>=1.6, legacy)
* [SYSTEMML-158] Updated deprecated Hadoop properties
* [SYSTEMML-476] Version-specific MR configuration handling (MRv2, MRv1)
* Fixes for backwards compatibility to MRv1 (?? shouldn't this be in -158
??)
* New pass-through mapred/mapreduce configurations through SystemML-config
* [SYSTEMML-584/585] New thread-local configuration handling (compiler/dml
config)
d) Deep learning support (experimental)
* [SYSTMEML-618] New DML-bodied function library for deep learning
* [SYSTEMML-540] New builtin singlenode operations: conv2d, maxpooling,
im2col, col2im, rotate
* New lenet-train DML script (staging)
e) API/script usability
* [SYSTEMML-607/604/611] Parser error handling
* [SYSTEMML-506/508/544/577/649/651] Extended MLContext/JMLC APIs
* [SYSTEMML-625/626/632] Improved source statement handling (e.g.,
imports, absolute paths)
* [SYSTEMML-617/631/654] Improved name space handling
* [SYSTEMML-240] Extended stats outputs for Spark
collect/broadcast/parallelize
* [SYSTEMML-495] SystemML configuration handling
* [SYSTEMML-209] Include algorithms in systemml jar
* [SYSTEMML-647/648] Deprecated castAsScalar, ppred
* [SYSTEMML-477] JSON meta data handling
* [SYSTEMML-294] Print matrix builtin function
* [SYSTEMML-296/676/670] Improved PyDML syntax: slicing, rand, cdf, elif
* [SYSTEMML-675] Support for negative for/parfor loop increments
f) New fused physical operators
* [SYSTEMML-488] Fused wdivmm w/ 4 operands
* [SYSTEMML-510] Fused wdivmm/wcemm w/ eps term
g) Various performance features
* [SYSTEMML-427/512] Extended IPA (propagate scalar variables)
* [SYSTEMML-282] Extended update-in-place support for parfor intermediates
* [SYSTEMML-552/399] Performance parallel binary/text readers (sort
sparse/nnz handling)
* [SYSTEMML-552/641] Cache-conscious operations: sparse-dense
wdivmm/wsloss, sparse-dense/sparse-sparse mm, dense-dense skinny rhs mm
* [SYSTEMML-641] Tuned special cases for block matrix multiplication:
e.g., mm w/ skinny rhs, colwise parallelization wide rhs
* [SYSTEMML-396/400] New/extended multithreaded operations:
cumsum/cummin/cummax/cumprod, transpose, and rand
* [SYSTEMML-510/694] New simplification rewrites: "pushdown
unaryagg-transpose", "simplify transpose-aggbin-binary chains", "reorder
minus-mmult", "canonicalize matmult-add-scalar", improved constant folding
(all unary)
* [SYSTEMML-653] Asynchronous bufferpool cleanup of evicted files / nio
file eviction
- MR iqm/quantile/median (qsort num reducers, qpick buffer size)
h) DML script updates
* [SYSTEMML-536] New KNN algorithm (still staging)
* [SYSTEMML-534] Optional console output univariate statistics
* [SYSTMEML-494] GLM compiler warnings
* Robustness input/output handling L2SVM, MSVM, and Naive Bayes
* Random data generator for ALS
i) Various fixes
- Dozens of fixes for diverse issues, fix pack for 0.9 release
j) Build, documentation, examples
* [SYSTEMML-551] Enhanced jmlc javadoc
* [SYSTEMML-484] Build javadoc jar
* [SYSTEMML-468] Contributing to SystemML doc
* [SYSTEMML-517/524] DML language ref updates
* [SYSTEMML-498] Trouble shooting guide
* SystemML Jupyter/Zeppelin notebook examples
Regards,
Matthias
Deron Eriksson ---06/13/2016 01:29:16 PM---Hi, We need to put together a
page of release notes for the 0.10.0 release
From: Deron Eriksson <de...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 06/13/2016 01:29 PM
Subject: Release notes for 0.10.0
Hi,
We need to put together a page of release notes for the 0.10.0 release
(similar to http://systemml.apache.org/0.9.0-incubating/release_notes.html
).
Could SystemML committers and contributors please respond to this thread
with what they feel are the major updates/improvements in this release?
Thanks!
Deron
Re: Release notes for 0.10.0
Posted by Matthias Boehm <mb...@us.ibm.com>.
ok, here is a first draft of the release notes - please feel free to extend
or prune this:
a) [SYSTEMML-377] Different types of spark matrix blocks
* Supported internal formats: MCSR (default), CSR, COO
* Automatic MCSR->CSR on Spark read/caching (for memory efficiency)
* Automatic MCSR->CSR on sparse update-in-place (avoid serialization)
b) [SYSTEMML-554] Frame support for JMLC API / CP
* New frame data type, deeply integrated into compiler and runtime
* New builtin functions: transformapply, transformencode,
transformdecode, transformmeta
* Supported operations: read/write, left/right indexing, casting, append,
transform/transformapply
c) Framework compatibility / configuration
* [SYSTEMML-418] Version-specific Spark memory budgets (>=1.6, legacy)
* [SYSTEMML-158] Updated deprecated Hadoop properties
* [SYSTEMML-476] Version-specific MR configuration handling (MRv2, MRv1)
* Fixes for backwards compatibility to MRv1
* New pass-through mapred/mapreduce configurations through
SystemML-config
* [SYSTEMML-584/585] New thread-local configuration handling
(compiler/dml config)
d) Deep learning support
* [SYSTMEML-618] New DML-script NN library
* [SYSTEMML-540] New builtin singlenode operations: conv2d, maxpooling,
im2col, col2im, rotate
* New lenet-train dml script
e) API/script usability
* [SYSTEMML-607/604/611] Parser error handling
* [SYSTEMML-506/508/544/577/649/651] Extended MLContext/JMLC APIs
* [SYSTEMML-625/626/632] Improved source statement handling (e.g.,
imports, absolute paths)
* [SYSTEMML-617/631/654] Improved name space handling
* [SYSTEMML-240] Extended stats outputs for Spark
collect/broadcast/parallelize
* [SYSTEMML-495] SystemML configuration handling
* [SYSTEMML-209] Include algorithms in systemml jar
* [SYSTEMML-647/648] Deprecated castAsScalar, ppred
* [SYSTEMML-477] JSON meta data handling
* [SYSTEMML-294] Print matrix builtin function
* [SYSTEMML-296/676/670] Improved PyDML syntax: slicing, rand, cdf, elif
* [SYSTEMML-675] Support for negative for/parfor loop increments
f) New fused physical operators
* [SYSTEMML-488] Fused wdivmm w/ 4 operands
* [SYSTEMML-510] Fused wdivmm/wcemm w/ eps term
g) Various performance features
* [SYSTEMML-427/512] Extended IPA (propagate scalar variables)
* [SYSTEMML-282] Extended update-in-place support for parfor intermediates
* [SYSTEMML-552/399] Performance parallel binary/text readers (sort
sparse/nnz handling)
* [SYSTEMML-552/641] Cache-conscious operations: sparse-dense
wdivmm/wsloss, sparse-dense/sparse-sparse mm, dense-dense skinny rhs mm
* [SYSTEMML-641] Tuned special cases for block matrix multiplication:
e.g., mm w/ skinny rhs, colwise parallelization wide rhs
* [SYSTEMML-396/400] New/extended multithreaded operations:
cumsum/cummin/cummax/cumprod, transpose, and rand
* [SYSTEMML-510/694] New simplification rewrites: "pushdown
unaryagg-transpose", "simplify transpose-aggbin-binary chains", "reorder
minus-mmult", "canonicalize matmult-add-scalar", improved constant folding
(all unary)
* [SYSTEMML-653] Asynchronous bufferpool cleanup of evicted files / nio
file eviction
- MR iqm/quantile/median (qsort num reducers, qpick buffer size)
h) DML script updates
* [SYSTEMML-536] New KNN algorithm (still staging)
* [SYSTEMML-534] Optional console output univariate statistics
* [SYSTMEML-494] GLM compiler warnings
* Robustness input/output handling L2SVM, MSVM, and Naive Bayes
* Random data generator for ALS
i) Various fixes
- Dozens of fixes for diverse issues, fix pack for 0.9 release
j) Build, documentation, examples
* [SYSTEMML-551] Enhanced jmlc javadoc
* [SYSTEMML-484] Build javadoc jar
* [SYSTEMML-468] Contributing to SystemML doc
* [SYSTEMML-517/524] DML language ref updates
* [SYSTEMML-498] Trouble shooting guide
* SystemML Jupyter/Zeppelin notebook examples
Regards,
Matthias
From: Deron Eriksson <de...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 06/13/2016 01:29 PM
Subject: Release notes for 0.10.0
Hi,
We need to put together a page of release notes for the 0.10.0 release
(similar to http://systemml.apache.org/0.9.0-incubating/release_notes.html
).
Could SystemML committers and contributors please respond to this thread
with what they feel are the major updates/improvements in this release?
Thanks!
Deron