You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by Acs S <ac...@yahoo.com.INVALID> on 2016/08/17 21:18:44 UTC
[DISCUSS] SystemML with Spark 2.0 support and roadmap
Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be uptodate with latest version of Spark. This brings us a challenge to support our consumers until they move to Spark 2.0.Based on some brainstorming, I can propose following options to keep SystemML being supported on latest Spark version quickly.
Supporting SystemML on Spark 1.x We can continue to support SystemML on Spark 1.x code base for short period of time by adding fixes and features on main branch. We will release SystemML with support to Spark 1.x next version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st 2016)
Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark 2.0, we can make SystemML on Spark 2.0 immediately based on branch created on top of latest master branch code. Glen has some prototype code to transform SystemML code to be compatible with Spark 2.0, he can merge his code with new branch targeted to support SystemML on Spark 2.0 This would be "Preview" version code, and we can update it on frequent basis (on bi-monthy basis). Supporting SystemML on Spark 2.0 We will have full support of SystemML on Spark 2.0 before end of year 2016. We will formalize release date by end of Sept 2016. At the same time we will discuss if we can move support of SystemML on Spark 1.x to maintenance mode (Only required bug fixes will be merged from main branch) or we need to support both SystemML on Spark 2.0 and Spark 1.x for some additional time.
SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep Learning (Library of Network layers?) - Frame - New MLContext API - Python DSL integration (Preview) - Compressed Linear Algebra (Preview) - Hydra R integration - New Algorithms (?)
0.12 (Spark 2.0) (Targeted to 4Q 2016) - GPU support (Local mode/Distributed mode?)
- New Algorithms (?)
Please feel free to comment on support and roadmap points.
-Arvind
Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Posted by du...@gmail.com.
I think this is a great idea so that we can simplify the official release and reduce confusion for potential users. Certainly we can still retain the potential to build the extra artifacts locally, just like Spark current does.
I would also like to suggest that we move away from the current Standalone package that is designed to be used with Java, and instead move to simply using Spark in local mode for all "standalone" applications. Since running Spark locally on a laptop consists of simply downloading a release binary and running it, without any installation, I think this is a much cleaner way now. This would allow us to immediately move to the goal of only releasing a single JAR file, as that same JAR file could be used in Spark locally, Spark on a cluster, and Hadoop on a cluster. Then we could just release the single JAR file and a folder of scripts as our official release. All other special artifacts could be kept as "download and build" artifacts.
--
Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry
Sent from my iPhone.
> On Aug 23, 2016, at 5:22 PM, Luciano Resende <lu...@gmail.com> wrote:
>
> On Tue, Aug 23, 2016 at 3:51 PM, Deron Eriksson <de...@gmail.com>
> wrote:
>
>> To simplify release candidate validation, I would like to propose that the
>> distribution profile only builds the following 7 (out of the current
>> included 10) artifacts:
>>
>> systemml-0.11.0-incubating-SNAPSHOT-javadoc.jar
>> systemml-0.11.0-incubating-SNAPSHOT-sources.jar
>> systemml-0.11.0-incubating-SNAPSHOT-src.tar.gz
>> systemml-0.11.0-incubating-SNAPSHOT-src.zip
>> systemml-0.11.0-incubating-SNAPSHOT-standalone.tar.gz (rename w/o
>> "-standalone")
>> systemml-0.11.0-incubating-SNAPSHOT-standalone.zip (rename w/o
>> "-standalone")
>> systemml-0.11.0-incubating-SNAPSHOT.jar
>>
>> The following could still be built using maven profiles but would not be in
>> the distribution profile:
>>
>> systemml-0.11.0-incubating-SNAPSHOT-standalone.jar
>> systemml-0.11.0-incubating-SNAPSHOT.tar.gz (also rename)
>> systemml-0.11.0-incubating-SNAPSHOT.zip (also rename)
>>
>> This would decrease the number of our artifacts by 30% which means that we
>> can validate the release faster, and the release candidate will also be
>> more likely to pass external validation/voting.
>>
>> Deron
> +1
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Posted by Luciano Resende <lu...@gmail.com>.
On Tue, Aug 23, 2016 at 3:51 PM, Deron Eriksson <de...@gmail.com>
wrote:
> To simplify release candidate validation, I would like to propose that the
> distribution profile only builds the following 7 (out of the current
> included 10) artifacts:
>
> systemml-0.11.0-incubating-SNAPSHOT-javadoc.jar
> systemml-0.11.0-incubating-SNAPSHOT-sources.jar
> systemml-0.11.0-incubating-SNAPSHOT-src.tar.gz
> systemml-0.11.0-incubating-SNAPSHOT-src.zip
> systemml-0.11.0-incubating-SNAPSHOT-standalone.tar.gz (rename w/o
> "-standalone")
> systemml-0.11.0-incubating-SNAPSHOT-standalone.zip (rename w/o
> "-standalone")
> systemml-0.11.0-incubating-SNAPSHOT.jar
>
> The following could still be built using maven profiles but would not be in
> the distribution profile:
>
> systemml-0.11.0-incubating-SNAPSHOT-standalone.jar
> systemml-0.11.0-incubating-SNAPSHOT.tar.gz (also rename)
> systemml-0.11.0-incubating-SNAPSHOT.zip (also rename)
>
> This would decrease the number of our artifacts by 30% which means that we
> can validate the release faster, and the release candidate will also be
> more likely to pass external validation/voting.
>
> Deron
>
>
+1
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Posted by Deron Eriksson <de...@gmail.com>.
To simplify release candidate validation, I would like to propose that the
distribution profile only builds the following 7 (out of the current
included 10) artifacts:
systemml-0.11.0-incubating-SNAPSHOT-javadoc.jar
systemml-0.11.0-incubating-SNAPSHOT-sources.jar
systemml-0.11.0-incubating-SNAPSHOT-src.tar.gz
systemml-0.11.0-incubating-SNAPSHOT-src.zip
systemml-0.11.0-incubating-SNAPSHOT-standalone.tar.gz (rename w/o
"-standalone")
systemml-0.11.0-incubating-SNAPSHOT-standalone.zip (rename w/o
"-standalone")
systemml-0.11.0-incubating-SNAPSHOT.jar
The following could still be built using maven profiles but would not be in
the distribution profile:
systemml-0.11.0-incubating-SNAPSHOT-standalone.jar
systemml-0.11.0-incubating-SNAPSHOT.tar.gz (also rename)
systemml-0.11.0-incubating-SNAPSHOT.zip (also rename)
This would decrease the number of our artifacts by 30% which means that we
can validate the release faster, and the release candidate will also be
more likely to pass external validation/voting.
Deron
On Thu, Aug 18, 2016 at 12:05 AM, Berthold Reinwald <re...@us.ibm.com>
wrote:
> This makes sense. Couple of comments.
>
> - wrt SystemML on Spark 1.x, SystemML 0.11 target date should be the
> latest in Oct. Ideally it should be earlier - maybe September - depending
> on community demand as it will contain bug fixes for features introduced
> in 0.10. The master branch will stay on Spark 1.x til then. New features
> can be integrated but as they are still partial, they will be marked as
> experimental.
>
> - wrt SystemML on Spark 2.0, as the timeline is fairly short, instead of
> bi-monthly I'd suggest more frequent sync with the branch on Spark 2.0
>
> - Comes early October, we should switch the master to Spark 2.0, and
> immediately create a release for 2.0.
>
> - wrt Roadmap 0.11 items, except for Frame support and MLContext and may
> be DL lib, all the other features should be marked experimental.
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinwald@us.ibm.com
>
>
>
> From: dusenberrymw@gmail.com
> To: dev@systemml.incubator.apache.org
> Date: 08/17/2016 04:52 PM
> Subject: Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
>
>
>
> +1
>
> This sounds like a good plan to allow us to continue supporting the Spark
> 1.x line in the short term, with a plan for moving to Spark 2.x support
> soon.
>
> -Mike
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>
> > On Aug 17, 2016, at 2:59 PM, Deron Eriksson <de...@gmail.com>
> wrote:
> >
> > +1
> > Continuing to support Spark 1.4/1.6 for now while setting a cutover date
> > for 2.0 sounds like a great idea. This allows for the creation of a
> really
> > solid release for 1.x, which greatly benefits SystemML users using Spark
> > 1.x. It also gives these users a general date that they can use to plan
> > migration to Spark 2.0 when that becomes the SystemML standard so that
> they
> > can benefit from the latest improvements to the project.
> >
> > Deron
> >
> >
> >> On Wed, Aug 17, 2016 at 2:32 PM, Acs S <ac...@yahoo.com.invalid> wrote:
> >>
> >> Seems, mail is not retaining format. I am attaching same text through
> PDF
> >> file.
> >> If there is any other better option please let me know.
> >>
> >>
> >> -Arvind
> >>
> >>
> >>
> >>
> >>
> >> ----- Forwarded Message -----
> >> *From:* Acs S <ac...@yahoo.com.INVALID>
> >> *To:* Dev <de...@systemml.incubator.apache.org>
> >> *Sent:* Wednesday, August 17, 2016 2:18 PM
> >> *Subject:* [DISCUSS] SystemML with Spark 2.0 support and roadmap
> >>
> >>
> >> Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be
> >> uptodate with latest version of Spark. This brings us a challenge to
> >> support our consumers until they move to Spark 2.0.Based on some
> >> brainstorming, I can propose following options to keep SystemML being
> >> supported on latest Spark version quickly.
> >>
> >> Supporting SystemML on Spark 1.x We can continue to support SystemML on
> >> Spark 1.x code base for short period of time by adding fixes and
> features
> >> on main branch. We will release SystemML with support to Spark 1.x
> next
> >> version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st
> 2016)
> >> Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark
> >> 2.0, we can make SystemML on Spark 2.0 immediately based on branch
> created
> >> on top of latest master branch code. Glen has some prototype code to
> >> transform SystemML code to be compatible with Spark 2.0, he can merge
> his
> >> code with new branch targeted to support SystemML on Spark 2.0 This
> would
> >> be "Preview" version code, and we can update it on frequent basis (on
> >> bi-monthy basis). Supporting SystemML on Spark 2.0 We will have full
> >> support of SystemML on Spark 2.0 before end of year 2016. We will
> formalize
> >> release date by end of Sept 2016. At the same time we will discuss if
> we
> >> can move support of SystemML on Spark 1.x to maintenance mode (Only
> >> required bug fixes will be merged from main branch) or we need to
> support
> >> both SystemML on Spark 2.0 and Spark 1.x for some additional time.
> >> SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep
> >> Learning (Library of Network layers?) - Frame - New MLContext API -
> >> Python DSL integration (Preview) - Compressed Linear Algebra (Preview)
> -
> >> Hydra R integration - New Algorithms (?)
> >> 0.12 (Spark 2.0) (Targeted to 4Q 2016) - GPU support (Local
> >> mode/Distributed mode?)
> >> - New Algorithms (?)
> >> Please feel free to comment on support and roadmap points.
> >>
> >>
> >> -Arvind
> >>
> >>
>
>
>
>
>
Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Posted by Berthold Reinwald <re...@us.ibm.com>.
This makes sense. Couple of comments.
- wrt SystemML on Spark 1.x, SystemML 0.11 target date should be the
latest in Oct. Ideally it should be earlier - maybe September - depending
on community demand as it will contain bug fixes for features introduced
in 0.10. The master branch will stay on Spark 1.x til then. New features
can be integrated but as they are still partial, they will be marked as
experimental.
- wrt SystemML on Spark 2.0, as the timeline is fairly short, instead of
bi-monthly I'd suggest more frequent sync with the branch on Spark 2.0
- Comes early October, we should switch the master to Spark 2.0, and
immediately create a release for 2.0.
- wrt Roadmap 0.11 items, except for Frame support and MLContext and may
be DL lib, all the other features should be marked experimental.
Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinwald@us.ibm.com
From: dusenberrymw@gmail.com
To: dev@systemml.incubator.apache.org
Date: 08/17/2016 04:52 PM
Subject: Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
+1
This sounds like a good plan to allow us to continue supporting the Spark
1.x line in the short term, with a plan for moving to Spark 2.x support
soon.
-Mike
--
Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry
Sent from my iPhone.
> On Aug 17, 2016, at 2:59 PM, Deron Eriksson <de...@gmail.com>
wrote:
>
> +1
> Continuing to support Spark 1.4/1.6 for now while setting a cutover date
> for 2.0 sounds like a great idea. This allows for the creation of a
really
> solid release for 1.x, which greatly benefits SystemML users using Spark
> 1.x. It also gives these users a general date that they can use to plan
> migration to Spark 2.0 when that becomes the SystemML standard so that
they
> can benefit from the latest improvements to the project.
>
> Deron
>
>
>> On Wed, Aug 17, 2016 at 2:32 PM, Acs S <ac...@yahoo.com.invalid> wrote:
>>
>> Seems, mail is not retaining format. I am attaching same text through
PDF
>> file.
>> If there is any other better option please let me know.
>>
>>
>> -Arvind
>>
>>
>>
>>
>>
>> ----- Forwarded Message -----
>> *From:* Acs S <ac...@yahoo.com.INVALID>
>> *To:* Dev <de...@systemml.incubator.apache.org>
>> *Sent:* Wednesday, August 17, 2016 2:18 PM
>> *Subject:* [DISCUSS] SystemML with Spark 2.0 support and roadmap
>>
>>
>> Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be
>> uptodate with latest version of Spark. This brings us a challenge to
>> support our consumers until they move to Spark 2.0.Based on some
>> brainstorming, I can propose following options to keep SystemML being
>> supported on latest Spark version quickly.
>>
>> Supporting SystemML on Spark 1.x We can continue to support SystemML on
>> Spark 1.x code base for short period of time by adding fixes and
features
>> on main branch. We will release SystemML with support to Spark 1.x
next
>> version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st
2016)
>> Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark
>> 2.0, we can make SystemML on Spark 2.0 immediately based on branch
created
>> on top of latest master branch code. Glen has some prototype code to
>> transform SystemML code to be compatible with Spark 2.0, he can merge
his
>> code with new branch targeted to support SystemML on Spark 2.0 This
would
>> be "Preview" version code, and we can update it on frequent basis (on
>> bi-monthy basis). Supporting SystemML on Spark 2.0 We will have full
>> support of SystemML on Spark 2.0 before end of year 2016. We will
formalize
>> release date by end of Sept 2016. At the same time we will discuss if
we
>> can move support of SystemML on Spark 1.x to maintenance mode (Only
>> required bug fixes will be merged from main branch) or we need to
support
>> both SystemML on Spark 2.0 and Spark 1.x for some additional time.
>> SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep
>> Learning (Library of Network layers?) - Frame - New MLContext API -
>> Python DSL integration (Preview) - Compressed Linear Algebra (Preview)
-
>> Hydra R integration - New Algorithms (?)
>> 0.12 (Spark 2.0) (Targeted to 4Q 2016) - GPU support (Local
>> mode/Distributed mode?)
>> - New Algorithms (?)
>> Please feel free to comment on support and roadmap points.
>>
>>
>> -Arvind
>>
>>
Re: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Posted by du...@gmail.com.
+1
This sounds like a good plan to allow us to continue supporting the Spark 1.x line in the short term, with a plan for moving to Spark 2.x support soon.
-Mike
--
Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry
Sent from my iPhone.
> On Aug 17, 2016, at 2:59 PM, Deron Eriksson <de...@gmail.com> wrote:
>
> +1
> Continuing to support Spark 1.4/1.6 for now while setting a cutover date
> for 2.0 sounds like a great idea. This allows for the creation of a really
> solid release for 1.x, which greatly benefits SystemML users using Spark
> 1.x. It also gives these users a general date that they can use to plan
> migration to Spark 2.0 when that becomes the SystemML standard so that they
> can benefit from the latest improvements to the project.
>
> Deron
>
>
>> On Wed, Aug 17, 2016 at 2:32 PM, Acs S <ac...@yahoo.com.invalid> wrote:
>>
>> Seems, mail is not retaining format. I am attaching same text through PDF
>> file.
>> If there is any other better option please let me know.
>>
>>
>> -Arvind
>>
>>
>>
>>
>>
>> ----- Forwarded Message -----
>> *From:* Acs S <ac...@yahoo.com.INVALID>
>> *To:* Dev <de...@systemml.incubator.apache.org>
>> *Sent:* Wednesday, August 17, 2016 2:18 PM
>> *Subject:* [DISCUSS] SystemML with Spark 2.0 support and roadmap
>>
>>
>> Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be
>> uptodate with latest version of Spark. This brings us a challenge to
>> support our consumers until they move to Spark 2.0.Based on some
>> brainstorming, I can propose following options to keep SystemML being
>> supported on latest Spark version quickly.
>>
>> Supporting SystemML on Spark 1.x We can continue to support SystemML on
>> Spark 1.x code base for short period of time by adding fixes and features
>> on main branch. We will release SystemML with support to Spark 1.x next
>> version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st 2016)
>> Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark
>> 2.0, we can make SystemML on Spark 2.0 immediately based on branch created
>> on top of latest master branch code. Glen has some prototype code to
>> transform SystemML code to be compatible with Spark 2.0, he can merge his
>> code with new branch targeted to support SystemML on Spark 2.0 This would
>> be "Preview" version code, and we can update it on frequent basis (on
>> bi-monthy basis). Supporting SystemML on Spark 2.0 We will have full
>> support of SystemML on Spark 2.0 before end of year 2016. We will formalize
>> release date by end of Sept 2016. At the same time we will discuss if we
>> can move support of SystemML on Spark 1.x to maintenance mode (Only
>> required bug fixes will be merged from main branch) or we need to support
>> both SystemML on Spark 2.0 and Spark 1.x for some additional time.
>> SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep
>> Learning (Library of Network layers?) - Frame - New MLContext API -
>> Python DSL integration (Preview) - Compressed Linear Algebra (Preview) -
>> Hydra R integration - New Algorithms (?)
>> 0.12 (Spark 2.0) (Targeted to 4Q 2016) - GPU support (Local
>> mode/Distributed mode?)
>> - New Algorithms (?)
>> Please feel free to comment on support and roadmap points.
>>
>>
>> -Arvind
>>
>>
Re: Fw: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Posted by Deron Eriksson <de...@gmail.com>.
+1
Continuing to support Spark 1.4/1.6 for now while setting a cutover date
for 2.0 sounds like a great idea. This allows for the creation of a really
solid release for 1.x, which greatly benefits SystemML users using Spark
1.x. It also gives these users a general date that they can use to plan
migration to Spark 2.0 when that becomes the SystemML standard so that they
can benefit from the latest improvements to the project.
Deron
On Wed, Aug 17, 2016 at 2:32 PM, Acs S <ac...@yahoo.com.invalid> wrote:
> Seems, mail is not retaining format. I am attaching same text through PDF
> file.
> If there is any other better option please let me know.
>
>
> -Arvind
>
>
>
>
>
> ----- Forwarded Message -----
> *From:* Acs S <ac...@yahoo.com.INVALID>
> *To:* Dev <de...@systemml.incubator.apache.org>
> *Sent:* Wednesday, August 17, 2016 2:18 PM
> *Subject:* [DISCUSS] SystemML with Spark 2.0 support and roadmap
>
>
> Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be
> uptodate with latest version of Spark. This brings us a challenge to
> support our consumers until they move to Spark 2.0.Based on some
> brainstorming, I can propose following options to keep SystemML being
> supported on latest Spark version quickly.
>
> Supporting SystemML on Spark 1.x We can continue to support SystemML on
> Spark 1.x code base for short period of time by adding fixes and features
> on main branch. We will release SystemML with support to Spark 1.x next
> version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st 2016)
> Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark
> 2.0, we can make SystemML on Spark 2.0 immediately based on branch created
> on top of latest master branch code. Glen has some prototype code to
> transform SystemML code to be compatible with Spark 2.0, he can merge his
> code with new branch targeted to support SystemML on Spark 2.0 This would
> be "Preview" version code, and we can update it on frequent basis (on
> bi-monthy basis). Supporting SystemML on Spark 2.0 We will have full
> support of SystemML on Spark 2.0 before end of year 2016. We will formalize
> release date by end of Sept 2016. At the same time we will discuss if we
> can move support of SystemML on Spark 1.x to maintenance mode (Only
> required bug fixes will be merged from main branch) or we need to support
> both SystemML on Spark 2.0 and Spark 1.x for some additional time.
> SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep
> Learning (Library of Network layers?) - Frame - New MLContext API -
> Python DSL integration (Preview) - Compressed Linear Algebra (Preview) -
> Hydra R integration - New Algorithms (?)
> 0.12 (Spark 2.0) (Targeted to 4Q 2016) - GPU support (Local
> mode/Distributed mode?)
> - New Algorithms (?)
> Please feel free to comment on support and roadmap points.
>
>
> -Arvind
>
>
Fw: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Posted by Acs S <ac...@yahoo.com.INVALID>.
Seems, mail is not retaining format. I am attaching same text through PDF file. If there is any other better option please let me know.
-Arvind
----- Forwarded Message -----
From: Acs S <ac...@yahoo.com.INVALID>
To: Dev <de...@systemml.incubator.apache.org>
Sent: Wednesday, August 17, 2016 2:18 PM
Subject: [DISCUSS] SystemML with Spark 2.0 support and roadmap
Spark 2.0 has released, we need to support SystemML on Spark 2.0 to be uptodate with latest version of Spark. This brings us a challenge to support our consumers until they move to Spark 2.0.Based on some brainstorming, I can propose following options to keep SystemML being supported on latest Spark version quickly.
Supporting SystemML on Spark 1.x We can continue to support SystemML on Spark 1.x code base for short period of time by adding fixes and features on main branch. We will release SystemML with support to Spark 1.x next version (0.11) around beginning of Oct 2016 (Lets target for Oct 1st 2016)
Supporting SystemML on Spark 2.0 (Preview code) For exploiters of Spark 2.0, we can make SystemML on Spark 2.0 immediately based on branch created on top of latest master branch code. Glen has some prototype code to transform SystemML code to be compatible with Spark 2.0, he can merge his code with new branch targeted to support SystemML on Spark 2.0 This would be "Preview" version code, and we can update it on frequent basis (on bi-monthy basis). Supporting SystemML on Spark 2.0 We will have full support of SystemML on Spark 2.0 before end of year 2016. We will formalize release date by end of Sept 2016. At the same time we will discuss if we can move support of SystemML on Spark 1.x to maintenance mode (Only required bug fixes will be merged from main branch) or we need to support both SystemML on Spark 2.0 and Spark 1.x for some additional time.
SystemML Roadmap 0.11 (on Spark 1.x) (Targeted to Oct 1st 2016) - Deep Learning (Library of Network layers?) - Frame - New MLContext API - Python DSL integration (Preview) - Compressed Linear Algebra (Preview) - Hydra R integration - New Algorithms (?)
0.12 (Spark 2.0) (Targeted to 4Q 2016) - GPU support (Local mode/Distributed mode?)
- New Algorithms (?)
Please feel free to comment on support and roadmap points.
-Arvind