You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by Janardhan <ja...@gmail.com> on 2020/08/12 12:22:52 UTC
Re: [QUESTION] About MLContext. Thanks.
Hi Sebastian,
What is the equivalent of the following snippet to SystemDS Python api?
>>> from systemml import MLContext>>> ml = MLContext(spark)
Welcome to Apache SystemML!Version 1.0.0-SNAPSHOT
I looked at the `l2svm` but do we have a generic object to which we could
pass spark.
Thank you,
Janardhan
On Mon, Jul 20, 2020 at 1:47 PM Janardhan <ja...@gmail.com>
wrote:
> Hi all,
>
> We have a bit of a backlog in MLContext test coverage. This one
> is open for discussion.
>
> The preliminary work[1] on reusing existing R scripts for codegen,
> for MLContext testing.
>
> Deciding on at least till release date helps our team to add algorithms
> one by one, to the extent of the codegen covered algorithms.
>
> [1] https://github.com/apache/systemds/pull/997
>
> Thank you,
> Janardhan
>
> On Fri, Jul 17, 2020 at 10:07 PM Janardhan <ja...@gmail.com>
> wrote:
>
>> Hi, SYSTEMDS-2572 the MLContext API is not loading into spark-shell. Of
>> course, everything else works fine!
>>
>> Thank you,
>> Janardhan
>>
>> On Fri, 17 Jul, 2020, 18:19 Matthias Boehm, <mb...@gmail.com> wrote:
>>
>>> yes, these nn scripts are packed into our SystemDS jar and can be used
>>> (via imports) in MLContext scripts as well. If there are issues, please
>>> feel free to file a bug report.
>>>
>>> Regards,
>>> Matthias
>>>
>>> On 7/17/2020 11:12 AM, Janardhan wrote:
>>> > Hi, Is `nn` library being provided with MLContext?
>>> >
>>> > Thank you,
>>> > Janardhan
>>> >
>>>
>>
Re: [QUESTION] About MLContext. Thanks.
Posted by Janardhan <ja...@gmail.com>.
Ability to call MLContext via python acts as a starting point for users to
adapt systemds into their pipelines for ML workloads (atleast, if not a
major portion of the pipeline!).
Case for MLContext python wrapper:
{{code}}
#1. Data processing with Spark
import pyspark.sql.functions as F
dataPath = "amazon0601.txt"
X_train = (sc.textFile(dataPath)
.filter(lambda l: not l.startswith("#"))
...)
#2. ML training with systemds
script = dml(pnmf).input(X=X_train, max_iter=100, rank=10).output("W", "H",
"losses")
losses = ml.execute(script).get("losses")
#3. Work with results with spark operations
xy = losses.toDF().sort("__INDEX").rdd.map(lambda r: (r[0], r[1])).collect()
{{code}}
Discussion PR: https://github.com/apache/systemds/pull/1024
<https://github.com/apache/systemds/pull/1024>
Thank you,
Janardhan
On Wed, Aug 12, 2020 at 6:51 PM Matthias Boehm <mb...@gmail.com> wrote:
> but just to be clear there is still room for adding a new python
> MLContext too (with similar APIs to our current).
>
> Regards,
> Matthias
>
> On 8/12/2020 2:43 PM, Baunsgaard, Sebastian wrote:
> >
> > Hi Janardhan,
> >
> >
> > The python interface currently does not run with spark, partially
> because we are focusing on federated execution.
> > The future intention is to not put the spark controls inside the python
> API, since it is the java execution that should handle spark.
> >
> > I think this is an upgrade since the handling of spark from both the
> python API and internally in systemds can be confusing.
> >
> >
> > best regards
> >
> > Sebastian
> >
> > ________________________________
> > From: Janardhan <ja...@gmail.com>
> > Sent: Wednesday, August 12, 2020 2:22:52 PM
> > To: dev@systemds.apache.org
> > Subject: Re: [QUESTION] About MLContext. Thanks.
> >
> > Hi Sebastian,
> >
> > What is the equivalent of the following snippet to SystemDS Python api?
> >
> >>>> from systemml import MLContext>>> ml = MLContext(spark)
> > Welcome to Apache SystemML!Version 1.0.0-SNAPSHOT
> >
> >
> > I looked at the `l2svm` but do we have a generic object to which we could
> > pass spark.
> >
> > Thank you,
> > Janardhan
> >
> >
> > On Mon, Jul 20, 2020 at 1:47 PM Janardhan <
> janardhan.pulivarthi@gmail.com>
> > wrote:
> >
> >> Hi all,
> >>
> >> We have a bit of a backlog in MLContext test coverage. This one
> >> is open for discussion.
> >>
> >> The preliminary work[1] on reusing existing R scripts for codegen,
> >> for MLContext testing.
> >>
> >> Deciding on at least till release date helps our team to add algorithms
> >> one by one, to the extent of the codegen covered algorithms.
> >>
> >> [1] https://github.com/apache/systemds/pull/997
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemds/pull/997>
> >
> > [SYSTEMDS-1863] Full MLContext test for LinearReg by j143 · Pull Request
> #997 · apache/systemds · GitHub<
> https://github.com/apache/systemds/pull/997>
> > github.com
> > Takes advantage of existing R algorithm scripts used for codegen
> testing. This would improve the testing by allowing us to provide all the
> necessary inputs into the script.
> >
> >
> >>
> >> Thank you,
> >> Janardhan
> >>
> >> On Fri, Jul 17, 2020 at 10:07 PM Janardhan <
> janardhan.pulivarthi@gmail.com>
> >> wrote:
> >>
> >>> Hi, SYSTEMDS-2572 the MLContext API is not loading into spark-shell. Of
> >>> course, everything else works fine!
> >>>
> >>> Thank you,
> >>> Janardhan
> >>>
> >>> On Fri, 17 Jul, 2020, 18:19 Matthias Boehm, <mb...@gmail.com> wrote:
> >>>
> >>>> yes, these nn scripts are packed into our SystemDS jar and can be used
> >>>> (via imports) in MLContext scripts as well. If there are issues,
> please
> >>>> feel free to file a bug report.
> >>>>
> >>>> Regards,
> >>>> Matthias
> >>>>
> >>>> On 7/17/2020 11:12 AM, Janardhan wrote:
> >>>>> Hi, Is `nn` library being provided with MLContext?
> >>>>>
> >>>>> Thank you,
> >>>>> Janardhan
> >>>>>
> >>>>
> >>>
> >
>
Re: [QUESTION] About MLContext. Thanks.
Posted by Matthias Boehm <mb...@gmail.com>.
but just to be clear there is still room for adding a new python
MLContext too (with similar APIs to our current).
Regards,
Matthias
On 8/12/2020 2:43 PM, Baunsgaard, Sebastian wrote:
>
> Hi Janardhan,
>
>
> The python interface currently does not run with spark, partially because we are focusing on federated execution.
> The future intention is to not put the spark controls inside the python API, since it is the java execution that should handle spark.
>
> I think this is an upgrade since the handling of spark from both the python API and internally in systemds can be confusing.
>
>
> best regards
>
> Sebastian
>
> ________________________________
> From: Janardhan <ja...@gmail.com>
> Sent: Wednesday, August 12, 2020 2:22:52 PM
> To: dev@systemds.apache.org
> Subject: Re: [QUESTION] About MLContext. Thanks.
>
> Hi Sebastian,
>
> What is the equivalent of the following snippet to SystemDS Python api?
>
>>>> from systemml import MLContext>>> ml = MLContext(spark)
> Welcome to Apache SystemML!Version 1.0.0-SNAPSHOT
>
>
> I looked at the `l2svm` but do we have a generic object to which we could
> pass spark.
>
> Thank you,
> Janardhan
>
>
> On Mon, Jul 20, 2020 at 1:47 PM Janardhan <ja...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> We have a bit of a backlog in MLContext test coverage. This one
>> is open for discussion.
>>
>> The preliminary work[1] on reusing existing R scripts for codegen,
>> for MLContext testing.
>>
>> Deciding on at least till release date helps our team to add algorithms
>> one by one, to the extent of the codegen covered algorithms.
>>
>> [1] https://github.com/apache/systemds/pull/997
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemds/pull/997>
>
> [SYSTEMDS-1863] Full MLContext test for LinearReg by j143 · Pull Request #997 · apache/systemds · GitHub<https://github.com/apache/systemds/pull/997>
> github.com
> Takes advantage of existing R algorithm scripts used for codegen testing. This would improve the testing by allowing us to provide all the necessary inputs into the script.
>
>
>>
>> Thank you,
>> Janardhan
>>
>> On Fri, Jul 17, 2020 at 10:07 PM Janardhan <ja...@gmail.com>
>> wrote:
>>
>>> Hi, SYSTEMDS-2572 the MLContext API is not loading into spark-shell. Of
>>> course, everything else works fine!
>>>
>>> Thank you,
>>> Janardhan
>>>
>>> On Fri, 17 Jul, 2020, 18:19 Matthias Boehm, <mb...@gmail.com> wrote:
>>>
>>>> yes, these nn scripts are packed into our SystemDS jar and can be used
>>>> (via imports) in MLContext scripts as well. If there are issues, please
>>>> feel free to file a bug report.
>>>>
>>>> Regards,
>>>> Matthias
>>>>
>>>> On 7/17/2020 11:12 AM, Janardhan wrote:
>>>>> Hi, Is `nn` library being provided with MLContext?
>>>>>
>>>>> Thank you,
>>>>> Janardhan
>>>>>
>>>>
>>>
>
Re: [QUESTION] About MLContext. Thanks.
Posted by "Baunsgaard, Sebastian" <ba...@tugraz.at>.
Hi Janardhan,
The python interface currently does not run with spark, partially because we are focusing on federated execution.
The future intention is to not put the spark controls inside the python API, since it is the java execution that should handle spark.
I think this is an upgrade since the handling of spark from both the python API and internally in systemds can be confusing.
best regards
Sebastian
________________________________
From: Janardhan <ja...@gmail.com>
Sent: Wednesday, August 12, 2020 2:22:52 PM
To: dev@systemds.apache.org
Subject: Re: [QUESTION] About MLContext. Thanks.
Hi Sebastian,
What is the equivalent of the following snippet to SystemDS Python api?
>>> from systemml import MLContext>>> ml = MLContext(spark)
Welcome to Apache SystemML!Version 1.0.0-SNAPSHOT
I looked at the `l2svm` but do we have a generic object to which we could
pass spark.
Thank you,
Janardhan
On Mon, Jul 20, 2020 at 1:47 PM Janardhan <ja...@gmail.com>
wrote:
> Hi all,
>
> We have a bit of a backlog in MLContext test coverage. This one
> is open for discussion.
>
> The preliminary work[1] on reusing existing R scripts for codegen,
> for MLContext testing.
>
> Deciding on at least till release date helps our team to add algorithms
> one by one, to the extent of the codegen covered algorithms.
>
> [1] https://github.com/apache/systemds/pull/997
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemds/pull/997>
[SYSTEMDS-1863] Full MLContext test for LinearReg by j143 · Pull Request #997 · apache/systemds · GitHub<https://github.com/apache/systemds/pull/997>
github.com
Takes advantage of existing R algorithm scripts used for codegen testing. This would improve the testing by allowing us to provide all the necessary inputs into the script.
>
> Thank you,
> Janardhan
>
> On Fri, Jul 17, 2020 at 10:07 PM Janardhan <ja...@gmail.com>
> wrote:
>
>> Hi, SYSTEMDS-2572 the MLContext API is not loading into spark-shell. Of
>> course, everything else works fine!
>>
>> Thank you,
>> Janardhan
>>
>> On Fri, 17 Jul, 2020, 18:19 Matthias Boehm, <mb...@gmail.com> wrote:
>>
>>> yes, these nn scripts are packed into our SystemDS jar and can be used
>>> (via imports) in MLContext scripts as well. If there are issues, please
>>> feel free to file a bug report.
>>>
>>> Regards,
>>> Matthias
>>>
>>> On 7/17/2020 11:12 AM, Janardhan wrote:
>>> > Hi, Is `nn` library being provided with MLContext?
>>> >
>>> > Thank you,
>>> > Janardhan
>>> >
>>>
>>