You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@gmail.com> on 2018/11/08 18:01:53 UTC
Arrow optimization in conversion from R DataFrame to Spark DataFrame
Hi all,
I am trying to introduce R Arrow optimization by reusing PySpark Arrow
optimization.
It boosts R DataFrame > Spark DataFrame up to roughly 900% ~ 1200% faster.
Looks working fine so far; however, I would appreciate if you guys have
some time to take a look (https://github.com/apache/spark/pull/22954) so
that we can directly go ahead as soon as R API of Arrow is released.
More importantly, I want some more people who're more into Arrow R API side
but also interested in Spark side. I have already cc'ed some people I know
but please come, review and discuss for both Spark side and Arrow side.
Thanks.
Re: Arrow optimization in conversion from R DataFrame to Spark DataFrame
Posted by Hyukjin Kwon <gu...@gmail.com>.
Thanks guys ! 👍
2018년 11월 10일 (토) 오전 7:35, Bryan Cutler <cu...@gmail.com>님이 작성:
> Great work Hyukjin! I'm not too familiar with R, but I'll take a look at
> the PR.
>
> Bryan
>
> On Fri, Nov 9, 2018 at 9:19 AM Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
>> Thanks Hyukjin! Very cool results
>>
>> Shivaram
>> On Fri, Nov 9, 2018 at 10:58 AM Felix Cheung <fe...@hotmail.com>
>> wrote:
>> >
>> > Very cool!
>> >
>> >
>> > ________________________________
>> > From: Hyukjin Kwon <gu...@gmail.com>
>> > Sent: Thursday, November 8, 2018 10:29 AM
>> > To: dev
>> > Subject: Arrow optimization in conversion from R DataFrame to Spark
>> DataFrame
>> >
>> > Hi all,
>> >
>> > I am trying to introduce R Arrow optimization by reusing PySpark Arrow
>> optimization.
>> >
>> > It boosts R DataFrame > Spark DataFrame up to roughly 900% ~ 1200%
>> faster.
>> >
>> > Looks working fine so far; however, I would appreciate if you guys have
>> some time to take a look (https://github.com/apache/spark/pull/22954) so
>> that we can directly go ahead as soon as R API of Arrow is released.
>> >
>> > More importantly, I want some more people who're more into Arrow R API
>> side but also interested in Spark side. I have already cc'ed some people I
>> know but please come, review and discuss for both Spark side and Arrow side.
>> >
>> > Thanks.
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: Arrow optimization in conversion from R DataFrame to Spark DataFrame
Posted by Bryan Cutler <cu...@gmail.com>.
Great work Hyukjin! I'm not too familiar with R, but I'll take a look at
the PR.
Bryan
On Fri, Nov 9, 2018 at 9:19 AM Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:
> Thanks Hyukjin! Very cool results
>
> Shivaram
> On Fri, Nov 9, 2018 at 10:58 AM Felix Cheung <fe...@hotmail.com>
> wrote:
> >
> > Very cool!
> >
> >
> > ________________________________
> > From: Hyukjin Kwon <gu...@gmail.com>
> > Sent: Thursday, November 8, 2018 10:29 AM
> > To: dev
> > Subject: Arrow optimization in conversion from R DataFrame to Spark
> DataFrame
> >
> > Hi all,
> >
> > I am trying to introduce R Arrow optimization by reusing PySpark Arrow
> optimization.
> >
> > It boosts R DataFrame > Spark DataFrame up to roughly 900% ~ 1200%
> faster.
> >
> > Looks working fine so far; however, I would appreciate if you guys have
> some time to take a look (https://github.com/apache/spark/pull/22954) so
> that we can directly go ahead as soon as R API of Arrow is released.
> >
> > More importantly, I want some more people who're more into Arrow R API
> side but also interested in Spark side. I have already cc'ed some people I
> know but please come, review and discuss for both Spark side and Arrow side.
> >
> > Thanks.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Arrow optimization in conversion from R DataFrame to Spark DataFrame
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
Thanks Hyukjin! Very cool results
Shivaram
On Fri, Nov 9, 2018 at 10:58 AM Felix Cheung <fe...@hotmail.com> wrote:
>
> Very cool!
>
>
> ________________________________
> From: Hyukjin Kwon <gu...@gmail.com>
> Sent: Thursday, November 8, 2018 10:29 AM
> To: dev
> Subject: Arrow optimization in conversion from R DataFrame to Spark DataFrame
>
> Hi all,
>
> I am trying to introduce R Arrow optimization by reusing PySpark Arrow optimization.
>
> It boosts R DataFrame > Spark DataFrame up to roughly 900% ~ 1200% faster.
>
> Looks working fine so far; however, I would appreciate if you guys have some time to take a look (https://github.com/apache/spark/pull/22954) so that we can directly go ahead as soon as R API of Arrow is released.
>
> More importantly, I want some more people who're more into Arrow R API side but also interested in Spark side. I have already cc'ed some people I know but please come, review and discuss for both Spark side and Arrow side.
>
> Thanks.
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Arrow optimization in conversion from R DataFrame to Spark
DataFrame
Posted by Felix Cheung <fe...@hotmail.com>.
Very cool!
________________________________
From: Hyukjin Kwon <gu...@gmail.com>
Sent: Thursday, November 8, 2018 10:29 AM
To: dev
Subject: Arrow optimization in conversion from R DataFrame to Spark DataFrame
Hi all,
I am trying to introduce R Arrow optimization by reusing PySpark Arrow optimization.
It boosts R DataFrame > Spark DataFrame up to roughly 900% ~ 1200% faster.
Looks working fine so far; however, I would appreciate if you guys have some time to take a look (https://github.com/apache/spark/pull/22954) so that we can directly go ahead as soon as R API of Arrow is released.
More importantly, I want some more people who're more into Arrow R API side but also interested in Spark side. I have already cc'ed some people I know but please come, review and discuss for both Spark side and Arrow side.
Thanks.