You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Steve Robinson <St...@aquilainsight.com> on 2017/04/12 14:45:08 UTC

Optimisation Tips

Hi,


Does anyone have any optimisation tips or could propose an alternative way to perform the below:


val groupedUserItems1 = userItems1.groupByKey{_.customer_id}
val groupedUserItems2 = userItems2.groupByKey{_.customer_id}
groupedUserItems1.cogroup(groupedUserItems2){
   case (_, userItems1, userItems2) =>
        processSingleUser(userItems1, userItems2)
    }
}

The userItems1 and userItems2 datasets are quite large (100's millions of records) so I'm finding the shuffle stage is shuffing Gigabytes of data.

Any help would be greatly appreciated.

Thanks,



Steve Robinson

Steve.Robinson@aquilainsight.com
0131 290 2300

[https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=coZwd1lxzVikDFySR9HQQUgSdTBDtR24zCE7VI3ucKE%3d&docid=0c609694b07114f03a01631f3c5cc4606&rev=1][https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=XWaweiSSd7YO1IFgfwqm3AAn7KKCsmBf%2f73IlT3d0zE%3d&docid=0cea80d160d954b9584aef7090a5c4ef5&rev=1][https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=hTrHq%2fmrgDTxOp4jWXzYVM04wsasy2aNJfmG6EJJ%2f9g%3d&docid=00a6dd20560b1438fadebfb8a1255be41&rev=1]
www.aquilainsight.com<http://www.aquilainsight.com>
[https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=N79xtBiBY4r5ry1TCu0P%2bce%2f%2b3HFTwwamnQ47PieOoo%3d&docid=03f7d1040c43f4fa0bcdf7f17fa89dfcc&rev=1]linkedin.com/aquilainsight<https://www.linkedin.com/company/aquila-insight>
[https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=fdX1gHdkBdEZ%2bOap1Nr7kTrjMoFxgTZI4RfHFw0R7mw%3d&docid=0869faaa87f6c402fa845a320c225e213&rev=1]twitter.com/aquilainsight<http://twitter.com/aquilainsight>

Re: Optimisation Tips

Posted by "Pushkar.Gujar" <pu...@gmail.com>.

Not a expert, but groupByKey operation is well known to cause lot of
shuffling and usually operation performed by groupbykey operation can be
replaced by reducebykey.

Here is great article on groupByKey operation -

https://github.com/awesome-spark/spark-gotchas/blob/master/04_rdd_actions_and_transformations_by_example.md#be-smart-about-groupbykey


this whole repository - "spark-gotchas" is filled with lot of helpful tips.




Thank you,
*Pushkar Gujar*


On Wed, Apr 12, 2017 at 10:48 AM, KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:

> Hi Steve,
>
> I have implemented repartitions on dataframe to 1. It helped the
> performance but not to a great extent. I am also looking for answers from
> the experts.
>
> Thanks,
> Asmath
>
> On Wed, Apr 12, 2017 at 9:45 AM, Steve Robinson <
> Steve.Robinson@aquilainsight.com> wrote:
>
>> Hi,
>>
>>
>> Does anyone have any optimisation tips or could propose an alternative
>> way to perform the below:
>>
>>
>> val groupedUserItems1 = userItems1.groupByKey{_.customer_id}
>> val groupedUserItems2 = userItems2.groupByKey{_.customer_id}
>> groupedUserItems1.cogroup(groupedUserItems2){
>>    case (_, userItems1, userItems2) =>
>>         processSingleUser(userItems1, userItems2)
>>     }
>> }
>>
>> The userItems1 and userItems2 datasets are quite large (100's millions of
>> records) so I'm finding the shuffle stage is shuffing Gigabytes of data.
>>
>> Any help would be greatly appreciated.
>>
>> Thanks,
>>
>>
>> Steve Robinson
>>
>> Steve.Robinson@aquilainsight.com
>> 0131 290 2300
>>
>>
>> www.aquilainsight.com
>> linkedin.com/aquilainsight
>> <https://www.linkedin.com/company/aquila-insight>
>>
>> twitter.com/aquilainsight
>>
>
>

Re: Optimisation Tips

Posted by KhajaAsmath Mohammed <md...@gmail.com>.

Hi Steve,

I have implemented repartitions on dataframe to 1. It helped the
performance but not to a great extent. I am also looking for answers from
the experts.

Thanks,
Asmath

On Wed, Apr 12, 2017 at 9:45 AM, Steve Robinson <
Steve.Robinson@aquilainsight.com> wrote:

> Hi,
>
>
> Does anyone have any optimisation tips or could propose an alternative way
> to perform the below:
>
>
> val groupedUserItems1 = userItems1.groupByKey{_.customer_id}
> val groupedUserItems2 = userItems2.groupByKey{_.customer_id}
> groupedUserItems1.cogroup(groupedUserItems2){
>    case (_, userItems1, userItems2) =>
>         processSingleUser(userItems1, userItems2)
>     }
> }
>
> The userItems1 and userItems2 datasets are quite large (100's millions of
> records) so I'm finding the shuffle stage is shuffing Gigabytes of data.
>
> Any help would be greatly appreciated.
>
> Thanks,
>
>
> Steve Robinson
>
> Steve.Robinson@aquilainsight.com
> 0131 290 2300
>
>
> www.aquilainsight.com
> linkedin.com/aquilainsight
> <https://www.linkedin.com/company/aquila-insight>
>
> twitter.com/aquilainsight
>