You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Trinadh Kaja <kt...@gmail.com> on 2016/09/21 11:37:23 UTC
SPARK PERFORMANCE TUNING
Hi all,
how to increase spark performance ,i am using pyspark.
cluster info :
Total memory :600gb
Cores :96
command :
spark-submit --master yarn-client --executor-memory 10G --num-executors 50
--executor-cores 2 --driver-memory 10g --queue thequeue
please help on this
--
Thanks&Regards
K.Trinadh
Ph-7348826118
Re: SPARK PERFORMANCE TUNING
Posted by Mich Talebzadeh <mi...@gmail.com>.
LOL
I think we should try the Chrystal ball to answer this question.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 21 September 2016 at 13:14, Jörn Franke <jo...@gmail.com> wrote:
> Do you mind sharing what your software does? What is the input data size?
> What is the spark version and apis used? How many nodes? What is the input
> data format? Is compression used?
>
> On 21 Sep 2016, at 13:37, Trinadh Kaja <kt...@gmail.com> wrote:
>
> Hi all,
>
> how to increase spark performance ,i am using pyspark.
>
> cluster info :
>
> Total memory :600gb
> Cores :96
>
> command :
> spark-submit --master yarn-client --executor-memory 10G --num-executors
> 50 --executor-cores 2 --driver-memory 10g --queue thequeue
>
>
> please help on this
>
> --
> Thanks&Regards
> K.Trinadh
> Ph-7348826118
>
>
Re: SPARK PERFORMANCE TUNING
Posted by Jörn Franke <jo...@gmail.com>.
Do you mind sharing what your software does? What is the input data size? What is the spark version and apis used? How many nodes? What is the input data format? Is compression used?
> On 21 Sep 2016, at 13:37, Trinadh Kaja <kt...@gmail.com> wrote:
>
> Hi all,
>
> how to increase spark performance ,i am using pyspark.
>
> cluster info :
>
> Total memory :600gb
> Cores :96
>
> command :
> spark-submit --master yarn-client --executor-memory 10G --num-executors 50 --executor-cores 2 --driver-memory 10g --queue thequeue
>
>
> please help on this
>
> --
> Thanks&Regards
> K.Trinadh
> Ph-7348826118