You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sa...@wellsfargo.com on 2016/01/15 15:28:46 UTC

jobs much slower in cluster mode vs local

Hello,

In general, I am usually able to run spark submit jobs in local mode, in a 32-cores node with plenty of memory ram. The performance is significantly faster in local mode than when using a cluster of spark workers.

How can this be explained and what measures can one take in order to improve such performance?
Usually a job that takes 35 seconds in local mode takes around 48 seconds in a small cluster.

Thanks,
Saif

RE: jobs much slower in cluster mode vs local

Posted by "Spencer, Alex (Santander)" <Al...@santander.co.uk.INVALID>.
That's not that much of a difference given the overhead of cluster management. I would have thought a job should take minutes before you'll see a performance improvement on using cluster mode?

Kind Regards,
Alex.

From: Saif.A.Ellafi@wellsfargo.com [mailto:Saif.A.Ellafi@wellsfargo.com]
Sent: 15 January 2016 14:29
To: user@spark.apache.org
Subject: jobs much slower in cluster mode vs local

Hello,

In general, I am usually able to run spark submit jobs in local mode, in a 32-cores node with plenty of memory ram. The performance is significantly faster in local mode than when using a cluster of spark workers.

How can this be explained and what measures can one take in order to improve such performance?
Usually a job that takes 35 seconds in local mode takes around 48 seconds in a small cluster.

Thanks,
Saif

RE: jobs much slower in cluster mode vs local

Posted by Sa...@wellsfargo.com.
Thank you, this looks useful indeed for what I have in mind.

Saif

From: Jiří Syrový [mailto:syrovy.jiri@gmail.com]
Sent: Friday, January 15, 2016 12:06 PM
To: Ellafi, Saif A.
Cc: user@spark.apache.org
Subject: Re: jobs much slower in cluster mode vs local

Hi,

you can try to use spark job server and submit jobs to it. The thing is that the most expensive part is context creation.
J.

2016-01-15 15:28 GMT+01:00 <Sa...@wellsfargo.com>>:
Hello,

In general, I am usually able to run spark submit jobs in local mode, in a 32-cores node with plenty of memory ram. The performance is significantly faster in local mode than when using a cluster of spark workers.

How can this be explained and what measures can one take in order to improve such performance?
Usually a job that takes 35 seconds in local mode takes around 48 seconds in a small cluster.

Thanks,
Saif

Re: jobs much slower in cluster mode vs local

Posted by Jiří Syrový <sy...@gmail.com>.
Hi,

you can try to use spark job server and submit jobs to it. The thing is
that the most expensive part is context creation.

J.

2016-01-15 15:28 GMT+01:00 <Sa...@wellsfargo.com>:

> Hello,
>
> In general, I am usually able to run spark submit jobs in local mode, in a
> 32-cores node with plenty of memory ram. The performance is significantly
> faster in local mode than when using a cluster of spark workers.
>
> How can this be explained and what measures can one take in order to
> improve such performance?
> Usually a job that takes 35 seconds in local mode takes around 48 seconds
> in a small cluster.
>
> Thanks,
> Saif
>
>