You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Walrus theCat <wa...@gmail.com> on 2013/11/11 21:13:28 UTC

spark.akka.threads recommendations?

Hi,

The docs say that we should be careful to increase spark.akka.threads when
our cluster size increases.  Is there a rule of thumb for how much we
should increase it?  For instance, the recommendation for default
parallelism is 2x cores.

Re: spark.akka.threads recommendations?

Posted by Walrus theCat <wa...@gmail.com>.

Good to know, thanks.


On Mon, Nov 11, 2013 at 5:29 PM, Matei Zaharia <ma...@gmail.com>wrote:

> Actually it doesn’t matter a lot from what I’ve seen. Only do it if you
> see a lot of communication going to the master (these threads do the
> serialization of tasks). I’ve never put more than 8 or so.
>
> Matei
>
> On Nov 11, 2013, at 12:13 PM, Walrus theCat <wa...@gmail.com>
> wrote:
>
> > Hi,
> >
> > The docs say that we should be careful to increase spark.akka.threads
> when our cluster size increases.  Is there a rule of thumb for how much we
> should increase it?  For instance, the recommendation for default
> parallelism is 2x cores.
>
>

Re: spark.akka.threads recommendations?

Posted by Matei Zaharia <ma...@gmail.com>.

Actually it doesn’t matter a lot from what I’ve seen. Only do it if you see a lot of communication going to the master (these threads do the serialization of tasks). I’ve never put more than 8 or so.

Matei

On Nov 11, 2013, at 12:13 PM, Walrus theCat <wa...@gmail.com> wrote:

> Hi,
> 
> The docs say that we should be careful to increase spark.akka.threads when our cluster size increases.  Is there a rule of thumb for how much we should increase it?  For instance, the recommendation for default parallelism is 2x cores.