You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Yiannis Gkoufas <jo...@gmail.com> on 2015/02/20 13:54:41 UTC
Setting the number of executors in standalone mode
Hi there,
I try to increase the number of executors per worker in the standalone mode
and I have failed to achieve that.
I followed a bit the instructions of this thread:
http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores
and did that:
spark.executor.memory 1g
SPARK_WORKER_MEMORY=8g
hoping to get 8 executors per worker but its still 1.
And the option num-executors is not available in the standalone mode.
Thanks a lot!
Re: Setting the number of executors in standalone mode
Posted by Kelvin Chu <2d...@gmail.com>.
Hi,
Currently, there is only one executor per worker. There is jira ticket to
relax this:
https://issues.apache.org/jira/browse/SPARK-1706
But, if you want to use more cores, maybe, you can try increasing
SPARK_WORKER_INSTANCES. It increases the number of workers per machine.
Take a look here:
http://spark.apache.org/docs/1.2.0/spark-standalone.html
Hope this help!
Kelvin
On Fri, Feb 20, 2015 at 10:08 AM, Mohammed Guller <mo...@glassbeam.com>
wrote:
> ASFAIK, in stand-alone mode, each Spark application gets one executor on
> each worker. You could run multiple workers on a machine though.
>
>
>
> Mohammed
>
>
>
> *From:* Yiannis Gkoufas [mailto:johngouf85@gmail.com]
> *Sent:* Friday, February 20, 2015 9:48 AM
> *To:* Mohammed Guller
> *Cc:* user@spark.apache.org
> *Subject:* Re: Setting the number of executors in standalone mode
>
>
>
> Hi Mohammed,
>
>
>
> thanks a lot for the reply.
>
> Ok, so from what I understand I cannot control the number of executors per
> worker in standalone cluster mode.
>
> Is that correct?
>
>
>
> BR
>
>
>
> On 20 February 2015 at 17:46, Mohammed Guller <mo...@glassbeam.com>
> wrote:
>
> SPARK_WORKER_MEMORY=8g
>
> Will allocate 8GB memory to Spark on each worker node. Nothing to do with
> # of executors.
>
>
>
>
>
> Mohammed
>
>
>
> *From:* Yiannis Gkoufas [mailto:johngouf85@gmail.com]
> *Sent:* Friday, February 20, 2015 4:55 AM
> *To:* user@spark.apache.org
> *Subject:* Setting the number of executors in standalone mode
>
>
>
> Hi there,
>
>
>
> I try to increase the number of executors per worker in the standalone
> mode and I have failed to achieve that.
>
> I followed a bit the instructions of this thread:
> http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores
>
>
>
> and did that:
>
> spark.executor.memory 1g
>
> SPARK_WORKER_MEMORY=8g
>
>
>
> hoping to get 8 executors per worker but its still 1.
>
> And the option num-executors is not available in the standalone mode.
>
>
>
> Thanks a lot!
>
>
>
RE: Setting the number of executors in standalone mode
Posted by Mohammed Guller <mo...@glassbeam.com>.
ASFAIK, in stand-alone mode, each Spark application gets one executor on each worker. You could run multiple workers on a machine though.
Mohammed
From: Yiannis Gkoufas [mailto:johngouf85@gmail.com]
Sent: Friday, February 20, 2015 9:48 AM
To: Mohammed Guller
Cc: user@spark.apache.org
Subject: Re: Setting the number of executors in standalone mode
Hi Mohammed,
thanks a lot for the reply.
Ok, so from what I understand I cannot control the number of executors per worker in standalone cluster mode.
Is that correct?
BR
On 20 February 2015 at 17:46, Mohammed Guller <mo...@glassbeam.com>> wrote:
SPARK_WORKER_MEMORY=8g
Will allocate 8GB memory to Spark on each worker node. Nothing to do with # of executors.
Mohammed
From: Yiannis Gkoufas [mailto:johngouf85@gmail.com<ma...@gmail.com>]
Sent: Friday, February 20, 2015 4:55 AM
To: user@spark.apache.org<ma...@spark.apache.org>
Subject: Setting the number of executors in standalone mode
Hi there,
I try to increase the number of executors per worker in the standalone mode and I have failed to achieve that.
I followed a bit the instructions of this thread: http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores
and did that:
spark.executor.memory 1g
SPARK_WORKER_MEMORY=8g
hoping to get 8 executors per worker but its still 1.
And the option num-executors is not available in the standalone mode.
Thanks a lot!
Re: Setting the number of executors in standalone mode
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Hi Mohammed,
thanks a lot for the reply.
Ok, so from what I understand I cannot control the number of executors per
worker in standalone cluster mode.
Is that correct?
BR
On 20 February 2015 at 17:46, Mohammed Guller <mo...@glassbeam.com>
wrote:
> SPARK_WORKER_MEMORY=8g
>
> Will allocate 8GB memory to Spark on each worker node. Nothing to do with
> # of executors.
>
>
>
>
>
> Mohammed
>
>
>
> *From:* Yiannis Gkoufas [mailto:johngouf85@gmail.com]
> *Sent:* Friday, February 20, 2015 4:55 AM
> *To:* user@spark.apache.org
> *Subject:* Setting the number of executors in standalone mode
>
>
>
> Hi there,
>
>
>
> I try to increase the number of executors per worker in the standalone
> mode and I have failed to achieve that.
>
> I followed a bit the instructions of this thread:
> http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores
>
>
>
> and did that:
>
> spark.executor.memory 1g
>
> SPARK_WORKER_MEMORY=8g
>
>
>
> hoping to get 8 executors per worker but its still 1.
>
> And the option num-executors is not available in the standalone mode.
>
>
>
> Thanks a lot!
>
RE: Setting the number of executors in standalone mode
Posted by Mohammed Guller <mo...@glassbeam.com>.
SPARK_WORKER_MEMORY=8g
Will allocate 8GB memory to Spark on each worker node. Nothing to do with # of executors.
Mohammed
From: Yiannis Gkoufas [mailto:johngouf85@gmail.com]
Sent: Friday, February 20, 2015 4:55 AM
To: user@spark.apache.org
Subject: Setting the number of executors in standalone mode
Hi there,
I try to increase the number of executors per worker in the standalone mode and I have failed to achieve that.
I followed a bit the instructions of this thread: http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores
and did that:
spark.executor.memory 1g
SPARK_WORKER_MEMORY=8g
hoping to get 8 executors per worker but its still 1.
And the option num-executors is not available in the standalone mode.
Thanks a lot!