You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by kant kodali <ka...@gmail.com> on 2017/02/18 00:55:46 UTC
question on SPARK_WORKER_CORES
when I submit a job using spark shell I get something like this
[Stage 0:========> (36814 + 4) / 220129]
Now all I want is I want to increase number of parallel tasks running from
4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in
conf/spark-env.sh. I though that should do it but it doesn't. It still
shows me 4. any idea?
Thanks much!
Re: question on SPARK_WORKER_CORES
Posted by "颜发才 (Yan Facai)" <fa...@gmail.com>.
Hi, kodali.
SPARK_WORKER_CORES is designed for cluster resource manager, see
http://spark.apache.org/docs/latest/cluster-overview.html if interested.
For standalone mode,
you should use the following 3 arguments to allocate resource for normal
spark tasks:
- --executor-memory
- --executor-cores
- --total-executor-cores
and the meaning is as below:
- Executor memory: --executor-memory
- Executor cores: --executor-cores
- Number of executors: --total-executor-cores/--executor-cores
more details see
http://spark.apache.org/docs/latest/submitting-applications.html.
On Sat, Feb 18, 2017 at 9:20 AM, kant kodali <ka...@gmail.com> wrote:
> Hi Satish,
>
> I am using spark 2.0.2. And no I have not passed those variables because
> I didn't want to shoot in the dark. According to the documentation it looks
> like SPARK_WORKER_CORES is the one which should do it. If not, can you
> please explain how these variables inter play together?
>
> --num-executors
> --executor-cores
> –total-executor-cores
> SPARK_WORKER_CORES
>
> Thanks!
>
>
> On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam <sa...@microsoft.com>
> wrote:
>
>> Have you tried passing --executor-cores or –total-executor-cores as
>> arguments, , depending on the spark version?
>>
>>
>>
>>
>>
>> *From:* kant kodali [mailto:kanth909@gmail.com]
>> *Sent:* Friday, February 17, 2017 5:03 PM
>> *To:* Alex Kozlov <al...@gmail.com>
>> *Cc:* user @spark <us...@spark.apache.org>
>> *Subject:* Re: question on SPARK_WORKER_CORES
>>
>>
>>
>> Standalone.
>>
>>
>>
>> On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov <al...@gmail.com> wrote:
>>
>> What Spark mode are you running the program in?
>>
>>
>>
>> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali <ka...@gmail.com> wrote:
>>
>> when I submit a job using spark shell I get something like this
>>
>>
>>
>> [Stage 0:========> (36814 + 4) / 220129]
>>
>>
>>
>> Now all I want is I want to increase number of parallel tasks running
>> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in
>> conf/spark-env.sh. I though that should do it but it doesn't. It still
>> shows me 4. any idea?
>>
>>
>>
>> Thanks much!
>>
>>
>>
>>
>>
>> --
>>
>> Alex Kozlov
>> (408) 507-4987
>> (650) 887-2135 efax
>> alexvk@gmail.com
>>
>>
>>
>
>
Re: question on SPARK_WORKER_CORES
Posted by kant kodali <ka...@gmail.com>.
one executor per Spark slave should be fine right I am not really sure what
benefit one would get by starting more executors (jvm's) on one node? End
of the day JVM creates native/kernel threads through system calls so if
those threads are spawned by one or multiple processes I dont see much
benefit (In theory it should be the same). With different processes one
would get different address spaces in the kernel but memory isn't an issue
so far.
On Fri, Feb 17, 2017 at 5:32 PM, Alex Kozlov <al...@gmail.com> wrote:
> I found in some previous CDH versions that Spark starts only one executor
> per Spark slave, and DECREASING the executor-cores in standalone makes
> the total # of executors go up. Just my 2¢.
>
> On Fri, Feb 17, 2017 at 5:20 PM, kant kodali <ka...@gmail.com> wrote:
>
>> Hi Satish,
>>
>> I am using spark 2.0.2. And no I have not passed those variables because
>> I didn't want to shoot in the dark. According to the documentation it looks
>> like SPARK_WORKER_CORES is the one which should do it. If not, can you
>> please explain how these variables inter play together?
>>
>> --num-executors
>> --executor-cores
>> –total-executor-cores
>> SPARK_WORKER_CORES
>>
>> Thanks!
>>
>>
>> On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam <sa...@microsoft.com>
>> wrote:
>>
>>> Have you tried passing --executor-cores or –total-executor-cores as
>>> arguments, , depending on the spark version?
>>>
>>>
>>>
>>>
>>>
>>> *From:* kant kodali [mailto:kanth909@gmail.com]
>>> *Sent:* Friday, February 17, 2017 5:03 PM
>>> *To:* Alex Kozlov <al...@gmail.com>
>>> *Cc:* user @spark <us...@spark.apache.org>
>>> *Subject:* Re: question on SPARK_WORKER_CORES
>>>
>>>
>>>
>>> Standalone.
>>>
>>>
>>>
>>> On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov <al...@gmail.com> wrote:
>>>
>>> What Spark mode are you running the program in?
>>>
>>>
>>>
>>> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali <ka...@gmail.com> wrote:
>>>
>>> when I submit a job using spark shell I get something like this
>>>
>>>
>>>
>>> [Stage 0:========> (36814 + 4) / 220129]
>>>
>>>
>>>
>>> Now all I want is I want to increase number of parallel tasks running
>>> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in
>>> conf/spark-env.sh. I though that should do it but it doesn't. It still
>>> shows me 4. any idea?
>>>
>>>
>>>
>>> Thanks much!
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Alex Kozlov
>>> (408) 507-4987
>>> (650) 887-2135 efax
>>> alexvk@gmail.com
>>>
>>>
>>>
>>
>>
>
>
> --
> Alex Kozlov
> (408) 507-4987
> (650) 887-2135 efax
> alexvk@gmail.com
>
Re: question on SPARK_WORKER_CORES
Posted by Alex Kozlov <al...@gmail.com>.
I found in some previous CDH versions that Spark starts only one executor
per Spark slave, and DECREASING the executor-cores in standalone makes the
total # of executors go up. Just my 2¢.
On Fri, Feb 17, 2017 at 5:20 PM, kant kodali <ka...@gmail.com> wrote:
> Hi Satish,
>
> I am using spark 2.0.2. And no I have not passed those variables because
> I didn't want to shoot in the dark. According to the documentation it looks
> like SPARK_WORKER_CORES is the one which should do it. If not, can you
> please explain how these variables inter play together?
>
> --num-executors
> --executor-cores
> –total-executor-cores
> SPARK_WORKER_CORES
>
> Thanks!
>
>
> On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam <sa...@microsoft.com>
> wrote:
>
>> Have you tried passing --executor-cores or –total-executor-cores as
>> arguments, , depending on the spark version?
>>
>>
>>
>>
>>
>> *From:* kant kodali [mailto:kanth909@gmail.com]
>> *Sent:* Friday, February 17, 2017 5:03 PM
>> *To:* Alex Kozlov <al...@gmail.com>
>> *Cc:* user @spark <us...@spark.apache.org>
>> *Subject:* Re: question on SPARK_WORKER_CORES
>>
>>
>>
>> Standalone.
>>
>>
>>
>> On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov <al...@gmail.com> wrote:
>>
>> What Spark mode are you running the program in?
>>
>>
>>
>> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali <ka...@gmail.com> wrote:
>>
>> when I submit a job using spark shell I get something like this
>>
>>
>>
>> [Stage 0:========> (36814 + 4) / 220129]
>>
>>
>>
>> Now all I want is I want to increase number of parallel tasks running
>> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in
>> conf/spark-env.sh. I though that should do it but it doesn't. It still
>> shows me 4. any idea?
>>
>>
>>
>> Thanks much!
>>
>>
>>
>>
>>
>> --
>>
>> Alex Kozlov
>> (408) 507-4987
>> (650) 887-2135 efax
>> alexvk@gmail.com
>>
>>
>>
>
>
--
Alex Kozlov
(408) 507-4987
(650) 887-2135 efax
alexvk@gmail.com
Re: question on SPARK_WORKER_CORES
Posted by kant kodali <ka...@gmail.com>.
Hi Satish,
I am using spark 2.0.2. And no I have not passed those variables because I
didn't want to shoot in the dark. According to the documentation it looks
like SPARK_WORKER_CORES is the one which should do it. If not, can you
please explain how these variables inter play together?
--num-executors
--executor-cores
–total-executor-cores
SPARK_WORKER_CORES
Thanks!
On Fri, Feb 17, 2017 at 5:13 PM, Satish Lalam <sa...@microsoft.com> wrote:
> Have you tried passing --executor-cores or –total-executor-cores as
> arguments, , depending on the spark version?
>
>
>
>
>
> *From:* kant kodali [mailto:kanth909@gmail.com]
> *Sent:* Friday, February 17, 2017 5:03 PM
> *To:* Alex Kozlov <al...@gmail.com>
> *Cc:* user @spark <us...@spark.apache.org>
> *Subject:* Re: question on SPARK_WORKER_CORES
>
>
>
> Standalone.
>
>
>
> On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov <al...@gmail.com> wrote:
>
> What Spark mode are you running the program in?
>
>
>
> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali <ka...@gmail.com> wrote:
>
> when I submit a job using spark shell I get something like this
>
>
>
> [Stage 0:========> (36814 + 4) / 220129]
>
>
>
> Now all I want is I want to increase number of parallel tasks running from
> 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in
> conf/spark-env.sh. I though that should do it but it doesn't. It still
> shows me 4. any idea?
>
>
>
> Thanks much!
>
>
>
>
>
> --
>
> Alex Kozlov
> (408) 507-4987
> (650) 887-2135 efax
> alexvk@gmail.com
>
>
>
RE: question on SPARK_WORKER_CORES
Posted by Satish Lalam <sa...@microsoft.com.INVALID>.
Have you tried passing --executor-cores or –total-executor-cores as arguments, , depending on the spark version?
From: kant kodali [mailto:kanth909@gmail.com]
Sent: Friday, February 17, 2017 5:03 PM
To: Alex Kozlov <al...@gmail.com>
Cc: user @spark <us...@spark.apache.org>
Subject: Re: question on SPARK_WORKER_CORES
Standalone.
On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov <al...@gmail.com>> wrote:
What Spark mode are you running the program in?
On Fri, Feb 17, 2017 at 4:55 PM, kant kodali <ka...@gmail.com>> wrote:
when I submit a job using spark shell I get something like this
[Stage 0:========> (36814 + 4) / 220129]
Now all I want is I want to increase number of parallel tasks running from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in conf/spark-env.sh. I though that should do it but it doesn't. It still shows me 4. any idea?
Thanks much!
--
Alex Kozlov
(408) 507-4987<tel:(408)%20507-4987>
(650) 887-2135<tel:(650)%20887-2135> efax
alexvk@gmail.com<ma...@gmail.com>
Re: question on SPARK_WORKER_CORES
Posted by kant kodali <ka...@gmail.com>.
Standalone.
On Fri, Feb 17, 2017 at 5:01 PM, Alex Kozlov <al...@gmail.com> wrote:
> What Spark mode are you running the program in?
>
> On Fri, Feb 17, 2017 at 4:55 PM, kant kodali <ka...@gmail.com> wrote:
>
>> when I submit a job using spark shell I get something like this
>>
>> [Stage 0:========> (36814 + 4) / 220129]
>>
>>
>> Now all I want is I want to increase number of parallel tasks running
>> from 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in
>> conf/spark-env.sh. I though that should do it but it doesn't. It still
>> shows me 4. any idea?
>>
>>
>> Thanks much!
>>
>>
>>
>
>
> --
> Alex Kozlov
> (408) 507-4987
> (650) 887-2135 efax
> alexvk@gmail.com
>
Re: question on SPARK_WORKER_CORES
Posted by Alex Kozlov <al...@gmail.com>.
What Spark mode are you running the program in?
On Fri, Feb 17, 2017 at 4:55 PM, kant kodali <ka...@gmail.com> wrote:
> when I submit a job using spark shell I get something like this
>
> [Stage 0:========> (36814 + 4) / 220129]
>
>
> Now all I want is I want to increase number of parallel tasks running from
> 4 to 16 so I exported an env variable called SPARK_WORKER_CORES=16 in
> conf/spark-env.sh. I though that should do it but it doesn't. It still
> shows me 4. any idea?
>
>
> Thanks much!
>
>
>
--
Alex Kozlov
(408) 507-4987
(650) 887-2135 efax
alexvk@gmail.com