You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com on 2015/10/03 00:32:16 UTC
how to speed up zeppelin spark job?
how to increase the number of spark executors started by zeppelin ?
--
Deepak
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
Any suggestions
On Thu, Oct 8, 2015 at 11:01 AM ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> I was able to add spark.executor.cores and the max on my cluster is 6.
> This allowed me to run 6 tasks in parallel on each executor, one per core.
>
> I would like to increase the number of executors itself as i have a 15
> node YARN cluster. How do i increase the number of executors from default
> 2.
>
> On Thu, Oct 8, 2015 at 9:21 AM, Mina Lee <mi...@nflabs.com> wrote:
>
>> Did you click "+" button after you added the property?
>>
>> On Fri, Oct 9, 2015 at 1:10 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> Unable to add spark.executor.cores through interpreter.
>>> Each time I add and click save it gets erased.
>>>
>>> On Thu, Oct 8, 2015 at 9:06 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>>> I want to increase number of executors (nodes/containers) and not the
>>>> number of cores per executor.
>>>>
>>>> On Thu, Oct 8, 2015 at 7:41 AM, Felix Cheung <felixcheung_m@hotmail.com
>>>> > wrote:
>>>>
>>>>> I think you should set 'spark.executor.cores' if you are running on
>>>>> YARN
>>>>>
>>>>> http://spark.apache.org/docs/latest/configuration.html#execution-behavior
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> From: deepujain@gmail.com
>>>>> Date: Thu, 8 Oct 2015 07:32:18 -0700
>>>>> Subject: Re: how to speed up zeppelin spark job?
>>>>> To: users@zeppelin.incubator.apache.org
>>>>>
>>>>>
>>>>> I still get only 2 executors and i see only 2 tasks are running
>>>>> concurrently. I would like to increase the number of executors so that the
>>>>> tasks can run in parallel.
>>>>>
>>>>> Any suggestions ?
>>>>>
>>>>> [image: Inline image 1]
>>>>>
>>>>>
>>>>> Only 2 executors received from cluster
>>>>> [image: Inline image 3][image: Inline image 4][image: Inline image 2]
>>>>>
>>>>> On Wed, Oct 7, 2015 at 11:43 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Is this number of cores per executor ? I would like to increase number
>>>>> of executors from 2 to a high value like 300 as I have 300 node cluster
>>>>> On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
>>>>>
>>>>> You can change number of executors by modifying your spark
>>>>> interpreter property `spark.cores.max` in Interpreter tab.
>>>>>
>>>>> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Any suggestions ?
>>>>>
>>>>> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Any suggestions ?
>>>>>
>>>>> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> It always gets three executors, 1 for driver and other 2 for
>>>>> execution. I have 15 data nodes that can be used as executors.
>>>>>
>>>>> I have these in zeppelin-conf
>>>>>
>>>>> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>>>>> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
>>>>> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
>>>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
>>>>> --executor-cores 1"*
>>>>>
>>>>> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> how to increase the number of spark executors started by zeppelin ?
>>>>>
>>>>> --
>>>>> Deepak
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deepak
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deepak
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deepak
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deepak
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>
>
> --
> Deepak
>
>
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
I was able to add spark.executor.cores and the max on my cluster is 6. This
allowed me to run 6 tasks in parallel on each executor, one per core.
I would like to increase the number of executors itself as i have a 15 node
YARN cluster. How do i increase the number of executors from default 2.
On Thu, Oct 8, 2015 at 9:21 AM, Mina Lee <mi...@nflabs.com> wrote:
> Did you click "+" button after you added the property?
>
> On Fri, Oct 9, 2015 at 1:10 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> Unable to add spark.executor.cores through interpreter.
>> Each time I add and click save it gets erased.
>>
>> On Thu, Oct 8, 2015 at 9:06 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> I want to increase number of executors (nodes/containers) and not the
>>> number of cores per executor.
>>>
>>> On Thu, Oct 8, 2015 at 7:41 AM, Felix Cheung <fe...@hotmail.com>
>>> wrote:
>>>
>>>> I think you should set 'spark.executor.cores' if you are running on
>>>> YARN
>>>>
>>>> http://spark.apache.org/docs/latest/configuration.html#execution-behavior
>>>>
>>>>
>>>> ------------------------------
>>>> From: deepujain@gmail.com
>>>> Date: Thu, 8 Oct 2015 07:32:18 -0700
>>>> Subject: Re: how to speed up zeppelin spark job?
>>>> To: users@zeppelin.incubator.apache.org
>>>>
>>>>
>>>> I still get only 2 executors and i see only 2 tasks are running
>>>> concurrently. I would like to increase the number of executors so that the
>>>> tasks can run in parallel.
>>>>
>>>> Any suggestions ?
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>> Only 2 executors received from cluster
>>>> [image: Inline image 3][image: Inline image 4][image: Inline image 2]
>>>>
>>>> On Wed, Oct 7, 2015 at 11:43 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>> wrote:
>>>>
>>>> Is this number of cores per executor ? I would like to increase number
>>>> of executors from 2 to a high value like 300 as I have 300 node cluster
>>>> On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
>>>>
>>>> You can change number of executors by modifying your spark interpreter
>>>> property `spark.cores.max` in Interpreter tab.
>>>>
>>>> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>> wrote:
>>>>
>>>> Any suggestions ?
>>>>
>>>> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>> wrote:
>>>>
>>>> Any suggestions ?
>>>>
>>>> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>> wrote:
>>>>
>>>> It always gets three executors, 1 for driver and other 2 for execution.
>>>> I have 15 data nodes that can be used as executors.
>>>>
>>>> I have these in zeppelin-conf
>>>>
>>>> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>>>> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
>>>> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
>>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
>>>> --executor-cores 1"*
>>>>
>>>> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>> wrote:
>>>>
>>>> how to increase the number of spark executors started by zeppelin ?
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>>
>> --
>> Deepak
>>
>>
>
--
Deepak
Re: how to speed up zeppelin spark job?
Posted by Mina Lee <mi...@nflabs.com>.
Did you click "+" button after you added the property?
On Fri, Oct 9, 2015 at 1:10 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Unable to add spark.executor.cores through interpreter.
> Each time I add and click save it gets erased.
>
> On Thu, Oct 8, 2015 at 9:06 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> I want to increase number of executors (nodes/containers) and not the
>> number of cores per executor.
>>
>> On Thu, Oct 8, 2015 at 7:41 AM, Felix Cheung <fe...@hotmail.com>
>> wrote:
>>
>>> I think you should set 'spark.executor.cores' if you are running on YARN
>>> http://spark.apache.org/docs/latest/configuration.html#execution-behavior
>>>
>>>
>>> ------------------------------
>>> From: deepujain@gmail.com
>>> Date: Thu, 8 Oct 2015 07:32:18 -0700
>>> Subject: Re: how to speed up zeppelin spark job?
>>> To: users@zeppelin.incubator.apache.org
>>>
>>>
>>> I still get only 2 executors and i see only 2 tasks are running
>>> concurrently. I would like to increase the number of executors so that the
>>> tasks can run in parallel.
>>>
>>> Any suggestions ?
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> Only 2 executors received from cluster
>>> [image: Inline image 3][image: Inline image 4][image: Inline image 2]
>>>
>>> On Wed, Oct 7, 2015 at 11:43 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>> Is this number of cores per executor ? I would like to increase number
>>> of executors from 2 to a high value like 300 as I have 300 node cluster
>>> On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
>>>
>>> You can change number of executors by modifying your spark interpreter
>>> property `spark.cores.max` in Interpreter tab.
>>>
>>> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>> Any suggestions ?
>>>
>>> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>> Any suggestions ?
>>>
>>> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>> It always gets three executors, 1 for driver and other 2 for execution.
>>> I have 15 data nodes that can be used as executors.
>>>
>>> I have these in zeppelin-conf
>>>
>>> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>>> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
>>> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
>>> --executor-cores 1"*
>>>
>>> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>> how to increase the number of spark executors started by zeppelin ?
>>>
>>> --
>>> Deepak
>>>
>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>>
>> --
>> Deepak
>>
>>
>
>
> --
> Deepak
>
>
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
Unable to add spark.executor.cores through interpreter.
Each time I add and click save it gets erased.
On Thu, Oct 8, 2015 at 9:06 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> I want to increase number of executors (nodes/containers) and not the
> number of cores per executor.
>
> On Thu, Oct 8, 2015 at 7:41 AM, Felix Cheung <fe...@hotmail.com>
> wrote:
>
>> I think you should set 'spark.executor.cores' if you are running on YARN
>> http://spark.apache.org/docs/latest/configuration.html#execution-behavior
>>
>>
>> ------------------------------
>> From: deepujain@gmail.com
>> Date: Thu, 8 Oct 2015 07:32:18 -0700
>> Subject: Re: how to speed up zeppelin spark job?
>> To: users@zeppelin.incubator.apache.org
>>
>>
>> I still get only 2 executors and i see only 2 tasks are running
>> concurrently. I would like to increase the number of executors so that the
>> tasks can run in parallel.
>>
>> Any suggestions ?
>>
>> [image: Inline image 1]
>>
>>
>> Only 2 executors received from cluster
>> [image: Inline image 3][image: Inline image 4][image: Inline image 2]
>>
>> On Wed, Oct 7, 2015 at 11:43 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>> Is this number of cores per executor ? I would like to increase number of
>> executors from 2 to a high value like 300 as I have 300 node cluster
>> On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
>>
>> You can change number of executors by modifying your spark interpreter
>> property `spark.cores.max` in Interpreter tab.
>>
>> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>> Any suggestions ?
>>
>> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>> Any suggestions ?
>>
>> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>> It always gets three executors, 1 for driver and other 2 for execution. I
>> have 15 data nodes that can be used as executors.
>>
>> I have these in zeppelin-conf
>>
>> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
>> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
>> --executor-cores 1"*
>>
>> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>> how to increase the number of spark executors started by zeppelin ?
>>
>> --
>> Deepak
>>
>>
>>
>>
>> --
>> Deepak
>>
>>
>>
>>
>> --
>> Deepak
>>
>>
>>
>>
>> --
>> Deepak
>>
>>
>>
>>
>>
>> --
>> Deepak
>>
>>
>
>
> --
> Deepak
>
>
--
Deepak
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
I want to increase number of executors (nodes/containers) and not the
number of cores per executor.
On Thu, Oct 8, 2015 at 7:41 AM, Felix Cheung <fe...@hotmail.com>
wrote:
> I think you should set 'spark.executor.cores' if you are running on YARN
> http://spark.apache.org/docs/latest/configuration.html#execution-behavior
>
>
> ------------------------------
> From: deepujain@gmail.com
> Date: Thu, 8 Oct 2015 07:32:18 -0700
> Subject: Re: how to speed up zeppelin spark job?
> To: users@zeppelin.incubator.apache.org
>
>
> I still get only 2 executors and i see only 2 tasks are running
> concurrently. I would like to increase the number of executors so that the
> tasks can run in parallel.
>
> Any suggestions ?
>
> [image: Inline image 1]
>
>
> Only 2 executors received from cluster
> [image: Inline image 3][image: Inline image 4][image: Inline image 2]
>
> On Wed, Oct 7, 2015 at 11:43 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
> wrote:
>
> Is this number of cores per executor ? I would like to increase number of
> executors from 2 to a high value like 300 as I have 300 node cluster
> On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
>
> You can change number of executors by modifying your spark interpreter
> property `spark.cores.max` in Interpreter tab.
>
> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
> Any suggestions ?
>
> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
> Any suggestions ?
>
> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
> It always gets three executors, 1 for driver and other 2 for execution. I
> have 15 data nodes that can be used as executors.
>
> I have these in zeppelin-conf
>
> export JAVA_HOME=/usr/src/jdk1.7.0_79/
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
> --executor-cores 1"*
>
> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
> how to increase the number of spark executors started by zeppelin ?
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
>
>
>
> --
> Deepak
>
>
--
Deepak
RE: how to speed up zeppelin spark job?
Posted by Felix Cheung <fe...@hotmail.com>.
I think you should set 'spark.executor.cores' if you are running on YARN
http://spark.apache.org/docs/latest/configuration.html#execution-behavior
From: deepujain@gmail.com
Date: Thu, 8 Oct 2015 07:32:18 -0700
Subject: Re: how to speed up zeppelin spark job?
To: users@zeppelin.incubator.apache.org
I still get only 2 executors and i see only 2 tasks are running concurrently. I would like to increase the number of executors so that the tasks can run in parallel.
Any suggestions ?
Only 2 executors received from cluster
On Wed, Oct 7, 2015 at 11:43 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
Is this number of cores per executor ? I would like to increase number of executors from 2 to a high value like 300 as I have 300 node cluster
On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
You can change number of executors by modifying your spark interpreter property `spark.cores.max` in Interpreter tab.
On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
Any suggestions ?
On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
Any suggestions ?
On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
It always gets three executors, 1 for driver and other 2 for execution. I have 15 data nodes that can be used as executors.
I have these in zeppelin-conf
export JAVA_HOME=/usr/src/jdk1.7.0_79/
export HADOOP_CONF_DIR=/etc/hadoop/conf
export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g --executor-cores 1"
On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
how to increase the number of spark executors started by zeppelin ?
--
Deepak
--
Deepak
--
Deepak
--
Deepak
--
Deepak
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
I still get only 2 executors and i see only 2 tasks are running
concurrently. I would like to increase the number of executors so that the
tasks can run in parallel.
Any suggestions ?
[image: Inline image 1]
Only 2 executors received from cluster
[image: Inline image 3][image: Inline image 4][image: Inline image 2]
On Wed, Oct 7, 2015 at 11:43 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Is this number of cores per executor ? I would like to increase number of
> executors from 2 to a high value like 300 as I have 300 node cluster
> On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
>
>> You can change number of executors by modifying your spark interpreter
>> property `spark.cores.max` in Interpreter tab.
>>
>> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> Any suggestions ?
>>>
>>> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>>> Any suggestions ?
>>>>
>>>> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>> wrote:
>>>>
>>>>> It always gets three executors, 1 for driver and other 2 for
>>>>> execution. I have 15 data nodes that can be used as executors.
>>>>>
>>>>> I have these in zeppelin-conf
>>>>>
>>>>> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>>>>>
>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>
>>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>>>>>
>>>>> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
>>>>> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
>>>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
>>>>> --executor-cores 1"*
>>>>>
>>>>> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> how to increase the number of spark executors started by zeppelin ?
>>>>>>
>>>>>> --
>>>>>> Deepak
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deepak
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
--
Deepak
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
Worked
On Thu, Oct 8, 2015 at 10:25 PM Felix Cheung <fe...@hotmail.com>
wrote:
> +1
> spark.executor.instances
> http://spark.apache.org/docs/latest/running-on-yarn.html
>
> ------------------------------
> Date: Fri, 9 Oct 2015 10:26:08 +0530
> From: praagarw@gmail.com
> To: users@zeppelin.incubator.apache.org
>
> Subject: Re: how to speed up zeppelin spark job?
>
> try spark.executor.instances=N
> and to increase the memory per instance try spark.executor.memory=Nmb
>
> Regards,
> -Pranav.
>
> On 08/10/15 12:13 pm, ÐΞ€ρ@Ҝ (๏̯͡๏) wrote:
>
> Is this number of cores per executor ? I would like to increase number of
> executors from 2 to a high value like 300 as I have 300 node cluster
> On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
>
> You can change number of executors by modifying your spark interpreter
> property `spark.cores.max` in Interpreter tab.
>
> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
> Any suggestions ?
>
> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
> Any suggestions ?
>
> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
> It always gets three executors, 1 for driver and other 2 for execution. I
> have 15 data nodes that can be used as executors.
>
> I have these in zeppelin-conf
>
> export JAVA_HOME=/usr/src/jdk1.7.0_79/
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
> --executor-cores 1"*
>
> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
> how to increase the number of spark executors started by zeppelin ?
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
>
>
RE: how to speed up zeppelin spark job?
Posted by Felix Cheung <fe...@hotmail.com>.
+1
spark.executor.instances
http://spark.apache.org/docs/latest/running-on-yarn.html
Date: Fri, 9 Oct 2015 10:26:08 +0530
From: praagarw@gmail.com
To: users@zeppelin.incubator.apache.org
Subject: Re: how to speed up zeppelin spark job?
try spark.executor.instances=N
and to increase the memory per instance try
spark.executor.memory=Nmb
Regards,
-Pranav.
On 08/10/15 12:13 pm, ÐΞ€ρ@Ҝ (๏̯͡๏)
wrote:
Is this number of cores per executor ? I would like to
increase number of executors from 2 to a high value like 300 as I
have 300 node cluster
On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com>
wrote:
You can change number of executors by modifying
your spark interpreter property `spark.cores.max` in
Interpreter tab.
On Thu, Oct 8, 2015 at 2:22 AM,
ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
wrote:
Any suggestions ?
On Sun, Oct 4, 2015 at
9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
wrote:
Any suggestions ?
On Fri, Oct 2,
2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
wrote:
It always gets three
executors, 1 for driver and other
2 for execution. I have 15 data
nodes that can be used as
executors.
I have these in zeppelin-conf
export
JAVA_HOME=/usr/src/jdk1.7.0_79/
export
HADOOP_CONF_DIR=/etc/hadoop/conf
export
ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
export
SPARK_SUBMIT_OPTIONS="--num-executors
15 --driver-memory 14g
--driver-java-options
-XX:MaxPermSize=512M
-Xmx4096M -Xms4096M
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
--executor-memory 14g
--executor-cores 1"
On
Fri, Oct 2, 2015 at 3:32 PM,
ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
wrote:
how to
increase the number of
spark executors started
by zeppelin ?
--
Deepak
--
Deepak
--
Deepak
--
Deepak
Re: how to speed up zeppelin spark job?
Posted by Pranav Kumar Agarwal <pr...@gmail.com>.
try spark.executor.instances=N
and to increase the memory per instance try spark.executor.memory=Nmb
Regards,
-Pranav.
On 08/10/15 12:13 pm, ÐΞ€ρ@Ҝ (๏̯͡๏) wrote:
> Is this number of cores per executor ? I would like to increase number
> of executors from 2 to a high value like 300 as I have 300 node cluster
> On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <minalee@nflabs.com
> <ma...@nflabs.com>> wrote:
>
> You can change number of executors by modifying your spark
> interpreter property `spark.cores.max` in Interpreter tab.
>
> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepujain@gmail.com
> <ma...@gmail.com>> wrote:
>
> Any suggestions ?
>
> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏)
> <deepujain@gmail.com <ma...@gmail.com>> wrote:
>
> Any suggestions ?
>
> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)
> <deepujain@gmail.com <ma...@gmail.com>> wrote:
>
> It always gets three executors, 1 for driver and other
> 2 for execution. I have 15 data nodes that can be used
> as executors.
>
> I have these in zeppelin-conf
>
> exportJAVA_HOME=/usr/src/jdk1.7.0_79/
>
> exportHADOOP_CONF_DIR=/etc/hadoop/conf
>
> exportZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>
> *exportSPARK_SUBMIT_OPTIONS="--num-executors 15
> --driver-memory 14g --driver-java-options
> -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> --executor-memory 14g --executor-cores 1"*
>
>
> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)
> <deepujain@gmail.com <ma...@gmail.com>> wrote:
>
> how to increase the number of spark executors
> started by zeppelin ?
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
>
>
> --
> Deepak
>
>
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
Is this number of cores per executor ? I would like to increase number of
executors from 2 to a high value like 300 as I have 300 node cluster
On Wed, Oct 7, 2015 at 9:24 PM Mina Lee <mi...@nflabs.com> wrote:
> You can change number of executors by modifying your spark interpreter
> property `spark.cores.max` in Interpreter tab.
>
> On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> Any suggestions ?
>>
>> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> Any suggestions ?
>>>
>>> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>>> It always gets three executors, 1 for driver and other 2 for execution.
>>>> I have 15 data nodes that can be used as executors.
>>>>
>>>> I have these in zeppelin-conf
>>>>
>>>> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>>>>
>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>
>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>>>>
>>>> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
>>>> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
>>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
>>>> --executor-cores 1"*
>>>>
>>>> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>>> wrote:
>>>>
>>>>> how to increase the number of spark executors started by zeppelin ?
>>>>>
>>>>> --
>>>>> Deepak
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>>
>> --
>> Deepak
>>
>>
>
Re: how to speed up zeppelin spark job?
Posted by Mina Lee <mi...@nflabs.com>.
You can change number of executors by modifying your spark interpreter
property `spark.cores.max` in Interpreter tab.
On Thu, Oct 8, 2015 at 2:22 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Any suggestions ?
>
> On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> Any suggestions ?
>>
>> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> It always gets three executors, 1 for driver and other 2 for execution.
>>> I have 15 data nodes that can be used as executors.
>>>
>>> I have these in zeppelin-conf
>>>
>>> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>>>
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>
>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>>>
>>> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
>>> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
>>> --executor-cores 1"*
>>>
>>> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>>> how to increase the number of spark executors started by zeppelin ?
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>>
>> --
>> Deepak
>>
>>
>
>
> --
> Deepak
>
>
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
Any suggestions ?
On Sun, Oct 4, 2015 at 9:26 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Any suggestions ?
>
> On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> It always gets three executors, 1 for driver and other 2 for execution. I
>> have 15 data nodes that can be used as executors.
>>
>> I have these in zeppelin-conf
>>
>> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>>
>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>
>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>>
>> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
>> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
>> --executor-cores 1"*
>>
>> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> how to increase the number of spark executors started by zeppelin ?
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>>
>> --
>> Deepak
>>
>>
>
>
> --
> Deepak
>
>
--
Deepak
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
Any suggestions ?
On Fri, Oct 2, 2015 at 3:40 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> It always gets three executors, 1 for driver and other 2 for execution. I
> have 15 data nodes that can be used as executors.
>
> I have these in zeppelin-conf
>
> export JAVA_HOME=/usr/src/jdk1.7.0_79/
>
> export HADOOP_CONF_DIR=/etc/hadoop/conf
>
> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
>
> *export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
> --driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
> --executor-cores 1"*
>
> On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> how to increase the number of spark executors started by zeppelin ?
>>
>> --
>> Deepak
>>
>>
>
>
> --
> Deepak
>
>
--
Deepak
Re: how to speed up zeppelin spark job?
Posted by ๏̯͡๏ <ÐΞ€ρ@Ҝ>,
de...@gmail.com.
It always gets three executors, 1 for driver and other 2 for execution. I
have 15 data nodes that can be used as executors.
I have these in zeppelin-conf
export JAVA_HOME=/usr/src/jdk1.7.0_79/
export HADOOP_CONF_DIR=/etc/hadoop/conf
export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.3.1.0-2574"
*export SPARK_SUBMIT_OPTIONS="--num-executors 15 --driver-memory 14g
--driver-java-options -XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps --executor-memory 14g
--executor-cores 1"*
On Fri, Oct 2, 2015 at 3:32 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> how to increase the number of spark executors started by zeppelin ?
>
> --
> Deepak
>
>
--
Deepak