You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Ophir Cohen <op...@gmail.com> on 2015/04/05 20:05:29 UTC
Fwd: Can set number of executors on yarn-client mode
Hi All,
I'm using Zeppelin on top Spark that runs on Yarn.
For some reason I can't set the number of executors.
With any configuration it gets only 3 cores.
I tried to change in two places:
1. conf/zeppelin-env.sh
2. interpreter.json
In both of them I tried to set "num-executors": "10" but I still get 3
(it's a 3 nodes cluster - I have 12 available cores).
Any idea?
Thanks!
Ophir
BTW
I also tried to pass compression variables to be used in the underline Hive
metastore and no luck as well - it seems that it does not passed.
Re: Can set number of executors on yarn-client mode
Posted by Ophir Cohen <op...@gmail.com>.
Succeded!
I updated ZEPPELIN_JAVA_OPTS:
export ZEPPELIN_JAVA_OPTS="-Dspark.executor.instances=6"
For some reason I couldn't find the right variable
(spark.executor.instances).
Thank you very much
On Mon, Apr 6, 2015 at 8:36 AM, Jongyoul Lee <jo...@gmail.com> wrote:
> Hi Ophir,
>
> If you don't use the latest master, you can use ZEPPELIN_JAVA_OPTS instead.
>
> JL
>
> On Mon, Apr 6, 2015 at 2:32 PM, Ophir Cohen <op...@gmail.com> wrote:
>
>> Thanks!
>> Trying...
>> BTW
>> What is INTP?
>>
>> On Mon, Apr 6, 2015 at 7:35 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>>
>>> Hi Ophir,
>>>
>>> You can set them by the following lines in you zeppelin-env.sh:
>>>
>>> -DZEPPELIN_INTP_JAVA_OPTS="-Dspark.executor.instances=10"
>>>
>>> JL.
>>>
>>> On Mon, Apr 6, 2015 at 3:05 AM, Ophir Cohen <op...@gmail.com> wrote:
>>>
>>>>
>>>> Hi All,
>>>> I'm using Zeppelin on top Spark that runs on Yarn.
>>>> For some reason I can't set the number of executors.
>>>> With any configuration it gets only 3 cores.
>>>> I tried to change in two places:
>>>> 1. conf/zeppelin-env.sh
>>>> 2. interpreter.json
>>>> In both of them I tried to set "num-executors": "10" but I still get 3
>>>> (it's a 3 nodes cluster - I have 12 available cores).
>>>>
>>>> Any idea?
>>>> Thanks!
>>>> Ophir
>>>>
>>>> BTW
>>>> I also tried to pass compression variables to be used in the underline
>>>> Hive metastore and no luck as well - it seems that it does not passed.
>>>>
>>>
>>>
>>>
>>> --
>>> 이종열, Jongyoul Lee, 李宗烈
>>> http://madeng.net
>>>
>>
>>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
Re: Can set number of executors on yarn-client mode
Posted by Jongyoul Lee <jo...@gmail.com>.
Hi Ophir,
If you don't use the latest master, you can use ZEPPELIN_JAVA_OPTS instead.
JL
On Mon, Apr 6, 2015 at 2:32 PM, Ophir Cohen <op...@gmail.com> wrote:
> Thanks!
> Trying...
> BTW
> What is INTP?
>
> On Mon, Apr 6, 2015 at 7:35 AM, Jongyoul Lee <jo...@gmail.com> wrote:
>
>> Hi Ophir,
>>
>> You can set them by the following lines in you zeppelin-env.sh:
>>
>> -DZEPPELIN_INTP_JAVA_OPTS="-Dspark.executor.instances=10"
>>
>> JL.
>>
>> On Mon, Apr 6, 2015 at 3:05 AM, Ophir Cohen <op...@gmail.com> wrote:
>>
>>>
>>> Hi All,
>>> I'm using Zeppelin on top Spark that runs on Yarn.
>>> For some reason I can't set the number of executors.
>>> With any configuration it gets only 3 cores.
>>> I tried to change in two places:
>>> 1. conf/zeppelin-env.sh
>>> 2. interpreter.json
>>> In both of them I tried to set "num-executors": "10" but I still get 3
>>> (it's a 3 nodes cluster - I have 12 available cores).
>>>
>>> Any idea?
>>> Thanks!
>>> Ophir
>>>
>>> BTW
>>> I also tried to pass compression variables to be used in the underline
>>> Hive metastore and no luck as well - it seems that it does not passed.
>>>
>>
>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: Can set number of executors on yarn-client mode
Posted by Ophir Cohen <op...@gmail.com>.
Thanks!
Trying...
BTW
What is INTP?
On Mon, Apr 6, 2015 at 7:35 AM, Jongyoul Lee <jo...@gmail.com> wrote:
> Hi Ophir,
>
> You can set them by the following lines in you zeppelin-env.sh:
>
> -DZEPPELIN_INTP_JAVA_OPTS="-Dspark.executor.instances=10"
>
> JL.
>
> On Mon, Apr 6, 2015 at 3:05 AM, Ophir Cohen <op...@gmail.com> wrote:
>
>>
>> Hi All,
>> I'm using Zeppelin on top Spark that runs on Yarn.
>> For some reason I can't set the number of executors.
>> With any configuration it gets only 3 cores.
>> I tried to change in two places:
>> 1. conf/zeppelin-env.sh
>> 2. interpreter.json
>> In both of them I tried to set "num-executors": "10" but I still get 3
>> (it's a 3 nodes cluster - I have 12 available cores).
>>
>> Any idea?
>> Thanks!
>> Ophir
>>
>> BTW
>> I also tried to pass compression variables to be used in the underline
>> Hive metastore and no luck as well - it seems that it does not passed.
>>
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
Re: Can set number of executors on yarn-client mode
Posted by Jongyoul Lee <jo...@gmail.com>.
Hi Ophir,
You can set them by the following lines in you zeppelin-env.sh:
-DZEPPELIN_INTP_JAVA_OPTS="-Dspark.executor.instances=10"
JL.
On Mon, Apr 6, 2015 at 3:05 AM, Ophir Cohen <op...@gmail.com> wrote:
>
> Hi All,
> I'm using Zeppelin on top Spark that runs on Yarn.
> For some reason I can't set the number of executors.
> With any configuration it gets only 3 cores.
> I tried to change in two places:
> 1. conf/zeppelin-env.sh
> 2. interpreter.json
> In both of them I tried to set "num-executors": "10" but I still get 3
> (it's a 3 nodes cluster - I have 12 available cores).
>
> Any idea?
> Thanks!
> Ophir
>
> BTW
> I also tried to pass compression variables to be used in the underline
> Hive metastore and no luck as well - it seems that it does not passed.
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net