You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Joel Wilsson <jo...@gmail.com> on 2018/05/27 12:22:25 UTC
0.8.0-rc3 Spark executor has only one core
Hi,
After downloading and unpacking the release at
https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.8.0-rc3/
(the bin-all version), running the first paragraph of the notebook
"Zeppelin Tutorial/Basic Features (Spark)", and visiting
http://localhost:4040/executors/ I see one executor with one core.
Doing the same thing with 0.7.3 shows one executor with 16 cores.
I've tried adjusting SPARK_SUBMIT_OPTIONS (adding --executor-cores 4),
changed master to "local[4]" in the interpreter settings, added
spark.executor.cores with a value of 4, and setting spark.cores.max to
4, but the executor still only has one core in 0.8.0.
Has anyone managed to get an executor with more than one core when using
Zeppelin 0.8.0 in standalone mode?
Is there anything else I can try?
Best regards,
Joel
Re: 0.8.0-rc3 Spark executor has only one core
Posted by Joel Wilsson <jo...@gmail.com>.
Thanks Jeff, that fixed it. After setting SPARK_HOME the settings are
picked up and there's one executor with 4 cores. Perfect!
Is this change for the embedded Spark mode intentional?
The way it works in 0.7.3 was better, in my opinion.
On Sun, May 27, 2018 at 3:20 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
> It seems happen only in the embedded spark mode. As long as you set
> SPARK_HOME, you can get the expected cores.
>
>
> Joel Wilsson <jo...@gmail.com>于2018年5月27日周日 下午8:22写道：
>
>> Hi,
>>
>> After downloading and unpacking the release at
>> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.8.0-rc3/
>> (the bin-all version), running the first paragraph of the notebook
>> "Zeppelin Tutorial/Basic Features (Spark)", and visiting
>> http://localhost:4040/executors/ I see one executor with one core.
>>
>> Doing the same thing with 0.7.3 shows one executor with 16 cores.
>>
>> I've tried adjusting SPARK_SUBMIT_OPTIONS (adding --executor-cores 4),
>> changed master to "local[4]" in the interpreter settings, added
>> spark.executor.cores with a value of 4, and setting spark.cores.max to
>> 4, but the executor still only has one core in 0.8.0.
>>
>> Has anyone managed to get an executor with more than one core when using
>> Zeppelin 0.8.0 in standalone mode?
>>
>> Is there anything else I can try?
>>
>> Best regards,
>> Joel
>>
>>
Re: 0.8.0-rc3 Spark executor has only one core
Posted by Jeff Zhang <zj...@gmail.com>.
It seems happen only in the embedded spark mode. As long as you set
SPARK_HOME, you can get the expected cores.
Joel Wilsson <jo...@gmail.com>于2018年5月27日周日 下午8:22写道：
> Hi,
>
> After downloading and unpacking the release at
> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.8.0-rc3/
> (the bin-all version), running the first paragraph of the notebook
> "Zeppelin Tutorial/Basic Features (Spark)", and visiting
> http://localhost:4040/executors/ I see one executor with one core.
>
> Doing the same thing with 0.7.3 shows one executor with 16 cores.
>
> I've tried adjusting SPARK_SUBMIT_OPTIONS (adding --executor-cores 4),
> changed master to "local[4]" in the interpreter settings, added
> spark.executor.cores with a value of 4, and setting spark.cores.max to
> 4, but the executor still only has one core in 0.8.0.
>
> Has anyone managed to get an executor with more than one core when using
> Zeppelin 0.8.0 in standalone mode?
>
> Is there anything else I can try?
>
> Best regards,
> Joel
>
>