You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@zeppelin.apache.org by Gerald Loeffler <ge...@googlemail.com> on 2016/02/03 12:31:55 UTC

increasing Spark driver memory using Zeppelin with Spark/YARN

dear Zeppelin users,

we’ve been using Zeppelin (0.5.5) with Spark/YARN successfully for quite
some time but now
we need a Spark driver with lots of memory and fail to achieve that using
Zeppelin (it works without issues using direct spark-submit). Currently
we’re using yarn-client mode.

   1. is the Spark driver executing in the same JVM as the Zeppelin
   RemoteInterpreterServer?
   2. how does one correctly size the Spark driver memory (Xmx) in this
   setting?
   3.

   as an alternative: is yarn-master supported with Zeppelin?


   thank you very much in advance for your help!,
   gerald


--
Gerald Loeffler
mailto:gerald.loeffler@googlemail.com
http://www.gerald-loeffler.net

Re: increasing Spark driver memory using Zeppelin with Spark/YARN

Posted by Hyung Sung Shim <hs...@nflabs.com>.

Hello.

I commented below your questions.

1. is the Spark driver executing in the same JVM as the Zeppelin
RemoteInterpreterServer?
*-> I know that is sperated. The interpreter server process is executed by
~/bin/interpreter.sh*

2. how does one correctly size the Spark driver memory (Xmx) in this
setting?
*->  you can refer to
http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Can-not-configure-driver-memory-size-td1513.html
<http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Can-not-configure-driver-memory-size-td1513.html>*

3. as an alternative: is yarn-master supported with Zeppelin?
*-> **I know that **zeppelin support only yarn-client mode.*

If I'm wrong, please fix.
Thanks.

2016-02-03 20:31 GMT+09:00 Gerald Loeffler <ge...@googlemail.com>:

> dear Zeppelin users,
>
> we’ve been using Zeppelin (0.5.5) with Spark/YARN successfully for quite
> some time but now
> we need a Spark driver with lots of memory and fail to achieve that using
> Zeppelin (it works without issues using direct spark-submit). Currently
> we’re using yarn-client mode.
>
>    1. is the Spark driver executing in the same JVM as the Zeppelin
>    RemoteInterpreterServer?
>    2. how does one correctly size the Spark driver memory (Xmx) in this
>    setting?
>    3.
>
>    as an alternative: is yarn-master supported with Zeppelin?
>
>
>    thank you very much in advance for your help!,
>    gerald
>
>
> --
> Gerald Loeffler
> mailto:gerald.loeffler@googlemail.com
> http://www.gerald-loeffler.net
>