You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by moon soo Lee <mo...@apache.org> on 2016/07/01 16:04:37 UTC
Re: spark interpreter
0.6.0 is currently in vote in dev@ list.
http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html
Thanks,
moon
On Thu, Jun 30, 2016 at 1:54 PM Leon Katsnelson <le...@ca.ibm.com> wrote:
> What is the expected day for v0.6?
>
>
>
>
> From: moon soo Lee <le...@gmail.com>
> To: users@zeppelin.apache.org
> Date: 2016/06/30 11:36 AM
> Subject: Re: spark interpreter
> ------------------------------
>
>
>
> Hi Ben,
>
> Livy interpreter is included in 0.6.0. If it is not listed when you create
> interpreter setting, could you check if your 'zeppelin.interpreters'
> property list Livy interpreter classes? (conf/zeppelin-site.xml)
>
> Thanks,
> moon
>
> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <*bbuild11@gmail.com*
> <bb...@gmail.com>> wrote:
> On a side note…
>
> Has anyone got the Livy interpreter to be added as an interpreter in the
> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on.
> Could this interfere?
>
> Thanks,
> Ben
>
>
> On Jun 29, 2016, at 11:18 AM, moon soo Lee <*moon@apache.org*
> <mo...@apache.org>> wrote:
>
> Livy interpreter internally creates multiple sessions for each user,
> independently from 3 binding modes supported in Zeppelin.
> Therefore, 'shared' mode, Livy interpreter will create sessions per each
> user, 'scoped' or 'isolated' mode will result create sessions per notebook,
> per user.
>
> Notebook is shared among users, they always use the same interpreter
> instance/process, for now. I think supporting per user interpreter
> instance/process would be future work.
>
> Thanks,
> moon
>
> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <*chen.song.82@gmail.com*
> <ch...@gmail.com>> wrote:
> Thanks for your explanation, Moon.
>
> Following up on this, I can see the difference in terms of single or
> multiple interpreter processes.
>
> With respect to spark drivers, since each interpreter spawns a separate
> Spark driver in regular Spark interpreter setting, it is clear to me the
> different implications of the 3 binding modes.
>
> However, when it comes to Livy server with impersonation turned on, I am a
> bit confused. Will Livy interpreter always create a new Spark driver (along
> with a Spark Context instance) for each user session, regardless of the
> binding mode of Livy interpreter? I am not very familiar with Livy, but
> from what I could tell, I see no difference between different binding modes
> for Livy on as far as how Spark drivers are concerned.
>
> Last question, when a notebook is shared among users, will they always use
> the same interpreter instance/process already created?
>
> Thanks
> Chen
>
>
>
> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <*moon@apache.org*
> <mo...@apache.org>> wrote:
> Hi,
>
> Thanks for asking question. It's not dumb question at all, Zeppelin docs
> does not explain very well.
>
> Spark Interpreter,
>
> 'shared' mode, a spark interpreter setting spawn a interpreter process to
> serve all notebooks which binded to this interpreter setting.
> 'scoped' mode, a spark interpreter setting spawn multiple interpreter
> processes per notebook which binded to this interpreter setting.
>
> Using Livy interpreter,
>
> Zeppelin propagate current user information to Livy interpreter. And Livy
> interpreter creates different session per user via Livy Server.
>
>
> Hope this helps.
>
> Thanks,
> moon
>
>
> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <*chen.song.82@gmail.com*
> <ch...@gmail.com>> wrote:
> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or
> `shared` Spark interpreter, every user share the same SparkContext. Sorry
> for the dumb question, how does it differ from Spark via Ivy Server?
>
>
> --
> Chen Song
>
>
>
>
>
Re: spark interpreter
Posted by Benjamin Kim <bb...@gmail.com>.
I see the download buttons.
Thanks,
Ben
On Saturday, July 2, 2016, moon soo Lee <mo...@apache.org> wrote:
> Thanks for testing it.
>
> When i run 0.6.0-rc1 bin-all binary, i can see CSV, TSV download buttons.
> Could you try clear browser cache?
>
> Regarding credential menu,
> Ahyoung is working on improvement and documentation on
> https://github.com/apache/zeppelin/pull/1100.
>
> Thanks,
> moon
>
> On Fri, Jul 1, 2016 at 11:34 AM Benjamin Kim <bbuild11@gmail.com
> <javascript:_e(%7B%7D,'cvml','bbuild11@gmail.com');>> wrote:
>
>> Moon,
>>
>> I have downloaded and tested the bin-all tarball, and it has some
>> deficiencies compared to the build-from-source version.
>>
>> - CSV, TSV download is missing
>> - Doesn’t work with HBase 1.2 in CDH 5.7.0
>> - Spark still does not work with Spark 1.6.0 in CDH 5.7.0 (JDK8)
>> - Using Livy is a good workaround
>> - Doesn’t work with Phoenix 4.7 in CDH 5.7.0
>>
>>
>> Everything else looks good especially in the area of multi-tenancy and
>> security. I would like to know how to use the Credentials feature on
>> securing usernames and passwords. I couldn’t find documentation on how.
>>
>> Thanks,
>> Ben
>>
>> On Jul 1, 2016, at 9:04 AM, moon soo Lee <moon@apache.org
>> <javascript:_e(%7B%7D,'cvml','moon@apache.org');>> wrote:
>>
>> 0.6.0 is currently in vote in dev@ list.
>>
>> http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html
>>
>> Thanks,
>> moon
>>
>> On Thu, Jun 30, 2016 at 1:54 PM Leon Katsnelson <leon@ca.ibm.com
>> <javascript:_e(%7B%7D,'cvml','leon@ca.ibm.com');>> wrote:
>>
>>> What is the expected day for v0.6?
>>>
>>>
>>>
>>>
>>> From: moon soo Lee <leemoonsoo@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','leemoonsoo@gmail.com');>>
>>> To: users@zeppelin.apache.org
>>> <javascript:_e(%7B%7D,'cvml','users@zeppelin.apache.org');>
>>> Date: 2016/06/30 11:36 AM
>>> Subject: Re: spark interpreter
>>> ------------------------------
>>>
>>>
>>>
>>> Hi Ben,
>>>
>>> Livy interpreter is included in 0.6.0. If it is not listed when you
>>> create interpreter setting, could you check if your 'zeppelin.interpreters'
>>> property list Livy interpreter classes? (conf/zeppelin-site.xml)
>>>
>>> Thanks,
>>> moon
>>>
>>> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <*bbuild11@gmail.com*
>>> <javascript:_e(%7B%7D,'cvml','bbuild11@gmail.com');>> wrote:
>>> On a side note…
>>>
>>> Has anyone got the Livy interpreter to be added as an interpreter in the
>>> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on.
>>> Could this interfere?
>>>
>>> Thanks,
>>> Ben
>>>
>>>
>>> On Jun 29, 2016, at 11:18 AM, moon soo Lee <*moon@apache.org*
>>> <javascript:_e(%7B%7D,'cvml','moon@apache.org');>> wrote:
>>>
>>> Livy interpreter internally creates multiple sessions for each user,
>>> independently from 3 binding modes supported in Zeppelin.
>>> Therefore, 'shared' mode, Livy interpreter will create sessions per each
>>> user, 'scoped' or 'isolated' mode will result create sessions per notebook,
>>> per user.
>>>
>>> Notebook is shared among users, they always use the same interpreter
>>> instance/process, for now. I think supporting per user interpreter
>>> instance/process would be future work.
>>>
>>> Thanks,
>>> moon
>>>
>>> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <*chen.song.82@gmail.com*
>>> <javascript:_e(%7B%7D,'cvml','chen.song.82@gmail.com');>> wrote:
>>> Thanks for your explanation, Moon.
>>>
>>> Following up on this, I can see the difference in terms of single or
>>> multiple interpreter processes.
>>>
>>> With respect to spark drivers, since each interpreter spawns a separate
>>> Spark driver in regular Spark interpreter setting, it is clear to me the
>>> different implications of the 3 binding modes.
>>>
>>> However, when it comes to Livy server with impersonation turned on, I am
>>> a bit confused. Will Livy interpreter always create a new Spark driver
>>> (along with a Spark Context instance) for each user session, regardless of
>>> the binding mode of Livy interpreter? I am not very familiar with Livy, but
>>> from what I could tell, I see no difference between different binding modes
>>> for Livy on as far as how Spark drivers are concerned.
>>>
>>> Last question, when a notebook is shared among users, will they always
>>> use the same interpreter instance/process already created?
>>>
>>> Thanks
>>> Chen
>>>
>>>
>>>
>>> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <*moon@apache.org*
>>> <javascript:_e(%7B%7D,'cvml','moon@apache.org');>> wrote:
>>> Hi,
>>>
>>> Thanks for asking question. It's not dumb question at all, Zeppelin docs
>>> does not explain very well.
>>>
>>> Spark Interpreter,
>>>
>>> 'shared' mode, a spark interpreter setting spawn a interpreter process
>>> to serve all notebooks which binded to this interpreter setting.
>>> 'scoped' mode, a spark interpreter setting spawn multiple interpreter
>>> processes per notebook which binded to this interpreter setting.
>>>
>>> Using Livy interpreter,
>>>
>>> Zeppelin propagate current user information to Livy interpreter. And
>>> Livy interpreter creates different session per user via Livy Server.
>>>
>>>
>>> Hope this helps.
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <*chen.song.82@gmail.com*
>>> <javascript:_e(%7B%7D,'cvml','chen.song.82@gmail.com');>> wrote:
>>> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or
>>> `shared` Spark interpreter, every user share the same SparkContext. Sorry
>>> for the dumb question, how does it differ from Spark via Ivy Server?
>>>
>>>
>>> --
>>> Chen Song
>>>
>>>
>>>
>>>
>>>
>>
Re: spark interpreter
Posted by moon soo Lee <mo...@apache.org>.
Thanks for testing it.
When i run 0.6.0-rc1 bin-all binary, i can see CSV, TSV download buttons.
Could you try clear browser cache?
Regarding credential menu,
Ahyoung is working on improvement and documentation on
https://github.com/apache/zeppelin/pull/1100.
Thanks,
moon
On Fri, Jul 1, 2016 at 11:34 AM Benjamin Kim <bb...@gmail.com> wrote:
> Moon,
>
> I have downloaded and tested the bin-all tarball, and it has some
> deficiencies compared to the build-from-source version.
>
> - CSV, TSV download is missing
> - Doesn’t work with HBase 1.2 in CDH 5.7.0
> - Spark still does not work with Spark 1.6.0 in CDH 5.7.0 (JDK8)
> - Using Livy is a good workaround
> - Doesn’t work with Phoenix 4.7 in CDH 5.7.0
>
>
> Everything else looks good especially in the area of multi-tenancy and
> security. I would like to know how to use the Credentials feature on
> securing usernames and passwords. I couldn’t find documentation on how.
>
> Thanks,
> Ben
>
> On Jul 1, 2016, at 9:04 AM, moon soo Lee <mo...@apache.org> wrote:
>
> 0.6.0 is currently in vote in dev@ list.
>
> http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html
>
> Thanks,
> moon
>
> On Thu, Jun 30, 2016 at 1:54 PM Leon Katsnelson <le...@ca.ibm.com> wrote:
>
>> What is the expected day for v0.6?
>>
>>
>>
>>
>> From: moon soo Lee <le...@gmail.com>
>> To: users@zeppelin.apache.org
>> Date: 2016/06/30 11:36 AM
>> Subject: Re: spark interpreter
>> ------------------------------
>>
>>
>>
>> Hi Ben,
>>
>> Livy interpreter is included in 0.6.0. If it is not listed when you
>> create interpreter setting, could you check if your 'zeppelin.interpreters'
>> property list Livy interpreter classes? (conf/zeppelin-site.xml)
>>
>> Thanks,
>> moon
>>
>> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <*bbuild11@gmail.com*
>> <bb...@gmail.com>> wrote:
>> On a side note…
>>
>> Has anyone got the Livy interpreter to be added as an interpreter in the
>> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on.
>> Could this interfere?
>>
>> Thanks,
>> Ben
>>
>>
>> On Jun 29, 2016, at 11:18 AM, moon soo Lee <*moon@apache.org*
>> <mo...@apache.org>> wrote:
>>
>> Livy interpreter internally creates multiple sessions for each user,
>> independently from 3 binding modes supported in Zeppelin.
>> Therefore, 'shared' mode, Livy interpreter will create sessions per each
>> user, 'scoped' or 'isolated' mode will result create sessions per notebook,
>> per user.
>>
>> Notebook is shared among users, they always use the same interpreter
>> instance/process, for now. I think supporting per user interpreter
>> instance/process would be future work.
>>
>> Thanks,
>> moon
>>
>> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <*chen.song.82@gmail.com*
>> <ch...@gmail.com>> wrote:
>> Thanks for your explanation, Moon.
>>
>> Following up on this, I can see the difference in terms of single or
>> multiple interpreter processes.
>>
>> With respect to spark drivers, since each interpreter spawns a separate
>> Spark driver in regular Spark interpreter setting, it is clear to me the
>> different implications of the 3 binding modes.
>>
>> However, when it comes to Livy server with impersonation turned on, I am
>> a bit confused. Will Livy interpreter always create a new Spark driver
>> (along with a Spark Context instance) for each user session, regardless of
>> the binding mode of Livy interpreter? I am not very familiar with Livy, but
>> from what I could tell, I see no difference between different binding modes
>> for Livy on as far as how Spark drivers are concerned.
>>
>> Last question, when a notebook is shared among users, will they always
>> use the same interpreter instance/process already created?
>>
>> Thanks
>> Chen
>>
>>
>>
>> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <*moon@apache.org*
>> <mo...@apache.org>> wrote:
>> Hi,
>>
>> Thanks for asking question. It's not dumb question at all, Zeppelin docs
>> does not explain very well.
>>
>> Spark Interpreter,
>>
>> 'shared' mode, a spark interpreter setting spawn a interpreter process to
>> serve all notebooks which binded to this interpreter setting.
>> 'scoped' mode, a spark interpreter setting spawn multiple interpreter
>> processes per notebook which binded to this interpreter setting.
>>
>> Using Livy interpreter,
>>
>> Zeppelin propagate current user information to Livy interpreter. And Livy
>> interpreter creates different session per user via Livy Server.
>>
>>
>> Hope this helps.
>>
>> Thanks,
>> moon
>>
>>
>> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <*chen.song.82@gmail.com*
>> <ch...@gmail.com>> wrote:
>> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or
>> `shared` Spark interpreter, every user share the same SparkContext. Sorry
>> for the dumb question, how does it differ from Spark via Ivy Server?
>>
>>
>> --
>> Chen Song
>>
>>
>>
>>
>>
>
Re: spark interpreter
Posted by Benjamin Kim <bb...@gmail.com>.
Moon,
I have downloaded and tested the bin-all tarball, and it has some deficiencies compared to the build-from-source version.
CSV, TSV download is missing
Doesn’t work with HBase 1.2 in CDH 5.7.0
Spark still does not work with Spark 1.6.0 in CDH 5.7.0 (JDK8)
Using Livy is a good workaround
Doesn’t work with Phoenix 4.7 in CDH 5.7.0
Everything else looks good especially in the area of multi-tenancy and security. I would like to know how to use the Credentials feature on securing usernames and passwords. I couldn’t find documentation on how.
Thanks,
Ben
> On Jul 1, 2016, at 9:04 AM, moon soo Lee <mo...@apache.org> wrote:
>
> 0.6.0 is currently in vote in dev@ list.
> http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html <http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html>
>
> Thanks,
> moon
>
> On Thu, Jun 30, 2016 at 1:54 PM Leon Katsnelson <leon@ca.ibm.com <ma...@ca.ibm.com>> wrote:
> What is the expected day for v0.6?
>
>
>
>
> From: moon soo Lee <leemoonsoo@gmail.com <ma...@gmail.com>>
> To: users@zeppelin.apache.org <ma...@zeppelin.apache.org>
> Date: 2016/06/30 11:36 AM
> Subject: Re: spark interpreter
>
>
>
> Hi Ben,
>
> Livy interpreter is included in 0.6.0. If it is not listed when you create interpreter setting, could you check if your 'zeppelin.interpreters' property list Livy interpreter classes? (conf/zeppelin-site.xml)
>
> Thanks,
> moon
>
> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <bbuild11@gmail.com <ma...@gmail.com>> wrote:
> On a side note…
>
> Has anyone got the Livy interpreter to be added as an interpreter in the latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on. Could this interfere?
>
> Thanks,
> Ben
>
>
> On Jun 29, 2016, at 11:18 AM, moon soo Lee <moon@apache.org <ma...@apache.org>> wrote:
>
> Livy interpreter internally creates multiple sessions for each user, independently from 3 binding modes supported in Zeppelin.
> Therefore, 'shared' mode, Livy interpreter will create sessions per each user, 'scoped' or 'isolated' mode will result create sessions per notebook, per user.
>
> Notebook is shared among users, they always use the same interpreter instance/process, for now. I think supporting per user interpreter instance/process would be future work.
>
> Thanks,
> moon
>
> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <chen.song.82@gmail.com <ma...@gmail.com>> wrote:
> Thanks for your explanation, Moon.
>
> Following up on this, I can see the difference in terms of single or multiple interpreter processes.
>
> With respect to spark drivers, since each interpreter spawns a separate Spark driver in regular Spark interpreter setting, it is clear to me the different implications of the 3 binding modes.
>
> However, when it comes to Livy server with impersonation turned on, I am a bit confused. Will Livy interpreter always create a new Spark driver (along with a Spark Context instance) for each user session, regardless of the binding mode of Livy interpreter? I am not very familiar with Livy, but from what I could tell, I see no difference between different binding modes for Livy on as far as how Spark drivers are concerned.
>
> Last question, when a notebook is shared among users, will they always use the same interpreter instance/process already created?
>
> Thanks
> Chen
>
>
>
> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <moon@apache.org <ma...@apache.org>> wrote:
> Hi,
>
> Thanks for asking question. It's not dumb question at all, Zeppelin docs does not explain very well.
>
> Spark Interpreter,
>
> 'shared' mode, a spark interpreter setting spawn a interpreter process to serve all notebooks which binded to this interpreter setting.
> 'scoped' mode, a spark interpreter setting spawn multiple interpreter processes per notebook which binded to this interpreter setting.
>
> Using Livy interpreter,
>
> Zeppelin propagate current user information to Livy interpreter. And Livy interpreter creates different session per user via Livy Server.
>
>
> Hope this helps.
>
> Thanks,
> moon
>
>
> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <chen.song.82@gmail.com <ma...@gmail.com>> wrote:
> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or `shared` Spark interpreter, every user share the same SparkContext. Sorry for the dumb question, how does it differ from Spark via Ivy Server?
>
>
> --
> Chen Song
>
>
>
>