You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Chen Song <ch...@gmail.com> on 2016/08/04 19:57:14 UTC

question on livy session in zeppelin

Hi

When using Zeppelin over Livy, it appears that Livy will delete the session
(and terminate the Spark job) after one hour of inactivity. After that, the
SparkContext is closed and the user will see an error like 404 Not Found in
Zeppelin notebook.

From this point, is the only way to proceed is to manually restart the
interpreter? This is not very user-friendly. Is there a way for interpreter
to reconnect itself to a new Livy session automatically?

Chen

Re: question on livy session in zeppelin

Posted by moon soo Lee <mo...@apache.org>.
Hi Chen,

1. Currently, Interpreter binding mode is more like whether each Note will
have a separate Interpreter session(scoped/isolated) or not(shared), rather
than per user.
https://issues.apache.org/jira/browse/ZEPPELIN-1210 will bring interpreter
session per user in the Zeppelin server side, but for now, Livy interpreter
itself manages session per user.

2. Interpreter Authorization is working on
https://issues.apache.org/jira/browse/ZEPPELIN-945.
Proposed patch does not distinguish edit/use/restart permission, but we can
think about it based on initial work.

3. For now, create multiple interpreter settings with different memory
configuration is only approach to do that, i think.

Thanks,
moon

On Fri, Aug 5, 2016 at 8:28 AM Chen Song <ch...@gmail.com> wrote:

> Few more follow-up questions to see if anyone has insights on them
>
> 1. What is the recommended way of interpreter binding mode setting
> Zeppelin over Livy in a multi-user environment?
> 2. Is there a way to allow the user restart interpreter but restrict the
> user from changing interpreter configs.
> 3. Is there a way to support different custom settings for the same
> interpreter. For example, one user may want more memory for his notebook
> than other users.
>
> Chen
>
> On Thu, Aug 4, 2016 at 10:58 PM Chen Song <ch...@gmail.com> wrote:
>
>> Thanks Vinay.
>>
>> Following up on this, in a multi-user environment, Livy will create a
>> session (and a separate Spark job) for each user. If one user restarts the
>> Livy interpreter, will that cause other users' sessions along with their
>> Spark job to be killed?
>>
>> If so, can I work around this by setting binding mode for the interpreter
>> to always be private?
>>
>>
>> On Thu, Aug 4, 2016 at 5:15 PM Vinay Shukla <vi...@gmail.com>
>> wrote:
>>
>>> Hi Chen,
>>>
>>> I agree this is not very user friendly & there should be no need to
>>> restart LivyInterpreter.
>>> One short term fix may be to increase the livy session timeout by
>>> changing livy.server.session.timeout (takes time in millisecond)
>>>
>>> Filed https://issues.apache.org/jira/browse/ZEPPELIN-1293 to track it.
>>>
>>> Thanks,
>>> Vinay
>>>
>>>
>>> On Thu, Aug 4, 2016 at 12:57 PM, Chen Song <ch...@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> When using Zeppelin over Livy, it appears that Livy will delete the
>>>> session (and terminate the Spark job) after one hour of inactivity. After
>>>> that, the SparkContext is closed and the user will see an error like 404
>>>> Not Found in Zeppelin notebook.
>>>>
>>>> From this point, is the only way to proceed is to manually restart the
>>>> interpreter? This is not very user-friendly. Is there a way for interpreter
>>>> to reconnect itself to a new Livy session automatically?
>>>>
>>>> Chen
>>>>
>>>
>>>

Re: question on livy session in zeppelin

Posted by Chen Song <ch...@gmail.com>.
Few more follow-up questions to see if anyone has insights on them

1. What is the recommended way of interpreter binding mode setting Zeppelin
over Livy in a multi-user environment?
2. Is there a way to allow the user restart interpreter but restrict the
user from changing interpreter configs.
3. Is there a way to support different custom settings for the same
interpreter. For example, one user may want more memory for his notebook
than other users.

Chen

On Thu, Aug 4, 2016 at 10:58 PM Chen Song <ch...@gmail.com> wrote:

> Thanks Vinay.
>
> Following up on this, in a multi-user environment, Livy will create a
> session (and a separate Spark job) for each user. If one user restarts the
> Livy interpreter, will that cause other users' sessions along with their
> Spark job to be killed?
>
> If so, can I work around this by setting binding mode for the interpreter
> to always be private?
>
>
> On Thu, Aug 4, 2016 at 5:15 PM Vinay Shukla <vi...@gmail.com> wrote:
>
>> Hi Chen,
>>
>> I agree this is not very user friendly & there should be no need to
>> restart LivyInterpreter.
>> One short term fix may be to increase the livy session timeout by
>> changing livy.server.session.timeout (takes time in millisecond)
>>
>> Filed https://issues.apache.org/jira/browse/ZEPPELIN-1293 to track it.
>>
>> Thanks,
>> Vinay
>>
>>
>> On Thu, Aug 4, 2016 at 12:57 PM, Chen Song <ch...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> When using Zeppelin over Livy, it appears that Livy will delete the
>>> session (and terminate the Spark job) after one hour of inactivity. After
>>> that, the SparkContext is closed and the user will see an error like 404
>>> Not Found in Zeppelin notebook.
>>>
>>> From this point, is the only way to proceed is to manually restart the
>>> interpreter? This is not very user-friendly. Is there a way for interpreter
>>> to reconnect itself to a new Livy session automatically?
>>>
>>> Chen
>>>
>>
>>

Re: question on livy session in zeppelin

Posted by Chen Song <ch...@gmail.com>.
Thanks Vinay.

Following up on this, in a multi-user environment, Livy will create a
session (and a separate Spark job) for each user. If one user restarts the
Livy interpreter, will that cause other users' sessions along with their
Spark job to be killed?

If so, can I work around this by setting binding mode for the interpreter
to always be private?


On Thu, Aug 4, 2016 at 5:15 PM Vinay Shukla <vi...@gmail.com> wrote:

> Hi Chen,
>
> I agree this is not very user friendly & there should be no need to
> restart LivyInterpreter.
> One short term fix may be to increase the livy session timeout by changing
> livy.server.session.timeout (takes time in millisecond)
>
> Filed https://issues.apache.org/jira/browse/ZEPPELIN-1293 to track it.
>
> Thanks,
> Vinay
>
>
> On Thu, Aug 4, 2016 at 12:57 PM, Chen Song <ch...@gmail.com> wrote:
>
>> Hi
>>
>> When using Zeppelin over Livy, it appears that Livy will delete the
>> session (and terminate the Spark job) after one hour of inactivity. After
>> that, the SparkContext is closed and the user will see an error like 404
>> Not Found in Zeppelin notebook.
>>
>> From this point, is the only way to proceed is to manually restart the
>> interpreter? This is not very user-friendly. Is there a way for interpreter
>> to reconnect itself to a new Livy session automatically?
>>
>> Chen
>>
>
>

Re: question on livy session in zeppelin

Posted by Vinay Shukla <vi...@gmail.com>.
Hi Chen,

I agree this is not very user friendly & there should be no need to restart
LivyInterpreter.
One short term fix may be to increase the livy session timeout by changing
livy.server.session.timeout (takes time in millisecond)

Filed https://issues.apache.org/jira/browse/ZEPPELIN-1293 to track it.

Thanks,
Vinay


On Thu, Aug 4, 2016 at 12:57 PM, Chen Song <ch...@gmail.com> wrote:

> Hi
>
> When using Zeppelin over Livy, it appears that Livy will delete the
> session (and terminate the Spark job) after one hour of inactivity. After
> that, the SparkContext is closed and the user will see an error like 404
> Not Found in Zeppelin notebook.
>
> From this point, is the only way to proceed is to manually restart the
> interpreter? This is not very user-friendly. Is there a way for interpreter
> to reconnect itself to a new Livy session automatically?
>
> Chen
>