You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2017/10/25 07:47:34 UTC
[GitHub] zeppelin pull request #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/2631
ZEPPELIN-2197. Interpreter Lifecycle Manager
### What is this PR for?
This PR implement the lifecycle manager. There're 2 implementions.
* NullLifecycleManager. Nothing is done as before. User has to start/stop interpreter explicitly in UI.
* TimeoutLifecycleManager. Interpreter will be closed after idle for one threshold of time. By default it is 1 hour.
### What type of PR is it?
[ Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2197
### How should this be tested?
Unit test is added.
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2197
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2631.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2631
----
commit cd112f9abb8538e5684fbf6ddb61013137150389
Author: Jeff Zhang <zj...@apache.org>
Date: 2017-09-28T05:02:23Z
save
commit 6376d7617617daf92f67168c1fc767054e5616b6
Author: Jeff Zhang <zj...@apache.org>
Date: 2017-10-25T06:18:31Z
ZEPPELIN-2197. Interpreter Lifecycle Manager
----
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by Gauravshah <gi...@git.apache.org>.
Github user Gauravshah commented on the issue:
https://github.com/apache/zeppelin/pull/2631
@zjffdu what if I am not on my desk and not polling the job ?
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2631
@Tagar Good point, Currently there's no such things in frontend. For now, what user see in frontend is that a new interpreter group is created. (For spark interpreter, a new spark app is started)
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2631
@Tagar It won't be killed, because LifecycleManager will know client is polling job progress via api Interpreter.getProgress. https://github.com/apache/zeppelin/pull/2631/files#diff-f55969c5887615d2142806214747bba9R330
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2631
One last thing - from user experience it would be convenient to know when their interpreters timed out.
Something like a popup or just some sort of a graphical flag would do, I guess?
Not sure how hard it'll be add this at this point.
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2631
Will merge it if no more comments
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2631
@zjffdu got it - thank you.
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2631
@jongyoul For now, only Interpreter Process's lifecycle is controlled in TimeoutLifecycleManager. session level control will be done in future if necessary. https://github.com/apache/zeppelin/pull/2631/files#diff-53b72da47096c21dca5f395fe86761b3R16
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2631
could we add some documentation on this? this sounds important...
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2631
Make sense, will add document for this feature.
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2631
@Leemoonsoo @jongyoul @Tagar Please help review. Thanks
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/2631
I have a basic question. Does it work in `scoped` and `isolated`?
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2631
@Gauravshah It won't be killed. Because JobRunner in zeppelin server process will poll the job status periodically . Add one more test to verify it.
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/2631
Got it. Thanks. I will be able to help you for the future feature.
---
[GitHub] zeppelin issue #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by Tagar <gi...@git.apache.org>.
Github user Tagar commented on the issue:
https://github.com/apache/zeppelin/pull/2631
Thank you @zjffdu.
I just thought about this scenario: a Spark job runs for 1.5 hours, would it be killed by the LifeCycleManager in this case? (assuming here default timeout of 1 hour)
If it is, then might be nice also to have a grace period when an interpreter wouldn't be killed if it has a running job?
In the above example, let's say timeout=1 hour and grace period=1 hour. So an interpreter would be killed if it is completely inactive for 1 hour, or in 2 hours if it had a Spark job that was still spinning.
Thoughts?
---
[GitHub] zeppelin pull request #2631: ZEPPELIN-2197. Interpreter Lifecycle Manager
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/2631
---