You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by akhiluanandh <gi...@git.apache.org> on 2015/08/19 08:32:33 UTC

[GitHub] incubator-zeppelin pull request: Restart spark interpreter when th...

GitHub user akhiluanandh opened a pull request:

    https://github.com/apache/incubator-zeppelin/pull/228

    Restart spark interpreter when the spark context has shutdown

    When an interpreter is loaded from `NoteInterpreterLoader`, if this is a `SparkInterpreter` and the spark context has terminated (say, due to `sc.stop()` being called), this interpreter is useless. Right now, the user needs to go to the interpreter settings and restart this group before it can be used. Restarting the spark interpreter creates a new spark context.
    
    In this patch, we do this automatically. When we load a spark interpreter and the spark context has stopped, we automatically restart it. We also give the user an option whether or not to enable this automatic restart or not (default true).
    
    To check whether restart is required, we perform a thrift RPC call `restartRequired`. All interpreters except SparkInterpreter return false. SparkInterpreter returns `restartEnabled && sc != null && sc.taskScheduler() == null`, which is true if automatic restart is enabled and spark context has been stopped (stopping a spark context makes its taskScheduler null).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/akhiluanandh/incubator-zeppelin auto_restart_interpreter

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-zeppelin/pull/228.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #228
    
----
commit 7aa5c42c4d0dcb9128df87d7054e385d5d14ea05
Author: Akhilesh Anandh <ak...@qubole.com>
Date:   2015-08-19T06:14:32Z

    Restart spark interpreter when the spark context has shutdown

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-zeppelin pull request: Restart spark interpreter when th...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:

    https://github.com/apache/incubator-zeppelin/pull/228#issuecomment-133934888
  
    We could call `sc.stop()` or `SparkIMain.reset()`?
    Though apparently SparkIMain reset() has some issue: https://issues.apache.org/jira/browse/SPARK-10039


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-zeppelin pull request: Restart spark interpreter when th...

Posted by akhiluanandh <gi...@git.apache.org>.
Github user akhiluanandh commented on the pull request:

    https://github.com/apache/incubator-zeppelin/pull/228#issuecomment-134128644
  
    Thanks for the comments everyone.
    I agree that it would probably be better to restart only the SparkContext and not the interpreter. I will try to implement that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-zeppelin pull request: Restart spark interpreter when th...

Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:

    https://github.com/apache/incubator-zeppelin/pull/228#issuecomment-133769850
  
    @akhiluanandh Thanks for the contribution.
    
    Can SparkInterpreter handle restart SparkContext locally without adding a new method in Interpreter class? ie. restart SparkContext only, not whole interpreter process.
    
    Reasons i think restart only SparkContext is is better,
    
     * We can keep Interpreter class simple. One less thing to understand to implement new interpreter for Zeppelin.
     * When dependency loader loads some libraries, restarting SparkContext can reuse loaded libraries, but restarting interpreter process can not use these libraries, in current architecture. So user may have different libraries loaded in this environment after restart and it can be very confusing, especially when restarting is done silently.
    
    What do you think @akhiluanandh ?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-zeppelin pull request: Restart spark interpreter when th...

Posted by djoelz <gi...@git.apache.org>.
Github user djoelz commented on the pull request:

    https://github.com/apache/incubator-zeppelin/pull/228#issuecomment-132559083
  
    @akhiluanandh could you please add unit tests to your changes? Especially to remoteinterpreterservice. Thanks a lot!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---