You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by del680202 <gi...@git.apache.org> on 2017/02/06 06:13:21 UTC
[GitHub] zeppelin pull request #1976: [ZEPPELIN-2037] Restart button does not work
GitHub user del680202 opened a pull request:
https://github.com/apache/zeppelin/pull/1976
[ZEPPELIN-2037] Restart button does not work
### What is this PR for?
When I enable authentication with ``shiro`` or switch interpreter mode to ``Per user & isolated``, ``Restart`` does not work and it is possible to make zombie process.
* Environment:
OS: CentOS 6
Zeppelin: master, branch-0.7
* Test flow:
1. Execute simple code such as ``sc.version``
2. Click "Restart" button
3. Execute simple code again
4. Shutdown zeppelin
5. Monitor java processes
There are two issues in ``Restart`` button.
* ``closeAndRemoveInterpreterGroupByUser`` does not work in some situations
When I enable authentication, then ``closeAndRemoveInterpreterGroupByUser`` will be called by clicking "Restart" button. The implement of ``closeAndRemoveInterpreterGroupByUser`` call ``dereference`` only once, if ``remoteInterpreterProcess.referenceCount`` number larger than ``1``, then ``Restart`` button does not work.
* ``closeAndRemoveInterpreterGroupByNoteId`` cloud not get correct processKey
When I switch interpreter mode to ``Per user & isolated``, closeAndRemoveInterpreterGroupByNoteId will be called by clicking "Restart" button.
And I will get ``intpKey=$username:`` and ``processKey=:``, therefore, ``isEqualInterpreterKeyProcessKey`` be false forever.
I fix this issue by simple ways, but I only test ``spark``, ``python``, ``livy`` interpreters, maybe other interpreters have the same problem.
### What type of PR is it?
Bug Fix
### Todos
* [ ] - Add unit test
### What is the Jira issue?
[ZEPPELIN-2037](https://issues.apache.org/jira/browse/ZEPPELIN-2037)
### How should this be tested?
1. Execute ``%spark sc.version`` on zeppelin
2. Click ``Restart`` button which in ``Notebook`` page or ``Interpreter`` Page
3. Monitor java processes by ``jps``
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/del680202/zeppelin restart-bug-2037
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1976.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1976
----
commit 94f42f900d0afd447b07cce3ba474779bf566375
Author: Chin Tzulin <jp...@w022341412910m.local>
Date: 2017-02-06T05:31:22Z
[ZEPPELIN-2037] Restart button does not work
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by AhyoungRyu <gi...@git.apache.org>.
Github user AhyoungRyu commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@del680202
>When I enable authentication with shiro or switch interpreter mode to Per user & isolated, Restart does not work and it is possible to make zombie process.
This issue has been solved by #2149. And the issue status was updated to `RESOLVED` in https://issues.apache.org/jira/browse/ZEPPELIN-2037. Can you check again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@AhyoungRyu
I test #2149 in my machine.
It works well, Thanks.
But I seem when I edit&save interpreter config then the zombie process will be occurred.
Anyway, ``Restart`` button looks well, I close this PR, thanks.
## Test cases:
### Global mode
Restart by button in notebook page: SUCCESS
Restart by button in interpreter page: SUCCESS
### Per user & isolated mode
Restart by button in notebook page: SUCCESS
Restart by button in interpreter page: SUCCESS
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/1976
I'm testing it now and found something weird.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@jongyoul sure :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@cloverhearts Can you check it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@del680202 I can not reproduce the issue on 0.7.1 rc2, regarding your instruction, when you do step 2, zeppelin would ask you to restart the interpreter, did you click the button to restart it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@del680202 I've made a PR for handling what you found. Can you please test it with that PR? #2191
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@AhyoungRyu Thanks for your reply.
>>But I seem when I edit&save interpreter config then the zombie process will be occurred
Sorry, I miss this test case.
I test it by current ``branch-0.7`` too.
My test steps as below:
1. Execute sample code with ``%python`` in ``global`` mode.
2. Switch python interpreter to ``per user`` mode
3. Restart/Shutdown Zeppelin
It looks like a issue of switching interpreter mode function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1976
Is it caused by ZEPPELIN-1921 ? It works in zeppelin 0.7 but not in master.
BTW, I think these methods are a little confusing, we might need to improve them later. @jongyoul is working on refactor the interpreter component, we might also need to consider to clean up the APIs that those components should expose. (InterpreterFactory, InterpreterSetting, InterpreterGroup)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@cloverhearts Thanks for your review.
I seem ``closeAndRemoveInterpreterGroupByUser`` has been applied by https://issues.apache.org/jira/browse/ZEPPELIN-1770
I think ``closeAndRemoveInterpreterGroupByUser`` is most important feature on the multiple user environment, my users need to restart interpreter by themselves.
But the implementation of ``closeAndRemoveInterpreterGroupByUser`` cloud not do it now.
Did I miss something about this issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@jongyoul Thanks, I will test it later
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@AhyoungRyu Thanks, I will check it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1976: [ZEPPELIN-2037] Restart button does not work
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 closed the pull request at:
https://github.com/apache/zeppelin/pull/1976
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by AhyoungRyu <gi...@git.apache.org>.
Github user AhyoungRyu commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@del680202 Also tested on both latest branch-0.7 and rc2 but couldn't reproduce. Here is my steps.
1. Activate shiro -> login
2. Run a paragraph with python intp(globally shared) -> 1 RemoteInterpreterServer shows up
3. Edit python interpreter as "Per User / isolated" -> Click save btn (in this step the interpreter will be restarted automatically) -> the previous RemoteInterpreterServer was killed
I checked `jps` in every steps, but couldn't see zombie remote interpreter process. Can you check again you were testing in latest branch-0.7? Or you can simply check it with 0.7.1-rc2 binary[1] as @zjffdu did.
[1] https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.7.1-rc2/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@zjffdu @AhyoungRyu
Yes, I tested it by latest branch-0.7.
Latest commit
```
commit 53e90578621574f466a58ad02a6577bca6335a5a
Author: Guillermo Cabrera <gu...@gmail.com>
Date: Mon Mar 20 16:00:52 2017 +0300
```
Oh..Sorry, I suppose I miss some steps.
I tested this problem with ``%python`` and ``%pyspark`` at the same time
1. Enable shiro for login
2. Execute sample code by ``%python`` and ``%pyspark``
.
3. Switch ``spark`` interpreter from ``global`` mode to ``Per user`` mode and save it.
4. Switch ``python`` interpreter from ``Per user`` mode to ``global`` mode save it.
5. Shutdown zeppelin
It was success when I change ``spark`` interpreter setting, but it make zombie process when I change ``python`` interpreter setting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@del680202 Thanks, if you finish to test it, we will make a new rc including it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by cloverhearts <gi...@git.apache.org>.
Github user cloverhearts commented on the issue:
https://github.com/apache/zeppelin/pull/1976
Thank you for your reply, @del680202
I understand your publicity purpose.
And I think it's right.
What I am worried call "terminateRemoteProcess ();" From "public void close (String sessionId) in InterpreterGroup.java".
"Close (String sessionId)" should only perform the action corresponding to the session.
And if all sessions are closed, the process must be terminated.
However, the current code always performs "terminateRemoteProcess ();" Within "close (String sessionId)".
"TerminateRemoteProcess ();" The function forces all sessions to be terminated and the process is terminated.
I think this is the wrong behavior of "close (String sessionId)".
We think this can be a problem, because we can write several users with an interpreter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by del680202 <gi...@git.apache.org>.
Github user del680202 commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@cloverhearts Thanks for your advice
I will try to find the other way to solve this problem
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1976: [ZEPPELIN-2037] Restart button does not work
Posted by AhyoungRyu <gi...@git.apache.org>.
Github user AhyoungRyu commented on the issue:
https://github.com/apache/zeppelin/pull/1976
@del680202 Thanks for your time and effort to test it out.
>But I seem when I edit&save interpreter config then the zombie process will be occurred
I checked it with current `branch-0.7` and I could see the interpreter process was killed.
Actually just tested in basic case (anonymous / global mode) with Spark interpreter.
1. Run some paragraphs \w Spark intp
2. Edit `zeppelin.spark.maxResult` from 1000 to 20 in the interpreter setting page -> save
3. The remote interpreter process was killed
Was your case different with mine?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---