You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by spektom <gi...@git.apache.org> on 2016/09/21 17:50:20 UTC
[GitHub] zeppelin pull request #1447: [ZEPPELIN-1293] Re-create Livy session if it's ...
GitHub user spektom opened a pull request:
https://github.com/apache/zeppelin/pull/1447
[ZEPPELIN-1293] Re-create Livy session if it's lost
### What is this PR for?
This patch restarts Livy session when it's lost.
### What type of PR is it?
Improvement/Bug Fix
### Todos
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1293
### How should this be tested?
* Configure `livy.server.session.timeout` parameter in Livy
* Start new Livy session through Zeppelin
* Wait for Livy session to expire
* See that new Livy session is started when running something in Zeppelin
### Questions:
* Does the licenses files need update? - No
* Is there breaking changes for older versions? - No
* Does this needs documentation? - No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/spektom/zeppelin ZEPPELIN-1293
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1447.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1447
----
commit acdecbe3b1a91f630d17f4c25c6657cb3fc8d751
Author: Michael Spector <sp...@gmail.com>
Date: 2016-09-21T17:36:18Z
[ZEPPELIN-1293] Re-create Livy session if it's lost
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1447
Sorry for late response. @spektom Could you add some additional message of session recreation to be displayed in frontend. Because if session is recreated, we may need to rerun all the paragraphs, without knowing session recreationusers may be confused for errors like some variables are not defined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by gss2002 <gi...@git.apache.org>.
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom just tested against my build by catching the exception and rethrowing.. It definitely solves the issue.
} catch (LivyNoSessionException e) {
throw e;
} catch (Exception e) {
LOGGER.error("error in interpretInput", e);
return new InterpreterResult(Code.ERROR, e.getMessage());
}
}
public InterpreterResult interpret(String stringLines,
The code base I'm using is here with you patch and a few of @zjffdu patches and one of mine for NestedRuntimeException for 404's with KerberosTemplate: https://github.com/gss2002/zeppelin/blob/GSS_PROD_BUILD/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by spektom <gi...@git.apache.org>.
Github user spektom commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@gss2002 are you using latest code from the branch? I can't find org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189) invocation at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106) as it's shown in your stack trace.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by gss2002 <gi...@git.apache.org>.
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom and @zjffdu is it possible to add a null check also. I've been doing some debugging over the past few days and certain situations can cause nulls to be returned and in theory if a null is being returned the session is dead.
**So instead of this:**
if (json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {
throw new LivyNoSessionException();
**This:**
boolean clearSession = false;
if (json != null) {
if (json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {
clearSession = true;
}
} else {
clearSession = true;
}
if (clearSession) {
throw new LivyNoSessionException();
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by gss2002 <gi...@git.apache.org>.
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom this fix is good. Did some extensive load testing with it this AM and it solves the session expiration issues. Thanks for the contribution
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom Thanks for the contribution, I am working on a PR about integration test of livy interpreter, could you mind to wait for that PR and then add integration test to that ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by spektom <gi...@git.apache.org>.
Github user spektom commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@gss2002 you shouldn't be concerned about performance as this code only runs when command is executed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by gss2002 <gi...@git.apache.org>.
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
Also I have some concerns on using Exceptions to as goto's as there is definite performance issues that occur by using exceptions... http://javarevisited.blogspot.com/2013/03/0-exception-handling-best-practices-in-Java-Programming.html. Specifically since code inside of the try catch cannot be optimized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@purechoc Do you want to rebase it and continue work on it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by spektom <gi...@git.apache.org>.
Github user spektom commented on the issue:
https://github.com/apache/zeppelin/pull/1447
Of course, it would be great.
On Fri, Sep 23, 2016 at 06:28 Jeff Zhang <no...@github.com> wrote:
> @spektom <https://github.com/spektom> Thanks for the contribution, I am
> working on a PR about integration test of livy interpreter, could you mind
> to wait for that PR and then add integration test to that ?
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/zeppelin/pull/1447#issuecomment-249093645>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAJpO0VBwVJ9NayJbPq3h9j1bRji_4cuks5qs0dkgaJpZM4KDF0H>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by purechoc <gi...@git.apache.org>.
Github user purechoc commented on the issue:
https://github.com/apache/zeppelin/pull/1447
LGTM. it's working nicely in my environment. (zeppelin 0.7 + livy 2.1)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom Sorry, I didn't get your reply, I create another PR for it #1861
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by gss2002 <gi...@git.apache.org>.
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom I think what happens here is this code fires.. which has nothing to do with the fix here..
in LivySparkInterpreter:
return livyHelper.interpretInput(line, interpreterContext, userSessionMap, out,
sessionId2AppIdMap.get(sessionId), sessionId2WebUIMap.get(sessionId), displayAppInfo);
That gets called before the NoSessionException occurs.. And then in
LivyHelper --> public InterpreterResult interpretInput grabs the exception and handles it. I guess the question is can we do a rootcause on this and rethrow?
} catch (Exception e) {
LOGGER.error("error in interpretInput", e);
return new InterpreterResult(Code.ERROR, e.getMessage());
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1447: [ZEPPELIN-1293] Re-create Livy session if it's lost
Posted by gss2002 <gi...@git.apache.org>.
Github user gss2002 commented on the issue:
https://github.com/apache/zeppelin/pull/1447
@spektom / @zjffdu @purechoc there is definitely an additional condition. Not sure if it's because the ConcurrentHashMaps are not being used correctly. But the exception doesn't get caught completely or correctly at times with the fix proposed here..
ERROR [2016-10-19 14:19:57,638] ({pool-2-thread-11} LivyHelper.java[executeHTTP]:378) - Error with 404 StatusCode: "Session '9' not found."
ERROR [2016-10-19 14:19:57,638] ({pool-2-thread-11} LivyHelper.java[interpretInput]:229) - error in interpretInput
org.apache.zeppelin.livy.LivyHelper$LivyNoSessionException: Session not found, Livy server would have restarted, or lost session.
at org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:312)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:241)
at org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189)
at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [2016-10-19 14:19:57,639] ({pool-2-thread-11} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1476901197622 finished by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
INFO [2016-10-19 14:19:57,819] ({pool-2-thread-34} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1476901197819 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
ERROR [2016-10-19 14:19:57,835] ({pool-2-thread-34} LivyHelper.java[executeHTTP]:378) - Error with 404 StatusCode: "Session '9' not found."
ERROR [2016-10-19 14:19:57,835] ({pool-2-thread-34} LivyHelper.java[interpretInput]:229) - error in interpretInput
org.apache.zeppelin.livy.LivyHelper$LivyNoSessionException: Session not found, Livy server would have restarted, or lost session.
at org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:312)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:241)
at org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189)
at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [2016-10-19 14:19:57,836] ({pool-2-thread-34} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1476901197819 finished by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1447: [ZEPPELIN-1293] Re-create Livy session if it's ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1447
---