You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by zj...@apache.org on 2017/05/18 14:34:12 UTC
zeppelin git commit: [ZEPPELIN-2564] - LivySparkSQLInterpreter throws
NullPointerException when getProgress is called
Repository: zeppelin
Updated Branches:
refs/heads/master 2f03469bc -> 05af3bf4b
[ZEPPELIN-2564] - LivySparkSQLInterpreter throws NullPointerException when getProgress is called
### What is this PR for?
When using %livy.sql interpreter following error is logged into interpreter log:
`ERROR [2017-05-17 15:01:32,897] (
{pool-1-thread-1}
TThreadPoolServer.java[run]:296) - Error occurred during processing of message.
java.lang.NullPointerException
at org.apache.zeppelin.livy.BaseLivyInterpreter.getProgress(BaseLivyInterpreter.java:178)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:121)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:558)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1899)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1884)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)`
and zeppelin server log:
`ERROR [2017-05-17 15:01:44,004] (
{JobProgressPoller, jobId=20170511-200435_1144682471}
JobProgressPoller.java[run]:58) - Can not get or update progress
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getProgress(RemoteInterpreter.java:468)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:121)
at org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:334)
at org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:55)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_getProgress(RemoteInterpreterService.java:321)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getProgress(RemoteInterpreterService.java:306)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getProgress(RemoteInterpreter.java:465)
... 3 more`
This happens because this.livyVersion is never assigned in LivySparkSQLInterpreter: https://github.com/apache/zeppelin/blob/master/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java#L63
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2564
### How should this be tested?
New test added to verify that livyVersion is not null
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: sergeymazin <se...@pipedrive.com>
Closes #2350 from sergeymazin/master and squashes the following commits:
3b95f5f [sergeymazin] Merge branch 'master' of https://github.com/sergeymazin/zeppelin
b506753 [sergeymazin] Check sparkInterpreter for null and move test
630520a [sergeymazin] Check sparkInterpreter for null and move test
c29c984 [sergeymazin] LivySparkSQLInterpreter to use getProgress of sparkInterpter
84db2d2 [sergeymazin] LivySparkSQLInterpreter throws NullPointerException when getProgress is called
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/05af3bf4
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/05af3bf4
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/05af3bf4
Branch: refs/heads/master
Commit: 05af3bf4b7f68c8434a8d1cb576fb6f4571551ab
Parents: 2f03469
Author: sergeymazin <se...@pipedrive.com>
Authored: Thu May 18 01:13:44 2017 +0300
Committer: Jeff Zhang <zj...@apache.org>
Committed: Thu May 18 10:34:08 2017 -0400
----------------------------------------------------------------------
.../org/apache/zeppelin/livy/LivySparkSQLInterpreter.java | 9 +++++++++
.../java/org/apache/zeppelin/livy/LivyInterpreterIT.java | 2 ++
2 files changed, 11 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/05af3bf4/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java
----------------------------------------------------------------------
diff --git a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java
index d132b5b..20d0448 100644
--- a/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java
+++ b/livy/src/main/java/org/apache/zeppelin/livy/LivySparkSQLInterpreter.java
@@ -240,6 +240,15 @@ public class LivySparkSQLInterpreter extends BaseLivyInterpreter {
}
@Override
+ public int getProgress(InterpreterContext context) {
+ if (this.sparkInterpreter != null) {
+ return this.sparkInterpreter.getProgress(context);
+ } else {
+ return 0;
+ }
+ }
+
+ @Override
protected String extractAppId() throws LivyException {
// it wont' be called because it would delegate to LivySparkInterpreter
throw new UnsupportedOperationException();
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/05af3bf4/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
----------------------------------------------------------------------
diff --git a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
index 6db75e9..80965ff 100644
--- a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
+++ b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
@@ -308,6 +308,8 @@ public class LivyInterpreterIT {
assertEquals(InterpreterResult.Code.SUCCESS, result.code());
assertEquals(InterpreterResult.Type.TABLE, result.message().get(0).getType());
assertTrue(result.message().get(0).getData().contains("tableName"));
+ int r = sqlInterpreter.getProgress(context);
+ assertTrue(r == 0);
} finally {
sqlInterpreter.close();
}