You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2020/02/15 06:38:44 UTC
[GitHub] [zeppelin] tylerba-f opened a new pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to r…
tylerba-f opened a new pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to r…
URL: https://github.com/apache/zeppelin/pull/3650
### What is this PR for?
Fix the stability problem of zengin for long time (7day) loop working ,
release the thread pools and socket connection when the interprether is closed
the same bug was found in the version 0.81 and 0.82
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-4600
### How should this be tested?
* Single tast works 30 hours with 7 second delay, test passed.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?No
* Is there breaking changes for older versions?No
* Does this needs documentation?No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3650: [ZEPPELIN-4600] Socket
connection between zeppelin server and interpreter is not closed properly
for branch 0.8
Posted by GitBox <gi...@apache.org>.
zjffdu commented on issue #3650: [ZEPPELIN-4600] Socket connection between zeppelin server and interpreter is not closed properly for branch 0.8
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-589453722
@tylerba-f Could you close this PR ? It has been merged already
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3650: [ZEPPELIN-4600] Socket
connection between zeppelin server and interpreter is not closed properly
for branch 0.8
Posted by GitBox <gi...@apache.org>.
zjffdu commented on issue #3650: [ZEPPELIN-4600] Socket connection between zeppelin server and interpreter is not closed properly for branch 0.8
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-587234123
Thanks, will merge it if no more comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379776174
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -51,13 +54,27 @@ public void setRemoteInterpreterEventPoller(RemoteInterpreterEventPoller eventPo
this.remoteInterpreterEventPoller = eventPoller;
}
+ public void shutdown()
+ {
+
+ // Close client socket connection
+ if (clientFactory != null)
+ {
+ for (TSocket eachTransfer: clientFactory.clientSocketMap.values())
+ {
+ eachTransfer.close();
+ }
+ }
+ }
+
public int getConnectTimeout() {
return connectTimeout;
}
public synchronized Client getClient() throws Exception {
if (clientPool == null || clientPool.isClosed()) {
- clientPool = new GenericObjectPool<>(new ClientFactory(getHost(), getPort()));
+ clientFactory = new ClientFactory(getHost(), getPort());
+ clientPool = new GenericObjectPool<>(clientFactory);
Review comment:
This change is not necessary ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379776280
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -19,12 +19,14 @@
import com.google.gson.Gson;
import org.apache.commons.pool2.impl.GenericObjectPool;
import org.apache.thrift.TException;
+import org.apache.thrift.transport.TSocket;
import org.apache.zeppelin.helium.ApplicationEventListener;
import org.apache.zeppelin.interpreter.InterpreterException;
import org.apache.zeppelin.interpreter.launcher.InterpreterClient;
import org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService.Client;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import sun.font.TextSource;
Review comment:
Unused import
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3650: [ZEPPELIN-4600]
nullpoint error in the long time process use api to run a note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on issue #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-586584411
Thanks @tylerba-f for quick fix, have you verified this PR and how ?
BTW, it think its better to rename the JIRA ticket to be 'Socket connection between zeppelin server and interpreter is not closed properly'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] tylerba-f closed pull request #3650: [ZEPPELIN-4600]
Socket connection between zeppelin server and interpreter is not closed
properly for branch 0.8
Posted by GitBox <gi...@apache.org>.
tylerba-f closed pull request #3650: [ZEPPELIN-4600] Socket connection between zeppelin server and interpreter is not closed properly for branch 0.8
URL: https://github.com/apache/zeppelin/pull/3650
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379776098
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -51,13 +54,27 @@ public void setRemoteInterpreterEventPoller(RemoteInterpreterEventPoller eventPo
this.remoteInterpreterEventPoller = eventPoller;
}
+ public void shutdown()
+ {
+
+ // Close client socket connection
+ if (clientFactory != null)
+ {
+ for (TSocket eachTransfer: clientFactory.clientSocketMap.values())
Review comment:
It's better to add `close` method in ClientFactory, can call it here like this
```
if (clientFactory != null) {
clientFactory.close();
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] tylerba-f commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
tylerba-f commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379794457
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -19,12 +19,14 @@
import com.google.gson.Gson;
import org.apache.commons.pool2.impl.GenericObjectPool;
import org.apache.thrift.TException;
+import org.apache.thrift.transport.TSocket;
import org.apache.zeppelin.helium.ApplicationEventListener;
import org.apache.zeppelin.interpreter.InterpreterException;
import org.apache.zeppelin.interpreter.launcher.InterpreterClient;
import org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService.Client;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import sun.font.TextSource;
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] tylerba-f commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
tylerba-f commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379794355
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -51,13 +54,27 @@ public void setRemoteInterpreterEventPoller(RemoteInterpreterEventPoller eventPo
this.remoteInterpreterEventPoller = eventPoller;
}
+ public void shutdown()
+ {
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3650: [ZEPPELIN-4600]
nullpoint error in the long time process use api to run a note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on issue #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-586584405
Thanks @tylerba-f for quick fix, have you verified this PR and how ?
BTW, it think its better to rename the JIRA ticket to be 'Socket connection between zeppelin server and interpreter is not closed properly'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3650: [ZEPPELIN-4600]
nullpoint error in the long time process use api to run a note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on issue #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-586562208
Thanks @tylerba-f Overall PR LGTM, I leave some minor issue comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3650: [ZEPPELIN-4600]
nullpoint error in the long time process use api to run a note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on issue #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-586584439
And could you create another PR for master branch ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3650: [ZEPPELIN-4600]
nullpoint error in the long time process use api to run a note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on issue #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-586584398
Thanks @tylerba-f for quick fix, have you verified this PR and how ?
BTW, it think its better to rename the JIRA ticket to be 'Socket connection between zeppelin server and interpreter is not closed properly'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] tylerba-f commented on issue #3650: [ZEPPELIN-4600]
Socket connection between zeppelin server and interpreter is not closed
properly for branch 0.8
Posted by GitBox <gi...@apache.org>.
tylerba-f commented on issue #3650: [ZEPPELIN-4600] Socket connection between zeppelin server and interpreter is not closed properly for branch 0.8
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-590016458
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379775500
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -51,13 +54,27 @@ public void setRemoteInterpreterEventPoller(RemoteInterpreterEventPoller eventPo
this.remoteInterpreterEventPoller = eventPoller;
}
+ public void shutdown()
+ {
Review comment:
code style issue. please use this style
```
public void shutdown() {
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] tylerba-f commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
tylerba-f commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379794402
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -51,13 +54,27 @@ public void setRemoteInterpreterEventPoller(RemoteInterpreterEventPoller eventPo
this.remoteInterpreterEventPoller = eventPoller;
}
+ public void shutdown()
+ {
+
+ // Close client socket connection
+ if (clientFactory != null)
+ {
+ for (TSocket eachTransfer: clientFactory.clientSocketMap.values())
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379793327
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -51,13 +54,27 @@ public void setRemoteInterpreterEventPoller(RemoteInterpreterEventPoller eventPo
this.remoteInterpreterEventPoller = eventPoller;
}
+ public void shutdown()
+ {
+
+ // Close client socket connection
+ if (clientFactory != null)
+ {
+ for (TSocket eachTransfer: clientFactory.clientSocketMap.values())
+ {
+ eachTransfer.close();
+ }
+ }
+ }
+
public int getConnectTimeout() {
return connectTimeout;
}
public synchronized Client getClient() throws Exception {
if (clientPool == null || clientPool.isClosed()) {
- clientPool = new GenericObjectPool<>(new ClientFactory(getHost(), getPort()));
+ clientFactory = new ClientFactory(getHost(), getPort());
+ clientPool = new GenericObjectPool<>(clientFactory);
Review comment:
I see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] tylerba-f commented on a change in pull request #3650:
[ZEPPELIN-4600] nullpoint error in the long time process use api to run a
note
Posted by GitBox <gi...@apache.org>.
tylerba-f commented on a change in pull request #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#discussion_r379791013
##########
File path: zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterProcess.java
##########
@@ -51,13 +54,27 @@ public void setRemoteInterpreterEventPoller(RemoteInterpreterEventPoller eventPo
this.remoteInterpreterEventPoller = eventPoller;
}
+ public void shutdown()
+ {
+
+ // Close client socket connection
+ if (clientFactory != null)
+ {
+ for (TSocket eachTransfer: clientFactory.clientSocketMap.values())
+ {
+ eachTransfer.close();
+ }
+ }
+ }
+
public int getConnectTimeout() {
return connectTimeout;
}
public synchronized Client getClient() throws Exception {
if (clientPool == null || clientPool.isClosed()) {
- clientPool = new GenericObjectPool<>(new ClientFactory(getHost(), getPort()));
+ clientFactory = new ClientFactory(getHost(), getPort());
+ clientPool = new GenericObjectPool<>(clientFactory);
Review comment:
the Factory should be saved when create, use to call the close method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] tylerba-f commented on issue #3650: [ZEPPELIN-4600]
nullpoint error in the long time process use api to run a note
Posted by GitBox <gi...@apache.org>.
tylerba-f commented on issue #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-586574059
code style adjust over
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] tylerba-f commented on issue #3650: [ZEPPELIN-4600]
Socket connection between zeppelin server and interpreter is not closed
properly for branch 0.8
Posted by GitBox <gi...@apache.org>.
tylerba-f commented on issue #3650: [ZEPPELIN-4600] Socket connection between zeppelin server and interpreter is not closed properly for branch 0.8
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-586953518
master branch PR was commited
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3650: [ZEPPELIN-4600]
nullpoint error in the long time process use api to run a note
Posted by GitBox <gi...@apache.org>.
zjffdu commented on issue #3650: [ZEPPELIN-4600] nullpoint error in the long time process use api to run a note
URL: https://github.com/apache/zeppelin/pull/3650#issuecomment-586797121
@tylerba-f Could you update the PR title as well ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services