You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by "Kislay Kumar (JIRA)" <ji...@apache.org> on 2018/11/12 02:39:00 UTC
[jira] [Created] (ZEPPELIN-3860) Zeppelin Broken pipe due to non
terminating RemoteInterpreterProcess
Kislay Kumar created ZEPPELIN-3860:
--------------------------------------
Summary: Zeppelin Broken pipe due to non terminating RemoteInterpreterProcess
Key: ZEPPELIN-3860
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3860
Project: Zeppelin
Issue Type: Bug
Affects Versions: 0.7.3
Environment: HDP 2.6
Reporter: Kislay Kumar
This issue may be related to : ZEPPELIN-1832 (it had temporary fix)
Broken pipe exception is repeatedly coming for Zeppelin. I starts with SocketException.
Some logs are as under:
WARNING: A servlet request to the URI https://xxyy:9995/api/login contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
Exception in thread "Thread-815" Exception in thread "Thread-809" java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
at org.apache.zeppelin.interpreter.InterpreterSetting.shutdownAndRemoveAllInterpreterGroups(InterpreterSetting.java:263)
at org.apache.zeppelin.interpreter.InterpreterSettingManager$7.run(InterpreterSettingManager.java:1083)
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
at org.apache.zeppelin.interpreter.InterpreterSetting.shutdownAndRemoveAllInterpreterGroups(InterpreterSetting.java:263)
at org.apache.zeppelin.interpreter.InterpreterSettingManager$7.run(InterpreterSettingManager.java:1083)
Zeppelin is restarting
More logs from
INFO [2018-11-09 00:00:01,334] (\{Thread-825} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException
-----------------------------------------
DEBUG [2018-11-09 00:01:31,742] (\{Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@39aaeb44\{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
DEBUG [2018-11-09 00:01:31,743] (\{qtp240166646-959} AbstractConnection.java[onFillInterestedFailed]:168) - HttpConnection@6b820a35[IDLE,DecryptedEndPoint@379cfbb3\{/10.111.1.138:60502<->9995,Open,in,out,-,-,30002/30000,HttpConnection}->SelectChannelEndPoint@7b0e8d49\{/10.111.1.138:60502<->9995,Open,in,out,-,-,1/30000,SslConnection}{io=1,kio=1,kro=1}][p=HttpParser\{s=CLOSED,0 of -1},g=HttpGenerator\{s=START},c=HttpChannelOverHttp@5456b387\{r=9,c=false,a=IDLE,uri=}] onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
DEBUG [2018-11-09 00:01:31,743] (\{Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@cbde656\{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
DEBUG [2018-11-09 00:01:31,744] (\{qtp240166646-975} AbstractConnection.java[onFillInterestedFailed]:168) - HttpConnection@671a181f[IDLE,DecryptedEndPoint@4bb91807\{/10.111.1.138:60503<->9995,Open,in,out,-,-,30001/30000,HttpConnection}->SelectChannelEndPoint@efc362c\{/10.111.1.138:60503<->9995,Open,in,out,-,-,0/30000,SslConnection}{io=1,kio=1,kro=1}][p=HttpParser\{s=CLOSED,0 of -1},g=HttpGenerator\{s=START},c=HttpChannelOverHttp@524d84e3\{r=8,c=false,a=IDLE,uri=}] onFillInterestedFailed java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
DEBUG [2018-11-09 00:01:32,477] (\{Scheduler-1989335500} WriteFlusher.java[onFail]:455) - ignored: WriteFlusher@4c568d5d\{IDLE} java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
---------------------------
INFO [2018-11-09 01:17:08,547] (\{Thread-824} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException
org.apache.thrift.transport.TTransportException
org.apache.thrift.transport.TTransportException
ERROR [2018-11-09 01:17:08,609] (\{qtp240166646-1125 - /api/helium/suggest/2D2V1VPF3/20171128-114541_374495247} ResourcePoolUtils.java[getAllResourcesExcept]:64) - org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
org.apache.thrift.transport.TTransportException
DEBUG [2018-11-09 01:17:09,424] (\{qtp240166646-949} WriteFlusher.java[onFail]:460) - failed: WriteFlusher@248e2b43\{PENDING} java.nio.channels.ClosedChannelException
INFO [2018-11-09 01:17:11,007] (\{Thread-1017} RemoteInterpreterProcess.java[dereference]:133) - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
org.apache.thrift.transport.TTransportException
--------------------------------------------------------------
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)