You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Hitesh Shah (JIRA)" <ji...@apache.org> on 2015/08/18 00:54:46 UTC

[jira] [Commented] (TEZ-2728) Wrap IPC connection Exception as SessionNotRunning - RM crash

    [ https://issues.apache.org/jira/browse/TEZ-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700383#comment-14700383 ] 

Hitesh Shah commented on TEZ-2728:
----------------------------------

Could you attach the full stack trace/log? 

This looks like the ipc call eventually timed out. I am not sure whether we can safely assume that the session is not running if the RM is down but could later come back up and recover the yarn application. 

Instead should Hive consider treating any exception from submitDAG as an excuse to try killing the session and re-trying with a new one?

 

> Wrap IPC connection Exception as SessionNotRunning - RM crash
> -------------------------------------------------------------
>
>                 Key: TEZ-2728
>                 URL: https://issues.apache.org/jira/browse/TEZ-2728
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.7.0, 0.5.4, 0.6.2, 0.8.0
>            Reporter: Gopal V
>            Assignee: Hitesh Shah
>
> Crashing the RM when a query session is open and restarting it does not result in a recoverable state for a Hive session.
> {code}
> 2015-08-17T22:34:21,981 INFO  [main]: ipc.Client (Client.java:handleConnectionFailure(885)) - Retrying connect to server: cn042-10.sandbox.hortonworks.com/172.19.128.42:10200. Already tried 48 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
> 2015-08-17T22:34:22,982 INFO  [main]: ipc.Client (Client.java:handleConnectionFailure(885)) - Retrying connect to server: cn042-10.sandbox.hortonworks.com/172.19.128.42:10200. Already tried 49 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
> 2015-08-17T22:34:22,987 ERROR [main]: exec.Task (TezTask.java:execute(195)) - Failed to execute tez graph.
> java.net.ConnectException: Call From cn041.sandbox.hortonworks.com/172.19.128.41 to cn042.sandbox.hortonworks.com:10200 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_51]
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_51]
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_51]
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_51]
>         at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) ~[hadoop-common-2.8.0-20150722.003145-873.jar:?]
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) ~[hadoop-common-2.8.0-20150722.003145-873.jar:?]
>         at org.apache.hadoop.ipc.Client.call(Client.java:1444) ~[hadoop-common-2.8.0-20150722.003145-873.jar:?]
>         at org.apache.hadoop.ipc.Client.call(Client.java:1371) ~[hadoop-common-2.8.0-20150722.003145-873.jar:?]
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[hadoop-common-2.8.0-20150722.003145-873.jar:?]
>         at com.sun.proxy.$Proxy41.getApplicationReport(Unknown Source) ~[?:?]
>         at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationHistoryProtocolPBClientImpl.getApplicationReport(ApplicationHistoryProtocolPBClientImpl.java:108) ~[hadoop-yarn-common-2.8.0-20150721.221214-843.jar:?]
>         at org.apache.hadoop.yarn.client.api.impl.AHSClientImpl.getApplicationReport(AHSClientImpl.java:101) ~[hadoop-yarn-client-2.8.0-20150721.221233-841.jar:?]
>         at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:442) ~[hadoop-yarn-client-2.8.0-20150721.221233-841.
> jar:?]
>         at org.apache.tez.client.TezYarnClient.getApplicationReport(TezYarnClient.java:89) ~[tez-api-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
>         at org.apache.tez.client.TezClientUtils.getAMProxy(TezClientUtils.java:835) ~[tez-api-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
>         at org.apache.tez.client.TezClient.getAMProxy(TezClient.java:713) ~[tez-api-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
>         at org.apache.tez.client.TezClient.waitForProxy(TezClient.java:723) ~[tez-api-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
>         at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:453) ~[tez-api-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
>         at org.apache.tez.client.TezClient.submitDAG(TezClient.java:391) ~[tez-api-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
>         at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:409) ~[hive-exec-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)