You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Vaibhav Gumashta (JIRA)" <ji...@apache.org> on 2015/05/05 03:26:07 UTC
[jira] [Commented] (HIVE-10570) HiveServer2 shut downs due to
temporary ZooKeeper unavailability, causes permanent outage instead of
temporary
[ https://issues.apache.org/jira/browse/HIVE-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527729#comment-14527729 ]
Vaibhav Gumashta commented on HIVE-10570:
-----------------------------------------
[~thejas] I think this is what you were mentioning about ZK network partitioning issues.
> HiveServer2 shut downs due to temporary ZooKeeper unavailability, causes permanent outage instead of temporary
> --------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-10570
> URL: https://issues.apache.org/jira/browse/HIVE-10570
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 0.14.0
> Environment: HDP 2.2
> Reporter: Hari Sekhon
> Priority: Critical
>
> HiveServer2 should not shut down when there is temporary ZooKeeper unavailability (eg. temporary network outage). This prevents retry and recovery later as HiveServer2 is no longer running and therefore cannot retry - HiveServer2 stays offline indefinitely until operator intervention to restart it, even for minor temporary problems.
> I believe this behaviour is due to recent ZooKeeper dependency addition for HiveServer2 HA.
> {code}2015-05-01 11:35:05,367 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1102)) - Session 0x14d004cb02c001e for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.SocketException: Network is unreachable
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
> at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> 2015-05-01 11:35:05,629 INFO client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:run(285)) - Client will use GSSAPI as SASL mechanism.
> 2015-05-01 11:35:05,630 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server <custom_scrubbed>/<ip>:2181. Will attempt to SASL-authenticate using Login Context section 'HiveZooKeeperClient'
> 2015-05-01 11:35:05,630 ERROR zookeeper.ClientCnxnSocketNIO (ClientCnxnSocketNIO.java:connect(289)) - Unable to open socket to <custom_scrubbed>/<ip>:2181
> 2015-05-01 11:35:05,630 ERROR zookeeper.ClientCnxnSocketNIO (ClientCnxnSocketNIO.java:connect(289)) - Unable to open socket to <custom_scrubbed>/<ip>:2181
> 2015-05-01 11:35:05,630 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1102)) - Session 0x14d004cb02c001e for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.SocketException: Network is unreachable
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
> at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> 2015-05-01 11:35:05,943 INFO server.HiveServer2 (HiveServer2.java:stop(299)) - Shutting down HiveServer2
> 2015-05-01 11:35:05,944 INFO thrift.ThriftCLIService (ThriftCLIService.java:stop(137)) - Thrift server has stopped
> 2015-05-01 11:35:05,944 INFO service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
> 2015-05-01 11:35:05,944 INFO service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
> 2015-05-01 11:35:05,944 INFO service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
> 2015-05-01 11:35:05,946 INFO server.HiveServer2 (HiveStringUtils.java:run(679)) - SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down HiveServer2 at <fqdn>/<ip>
> ************************************************************/{code}
> Hari Sekhon
> http://www.linkedin.com/in/harisekhon
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)