You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Gaojinchao <ga...@huawei.com> on 2011/04/22 05:36:10 UTC
Creating table with regions failed when zk crashed.
Is there any issue about this ?
2011-04-21 14:48:24,676 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed wfan_1,3238613814230765,1303367938566.63a83bdc9b55ec115cbc9b4bbe318214.
2011-04-21 14:48:24,676 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 4 on 20000.logSyncer interrupted while waiting for sync requests
2011-04-21 14:48:24,676 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 4 on 20000.logSyncer exiting
2011-04-21 14:48:24,677 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://162.2.6.187:9000/hbase/wfan_1/63a83bdc9b55ec115cbc9b4bbe318214/.logs
2011-04-21 14:48:24,719 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Moved 1 log files to /hbase/wfan_1/63a83bdc9b55ec115cbc9b4bbe318214/.oldlogs
2011-04-21 14:48:24,728 WARN org.apache.hadoop.hbase.zookeeper.ZKTable: Moving table wfan_1 state to enabled but was already enabled
2011-04-21 14:48:24,912 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.77.0:2181
2011-04-21 14:48:24,912 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
2011-04-21 14:48:24,914 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
2011-04-21 14:48:25,308 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.77.0:2181
2011-04-21 14:48:25,308 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
2011-04-21 14:48:25,309 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0004, likely server has closed socket, closing socket connection and attempting reconnect
2011-04-21 14:48:25,410 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call createTable({NAME => 'wfan_1', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '5184000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}, [[B@2637df06) from 162.2.134.26:3705: output error
2011-04-21 14:48:25,410 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 20000 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
2011-04-21 14:48:25,416 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.6.187:2181
2011-04-21 14:48:25,417 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0007 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
2011-04-21 14:48:25,678 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
2011-04-21 14:48:25,678 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-16-6/162.2.16.6:2181, initiating session
2011-04-21 14:48:25,679 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-77-0/162.2.77.0:2181
2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
2011-04-21 14:48:26,044 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
2011-04-21 14:48:26,297 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.6.187:2181
2011-04-21 14:48:26,298 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0004 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
2011-04-21 14:48:26,472 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
2011-04-21 14:48:26,472 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-16-6/162.2.16.6:2181, initiating session
2011-04-21 14:48:26,474 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server 162-2-16-6/162.2.16.6:2181, sessionid = 0x12f76bfa98a0004, negotiated timeout = 40000
2011-04-21 14:48:27,052 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-6-187/162.2.6.187:2181
2011-04-21 14:48:27,053 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0007 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
2011-04-21 14:48:27,318 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
Re: Creating table with regions failed when zk crashed.
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Ah yeah the ConnectionLoss bubbled all the way up and it should have
been retried, see https://issues.apache.org/jira/browse/HBASE-3065
J-D
On Mon, Apr 25, 2011 at 11:25 PM, Gaojinchao <ga...@huawei.com> wrote:
> Sorry, I don't know about zk. Please help me.
> Thanks.
> Do you mean that need change any ZK parameter?
>
> This is all logs about zk, Hmaster and client.
> It seems like the problem is zk leader crashed.
>
> client logs:
>
> 11/04/26 12:25:04 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a81003b, likely server has closed socket, closing socket connection and attempting reconnect
> java.io.IOException: java.io.IOException: Unable to ensure that the table will be enabled because of a ZooKeeper issue
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)
> at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:342)
> at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:302)
> at com.huawei.smartcare.performanceTuning.createTable.main(createTable.java:103)
> Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Unable to ensure that the table will be enabled because of a ZooKeeper issue
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:791)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:776)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:743)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/table/ufdr
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:978)
> at org.apache.hadoop.hbase.zookeeper.ZKTable.setEnabledTable(ZKTable.java:297)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:789)
> ... 8 more
>
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> at $Proxy4.createTable(Unknown Source)
> at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:340)
> ... 2 more
>
> Hmaster Logs:
> 2011-04-26 12:25:04,477 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://C4C1:9000/hbase/ufdr/93b8fb6c7f35c2b32a2d510be764ccd2/.logs
> 2011-04-26 12:25:04,513 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Moved 1 log files to /hbase/ufdr/93b8fb6c7f35c2b32a2d510be764ccd2/.oldlogs
> 2011-04-26 12:25:04,537 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a810006, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-26 12:25:04,538 WARN org.apache.hadoop.hbase.zookeeper.ZKTable: Moving table ufdr state to enabled but was already enabled
> 2011-04-26 12:25:04,642 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C2/157.5.100.2:2181
> 2011-04-26 12:25:04,642 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to C4C2/157.5.100.2:2181, initiating session
> 2011-04-26 12:25:04,644 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a810006, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-26 12:25:05,430 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C3/157.5.100.3:2181
> 2011-04-26 12:25:05,431 WARN org.apache.zookeeper.ClientCnxn: Session 0x2f8fc38a810006 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
> 2011-04-26 12:25:05,606 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C1/157.5.100.1:2181
> 2011-04-26 12:25:05,607 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to C4C1/157.5.100.1:2181, initiating session
> 2011-04-26 12:25:05,607 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a810006, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-26 12:25:06,569 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C2/157.5.100.2:2181
> 2011-04-26 12:25:06,570 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to C4C2/157.5.100.2:2181, initiating session
> 2011-04-26 12:25:06,571 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a810006, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-26 12:25:06,722 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C3/157.5.100.3:2181
> 2011-04-26 12:25:06,723 WARN org.apache.zookeeper.ClientCnxn: Session 0x2f8fc38a810006 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
> 2011-04-26 12:25:07,496 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C1/157.5.100.1:2181
> 2011-04-26 12:25:07,496 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to C4C1/157.5.100.1:2181, initiating session
> 2011-04-26 12:25:07,497 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server C4C1/157.5.100.1:2181, sessionid = 0x2f8fc38a810006, negotiated timeout = 180000
>
> ZK logs:
> 2011-04-26 12:25:04,507 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Connection broken for id 2, my id = 1, error = java.io.IOException: Channel eof
> 2011-04-26 12:25:04,507 WARN org.apache.zookeeper.server.quorum.Learner: Exception when following the leader
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
> at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
> at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:140)
> at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:644)
> 2011-04-26 12:25:04,508 INFO org.apache.zookeeper.server.quorum.Learner: shutdown called
> java.lang.Exception: shutdown Follower
> at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:648)
> 2011-04-26 12:25:04,508 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /157.5.100.2:58431 which had sessionid 0x12f8fc388e70002
> 2011-04-26 12:25:04,509 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /157.5.100.3:47415 which had sessionid 0x12f8fc388e70004
> 2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request processor complete
> 2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.quorum.CommitProcessor: CommitProcessor exited loop!
> 2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.quorum.FollowerRequestProcessor: FollowerRequestProcessor exited loop!
> 2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor exited!
> 2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING
> 2011-04-26 12:25:04,529 INFO org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot /tmp/hbase-root/zookeeper/version-2/snapshot.ad00000000
> 2011-04-26 12:25:04,617 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /157.5.100.1:47986
> 2011-04-26 12:25:04,618 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
> 2011-04-26 12:25:04,618 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /157.5.100.1:47986 (no session established for client)
> 2011-04-26 12:25:04,691 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /157.5.100.5:46718
> 2011-04-26 12:25:04,692 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
> 2011-04-26 12:25:04,692 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /157.5.100.5:46718 (no session established for client)
> 2011-04-26 12:25:04,709 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id = 1, Proposed zxid = 743029366343
> 2011-04-26 12:25:04,710 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 1 (n.leader), 743029366343 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
> 2011-04-26 12:25:04,710 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Exception when using channel: for id 2 my id = 1 error = java.nio.channels.ClosedChannelException
> 2011-04-26 12:25:04,710 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Send worker leaving thread
> 2011-04-26 12:25:04,717 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 0 (n.leader), 743029366343 (n.zxid), 2 (n.round), LOOKING (n.state), 0 (n.sid), LOOKING (my state)
> 2011-04-26 12:25:04,718 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 1 (n.leader), 743029366343 (n.zxid), 2 (n.round), LOOKING (n.state), 0 (n.sid), LOOKING (my state)
> 2011-04-26 12:25:04,918 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LEADING
> 2011-04-26 12:25:04,920 INFO org.apache.zookeeper.server.quorum.Leader: TCP NoDelay set to: true
> 2011-04-26 12:25:04,921 INFO org.apa
>
> --
>
Re: Creating table with regions failed when zk crashed.
Posted by Gaojinchao <ga...@huawei.com>.
Sorry, I don't know about zk. Please help me.
Thanks.
Do you mean that need change any ZK parameter?
This is all logs about zk, Hmaster and client.
It seems like the problem is zk leader crashed.
client logs:
11/04/26 12:25:04 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a81003b, likely server has closed socket, closing socket connection and attempting reconnect
java.io.IOException: java.io.IOException: Unable to ensure that the table will be enabled because of a ZooKeeper issue
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:80)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:342)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:302)
at com.huawei.smartcare.performanceTuning.createTable.main(createTable.java:103)
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Unable to ensure that the table will be enabled because of a ZooKeeper issue
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:791)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:776)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:743)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/table/ufdr
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:978)
at org.apache.hadoop.hbase.zookeeper.ZKTable.setEnabledTable(ZKTable.java:297)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:789)
... 8 more
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy4.createTable(Unknown Source)
at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:340)
... 2 more
Hmaster Logs:
2011-04-26 12:25:04,477 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://C4C1:9000/hbase/ufdr/93b8fb6c7f35c2b32a2d510be764ccd2/.logs
2011-04-26 12:25:04,513 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Moved 1 log files to /hbase/ufdr/93b8fb6c7f35c2b32a2d510be764ccd2/.oldlogs
2011-04-26 12:25:04,537 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a810006, likely server has closed socket, closing socket connection and attempting reconnect
2011-04-26 12:25:04,538 WARN org.apache.hadoop.hbase.zookeeper.ZKTable: Moving table ufdr state to enabled but was already enabled
2011-04-26 12:25:04,642 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C2/157.5.100.2:2181
2011-04-26 12:25:04,642 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to C4C2/157.5.100.2:2181, initiating session
2011-04-26 12:25:04,644 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a810006, likely server has closed socket, closing socket connection and attempting reconnect
2011-04-26 12:25:05,430 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C3/157.5.100.3:2181
2011-04-26 12:25:05,431 WARN org.apache.zookeeper.ClientCnxn: Session 0x2f8fc38a810006 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
2011-04-26 12:25:05,606 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C1/157.5.100.1:2181
2011-04-26 12:25:05,607 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to C4C1/157.5.100.1:2181, initiating session
2011-04-26 12:25:05,607 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a810006, likely server has closed socket, closing socket connection and attempting reconnect
2011-04-26 12:25:06,569 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C2/157.5.100.2:2181
2011-04-26 12:25:06,570 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to C4C2/157.5.100.2:2181, initiating session
2011-04-26 12:25:06,571 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x2f8fc38a810006, likely server has closed socket, closing socket connection and attempting reconnect
2011-04-26 12:25:06,722 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C3/157.5.100.3:2181
2011-04-26 12:25:06,723 WARN org.apache.zookeeper.ClientCnxn: Session 0x2f8fc38a810006 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
2011-04-26 12:25:07,496 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server C4C1/157.5.100.1:2181
2011-04-26 12:25:07,496 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to C4C1/157.5.100.1:2181, initiating session
2011-04-26 12:25:07,497 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server C4C1/157.5.100.1:2181, sessionid = 0x2f8fc38a810006, negotiated timeout = 180000
ZK logs:
2011-04-26 12:25:04,507 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Connection broken for id 2, my id = 1, error = java.io.IOException: Channel eof
2011-04-26 12:25:04,507 WARN org.apache.zookeeper.server.quorum.Learner: Exception when following the leader
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:84)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:140)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:78)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:644)
2011-04-26 12:25:04,508 INFO org.apache.zookeeper.server.quorum.Learner: shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:648)
2011-04-26 12:25:04,508 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /157.5.100.2:58431 which had sessionid 0x12f8fc388e70002
2011-04-26 12:25:04,509 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /157.5.100.3:47415 which had sessionid 0x12f8fc388e70004
2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request processor complete
2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.quorum.CommitProcessor: CommitProcessor exited loop!
2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.quorum.FollowerRequestProcessor: FollowerRequestProcessor exited loop!
2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor exited!
2011-04-26 12:25:04,510 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING
2011-04-26 12:25:04,529 INFO org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot /tmp/hbase-root/zookeeper/version-2/snapshot.ad00000000
2011-04-26 12:25:04,617 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /157.5.100.1:47986
2011-04-26 12:25:04,618 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2011-04-26 12:25:04,618 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /157.5.100.1:47986 (no session established for client)
2011-04-26 12:25:04,691 INFO org.apache.zookeeper.server.NIOServerCnxn: Accepted socket connection from /157.5.100.5:46718
2011-04-26 12:25:04,692 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2011-04-26 12:25:04,692 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /157.5.100.5:46718 (no session established for client)
2011-04-26 12:25:04,709 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id = 1, Proposed zxid = 743029366343
2011-04-26 12:25:04,710 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 1 (n.leader), 743029366343 (n.zxid), 2 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)
2011-04-26 12:25:04,710 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Exception when using channel: for id 2 my id = 1 error = java.nio.channels.ClosedChannelException
2011-04-26 12:25:04,710 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Send worker leaving thread
2011-04-26 12:25:04,717 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 0 (n.leader), 743029366343 (n.zxid), 2 (n.round), LOOKING (n.state), 0 (n.sid), LOOKING (my state)
2011-04-26 12:25:04,718 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 1 (n.leader), 743029366343 (n.zxid), 2 (n.round), LOOKING (n.state), 0 (n.sid), LOOKING (my state)
2011-04-26 12:25:04,918 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LEADING
2011-04-26 12:25:04,920 INFO org.apache.zookeeper.server.quorum.Leader: TCP NoDelay set to: true
2011-04-26 12:25:04,921 INFO org.apa
--
Re: Creating table with regions failed when zk crashed.
Posted by Jean-Daniel Cryans <jd...@apache.org>.
So I'm guessing that the log you pasted was from the master, and I can
see the zookeeper doing retries and strangely enough it was kicked out
by the other ZK peers:
2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to 162-2-77-0/162.2.77.0:2181, initiating
session
2011-04-21 14:48:26,044 INFO org.apache.zookeeper.ClientCnxn: Unable
to read additional data from server sessionid 0x12f76bfa98a0007,
likely server has closed socket, closing socket connection and
attempting reconnect
What's going on in the ZK logs during that time. It seems to me that
something strange is going on.
Also if the line you pointed out did throw an exception, it should
have been sent back to the client... what was it? SessionExpired?
To me it looks like HBase did the right thing, it refused to create a
table because it wasn't table to write to zookeeper. but the reason
for why it wasn't able to isn't clear to me.
J-D
On Sat, Apr 23, 2011 at 1:35 AM, Gaojinchao <ga...@huawei.com> wrote:
> In this case:
> 1. creating table with regions .
> 2. zk crashed ( My cluster has 3 zk process , when leader crashed)
> 3.creating table failed.
>
> I had found the code:
>
> Please confirm whether it need fixed?
>
> for(HRegionInfo newRegion : newRegions) {
>
> // 1. Set table enabling flag up in zk.
> try {
> assignmentManager.getZKTable().setEnabledTable(tableName); // This function throw exception.
> } catch (KeeperException e) {
> throw new IOException("Unable to ensure that the table will be" +
> " enabled because of a ZooKeeper issue", e);
> }
>
> // 2. Create HRegion
> HRegion region = HRegion.createHRegion(newRegion,
> fileSystemManager.getRootDir(), conf);
>
> // 3. Insert into META
> MetaEditor.addRegionToMeta(catalogTracker, region.getRegionInfo());
>
> // 4. Close the new region to flush to disk. Close log file too.
> region.close();
> region.getLog().closeAndDelete();
> }
>
>
>
> -----邮件原件-----
> 发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
> 发送时间: 2011年4月23日 3:10
> 收件人: user@hbase.apache.org
> 主题: Re: Creating table with regions failed when zk crashed.
>
> What exactly happened here? As much as I enjoy reading logs, I also
> enjoy short descriptions of the context of what I'm looking at.
>
> J-D
>
> On Thu, Apr 21, 2011 at 8:36 PM, Gaojinchao <ga...@huawei.com> wrote:
>> Is there any issue about this ?
>>
>>
>> 2011-04-21 14:48:24,676 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed wfan_1,3238613814230765,1303367938566.63a83bdc9b55ec115cbc9b4bbe318214.
>> 2011-04-21 14:48:24,676 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 4 on 20000.logSyncer interrupted while waiting for sync requests
>> 2011-04-21 14:48:24,676 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 4 on 20000.logSyncer exiting
>> 2011-04-21 14:48:24,677 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://162.2.6.187:9000/hbase/wfan_1/63a83bdc9b55ec115cbc9b4bbe318214/.logs
>> 2011-04-21 14:48:24,719 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Moved 1 log files to /hbase/wfan_1/63a83bdc9b55ec115cbc9b4bbe318214/.oldlogs
>> 2011-04-21 14:48:24,728 WARN org.apache.hadoop.hbase.zookeeper.ZKTable: Moving table wfan_1 state to enabled but was already enabled
>> 2011-04-21 14:48:24,912 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.77.0:2181
>> 2011-04-21 14:48:24,912 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
>> 2011-04-21 14:48:24,914 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
>> 2011-04-21 14:48:25,308 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.77.0:2181
>> 2011-04-21 14:48:25,308 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
>> 2011-04-21 14:48:25,309 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0004, likely server has closed socket, closing socket connection and attempting reconnect
>> 2011-04-21 14:48:25,410 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call createTable({NAME => 'wfan_1', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '5184000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}, [[B@2637df06) from 162.2.134.26:3705: output error
>> 2011-04-21 14:48:25,410 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 20000 caught: java.nio.channels.ClosedChannelException
>> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>> at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
>> at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>> at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>>
>> 2011-04-21 14:48:25,416 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.6.187:2181
>> 2011-04-21 14:48:25,417 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0007 for server null, unexpected error, closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> 2011-04-21 14:48:25,678 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
>> 2011-04-21 14:48:25,678 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-16-6/162.2.16.6:2181, initiating session
>> 2011-04-21 14:48:25,679 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
>> 2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-77-0/162.2.77.0:2181
>> 2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
>> 2011-04-21 14:48:26,044 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
>> 2011-04-21 14:48:26,297 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.6.187:2181
>> 2011-04-21 14:48:26,298 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0004 for server null, unexpected error, closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> 2011-04-21 14:48:26,472 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
>> 2011-04-21 14:48:26,472 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-16-6/162.2.16.6:2181, initiating session
>> 2011-04-21 14:48:26,474 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server 162-2-16-6/162.2.16.6:2181, sessionid = 0x12f76bfa98a0004, negotiated timeout = 40000
>> 2011-04-21 14:48:27,052 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-6-187/162.2.6.187:2181
>> 2011-04-21 14:48:27,053 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0007 for server null, unexpected error, closing socket connection and attempting reconnect
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> 2011-04-21 14:48:27,318 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
>>
>
Re: Creating table with regions failed when zk crashed.
Posted by Gaojinchao <ga...@huawei.com>.
In this case:
1. creating table with regions .
2. zk crashed ( My cluster has 3 zk process , when leader crashed)
3.creating table failed.
I had found the code:
Please confirm whether it need fixed?
for(HRegionInfo newRegion : newRegions) {
// 1. Set table enabling flag up in zk.
try {
assignmentManager.getZKTable().setEnabledTable(tableName); // This function throw exception.
} catch (KeeperException e) {
throw new IOException("Unable to ensure that the table will be" +
" enabled because of a ZooKeeper issue", e);
}
// 2. Create HRegion
HRegion region = HRegion.createHRegion(newRegion,
fileSystemManager.getRootDir(), conf);
// 3. Insert into META
MetaEditor.addRegionToMeta(catalogTracker, region.getRegionInfo());
// 4. Close the new region to flush to disk. Close log file too.
region.close();
region.getLog().closeAndDelete();
}
-----邮件原件-----
发件人: jdcryans@gmail.com [mailto:jdcryans@gmail.com] 代表 Jean-Daniel Cryans
发送时间: 2011年4月23日 3:10
收件人: user@hbase.apache.org
主题: Re: Creating table with regions failed when zk crashed.
What exactly happened here? As much as I enjoy reading logs, I also
enjoy short descriptions of the context of what I'm looking at.
J-D
On Thu, Apr 21, 2011 at 8:36 PM, Gaojinchao <ga...@huawei.com> wrote:
> Is there any issue about this ?
>
>
> 2011-04-21 14:48:24,676 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed wfan_1,3238613814230765,1303367938566.63a83bdc9b55ec115cbc9b4bbe318214.
> 2011-04-21 14:48:24,676 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 4 on 20000.logSyncer interrupted while waiting for sync requests
> 2011-04-21 14:48:24,676 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 4 on 20000.logSyncer exiting
> 2011-04-21 14:48:24,677 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://162.2.6.187:9000/hbase/wfan_1/63a83bdc9b55ec115cbc9b4bbe318214/.logs
> 2011-04-21 14:48:24,719 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Moved 1 log files to /hbase/wfan_1/63a83bdc9b55ec115cbc9b4bbe318214/.oldlogs
> 2011-04-21 14:48:24,728 WARN org.apache.hadoop.hbase.zookeeper.ZKTable: Moving table wfan_1 state to enabled but was already enabled
> 2011-04-21 14:48:24,912 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.77.0:2181
> 2011-04-21 14:48:24,912 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
> 2011-04-21 14:48:24,914 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-21 14:48:25,308 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.77.0:2181
> 2011-04-21 14:48:25,308 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
> 2011-04-21 14:48:25,309 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0004, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-21 14:48:25,410 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call createTable({NAME => 'wfan_1', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '5184000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}, [[B@2637df06) from 162.2.134.26:3705: output error
> 2011-04-21 14:48:25,410 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 20000 caught: java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>
> 2011-04-21 14:48:25,416 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.6.187:2181
> 2011-04-21 14:48:25,417 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0007 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> 2011-04-21 14:48:25,678 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
> 2011-04-21 14:48:25,678 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-16-6/162.2.16.6:2181, initiating session
> 2011-04-21 14:48:25,679 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-77-0/162.2.77.0:2181
> 2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
> 2011-04-21 14:48:26,044 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-21 14:48:26,297 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.6.187:2181
> 2011-04-21 14:48:26,298 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0004 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> 2011-04-21 14:48:26,472 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
> 2011-04-21 14:48:26,472 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-16-6/162.2.16.6:2181, initiating session
> 2011-04-21 14:48:26,474 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server 162-2-16-6/162.2.16.6:2181, sessionid = 0x12f76bfa98a0004, negotiated timeout = 40000
> 2011-04-21 14:48:27,052 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-6-187/162.2.6.187:2181
> 2011-04-21 14:48:27,053 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0007 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> 2011-04-21 14:48:27,318 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
>
Re: Creating table with regions failed when zk crashed.
Posted by Jean-Daniel Cryans <jd...@apache.org>.
What exactly happened here? As much as I enjoy reading logs, I also
enjoy short descriptions of the context of what I'm looking at.
J-D
On Thu, Apr 21, 2011 at 8:36 PM, Gaojinchao <ga...@huawei.com> wrote:
> Is there any issue about this ?
>
>
> 2011-04-21 14:48:24,676 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed wfan_1,3238613814230765,1303367938566.63a83bdc9b55ec115cbc9b4bbe318214.
> 2011-04-21 14:48:24,676 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 4 on 20000.logSyncer interrupted while waiting for sync requests
> 2011-04-21 14:48:24,676 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: IPC Server handler 4 on 20000.logSyncer exiting
> 2011-04-21 14:48:24,677 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: closing hlog writer in hdfs://162.2.6.187:9000/hbase/wfan_1/63a83bdc9b55ec115cbc9b4bbe318214/.logs
> 2011-04-21 14:48:24,719 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Moved 1 log files to /hbase/wfan_1/63a83bdc9b55ec115cbc9b4bbe318214/.oldlogs
> 2011-04-21 14:48:24,728 WARN org.apache.hadoop.hbase.zookeeper.ZKTable: Moving table wfan_1 state to enabled but was already enabled
> 2011-04-21 14:48:24,912 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.77.0:2181
> 2011-04-21 14:48:24,912 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
> 2011-04-21 14:48:24,914 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-21 14:48:25,308 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.77.0:2181
> 2011-04-21 14:48:25,308 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
> 2011-04-21 14:48:25,309 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0004, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-21 14:48:25,410 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call createTable({NAME => 'wfan_1', FAMILIES => [{NAME => 'value', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '5184000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}, [[B@2637df06) from 162.2.134.26:3705: output error
> 2011-04-21 14:48:25,410 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 20000 caught: java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>
> 2011-04-21 14:48:25,416 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.6.187:2181
> 2011-04-21 14:48:25,417 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0007 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> 2011-04-21 14:48:25,678 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
> 2011-04-21 14:48:25,678 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-16-6/162.2.16.6:2181, initiating session
> 2011-04-21 14:48:25,679 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-77-0/162.2.77.0:2181
> 2011-04-21 14:48:26,043 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-77-0/162.2.77.0:2181, initiating session
> 2011-04-21 14:48:26,044 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x12f76bfa98a0007, likely server has closed socket, closing socket connection and attempting reconnect
> 2011-04-21 14:48:26,297 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /162.2.6.187:2181
> 2011-04-21 14:48:26,298 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0004 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> 2011-04-21 14:48:26,472 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
> 2011-04-21 14:48:26,472 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to 162-2-16-6/162.2.16.6:2181, initiating session
> 2011-04-21 14:48:26,474 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server 162-2-16-6/162.2.16.6:2181, sessionid = 0x12f76bfa98a0004, negotiated timeout = 40000
> 2011-04-21 14:48:27,052 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-6-187/162.2.6.187:2181
> 2011-04-21 14:48:27,053 WARN org.apache.zookeeper.ClientCnxn: Session 0x12f76bfa98a0007 for server null, unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> 2011-04-21 14:48:27,318 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 162-2-16-6/162.2.16.6:2181
>