You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Nitay Joffe (JIRA)" <ji...@apache.org> on 2009/06/06 02:04:07 UTC
[jira] Created: (HBASE-1491) ZooKeeper errors: "Client has seen
zxid 0xe our last zxid is 0xd"
ZooKeeper errors: "Client has seen zxid 0xe our last zxid is 0xd"
-----------------------------------------------------------------
Key: HBASE-1491
URL: https://issues.apache.org/jira/browse/HBASE-1491
Project: Hadoop HBase
Issue Type: Bug
Components: test
Reporter: Nitay Joffe
Assignee: Nitay Joffe
Fix For: 0.20.0
We have been seeing a lot of these messages in tests:
[junit] 2009-06-02 11:57:23,658 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(514): Client has seen zxid 0xe our last zxid is 0xd
They usually repeat in a seemingly endless loop, such as:
[junit] 2009-06-02 13:27:54,083 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(737): Priming connection to java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:56511 remote=localhost/0:0:0:0:0:0:0:1:21810]
[junit] 2009-06-02 13:27:54,084 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(889): Server connection successful
[junit] 2009-06-02 13:27:54,093 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(532): Connected to /0:0:0:0:0:0:0:1%0:56511 lastZxid 16
[junit] 2009-06-02 13:27:54,094 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(543): Client has seen zxid 0x10 our last zxid is 0x4
[junit] 2009-06-02 13:27:54,094 WARN [NIOServerCxn.Factory:21810] server.NIOServerCnxn(444): Exception causing close of session 0x0 due to java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
[junit] 2009-06-02 13:27:54,094 DEBUG [NIOServerCxn.Facto777ry:21810] server.NIOServerCnxn(447): IOException stack trace
[junit] java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
[junit] at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:544)
[junit] at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:331)
[junit] at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:176)
[junit] 2009-06-02 13:27:54,094 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(777): closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:21810 remote=/0:0:0:0:0:0:0:1%0:56511]
[junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(919): Exception closing session 0x121a2a7c43a0002 to sun.nio.ch.SelectionKeyImpl@2c662b4e
[junit] java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
[junit] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
[junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
[junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(953): Ignoring exception during shutdown input
[junit] java.net.SocketException: Socket is not connected
[junit] at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
[junit] at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
[junit] at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
[junit] at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
[junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1491) ZooKeeper errors: "Client has seen
zxid 0xe our last zxid is 0xd"
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nitay Joffe updated HBASE-1491:
-------------------------------
Status: Patch Available (was: Open)
> ZooKeeper errors: "Client has seen zxid 0xe our last zxid is 0xd"
> -----------------------------------------------------------------
>
> Key: HBASE-1491
> URL: https://issues.apache.org/jira/browse/HBASE-1491
> Project: Hadoop HBase
> Issue Type: Bug
> Components: test
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1491.patch
>
>
> We have been seeing a lot of these messages in tests:
> [junit] 2009-06-02 11:57:23,658 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(514): Client has seen zxid 0xe our last zxid is 0xd
> They usually repeat in a seemingly endless loop, such as:
> [junit] 2009-06-02 13:27:54,083 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(737): Priming connection to java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:56511 remote=localhost/0:0:0:0:0:0:0:1:21810]
> [junit] 2009-06-02 13:27:54,084 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(889): Server connection successful
> [junit] 2009-06-02 13:27:54,093 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(532): Connected to /0:0:0:0:0:0:0:1%0:56511 lastZxid 16
> [junit] 2009-06-02 13:27:54,094 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(543): Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 WARN [NIOServerCxn.Factory:21810] server.NIOServerCnxn(444): Exception causing close of session 0x0 due to java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 DEBUG [NIOServerCxn.Facto777ry:21810] server.NIOServerCnxn(447): IOException stack trace
> [junit] java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:544)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:331)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:176)
> [junit] 2009-06-02 13:27:54,094 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(777): closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:21810 remote=/0:0:0:0:0:0:0:1%0:56511]
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(919): Exception closing session 0x121a2a7c43a0002 to sun.nio.ch.SelectionKeyImpl@2c662b4e
> [junit] java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(953): Ignoring exception during shutdown input
> [junit] java.net.SocketException: Socket is not connected
> [junit] at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> [junit] at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> [junit] at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1491) ZooKeeper errors: "Client has seen
zxid 0xe our last zxid is 0xd"
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716924#action_12716924 ]
Jonathan Gray commented on HBASE-1491:
--------------------------------------
Tried out and all tests I tried still pass. Have not reviewed the patch yet.
For some reason I'm having problems finding those zxid messages without the patch :) More to follow...
> ZooKeeper errors: "Client has seen zxid 0xe our last zxid is 0xd"
> -----------------------------------------------------------------
>
> Key: HBASE-1491
> URL: https://issues.apache.org/jira/browse/HBASE-1491
> Project: Hadoop HBase
> Issue Type: Bug
> Components: test
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1491.patch
>
>
> We have been seeing a lot of these messages in tests:
> [junit] 2009-06-02 11:57:23,658 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(514): Client has seen zxid 0xe our last zxid is 0xd
> They usually repeat in a seemingly endless loop, such as:
> [junit] 2009-06-02 13:27:54,083 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(737): Priming connection to java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:56511 remote=localhost/0:0:0:0:0:0:0:1:21810]
> [junit] 2009-06-02 13:27:54,084 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(889): Server connection successful
> [junit] 2009-06-02 13:27:54,093 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(532): Connected to /0:0:0:0:0:0:0:1%0:56511 lastZxid 16
> [junit] 2009-06-02 13:27:54,094 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(543): Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 WARN [NIOServerCxn.Factory:21810] server.NIOServerCnxn(444): Exception causing close of session 0x0 due to java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 DEBUG [NIOServerCxn.Facto777ry:21810] server.NIOServerCnxn(447): IOException stack trace
> [junit] java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:544)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:331)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:176)
> [junit] 2009-06-02 13:27:54,094 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(777): closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:21810 remote=/0:0:0:0:0:0:0:1%0:56511]
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(919): Exception closing session 0x121a2a7c43a0002 to sun.nio.ch.SelectionKeyImpl@2c662b4e
> [junit] java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(953): Ignoring exception during shutdown input
> [junit] java.net.SocketException: Socket is not connected
> [junit] at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> [junit] at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> [junit] at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1491) ZooKeeper errors: "Client has seen
zxid 0xe our last zxid is 0xd"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717119#action_12717119 ]
stack commented on HBASE-1491:
------------------------------
Talked w/ Nitay... and jgray is ok w/ committing.
Nitay, on commit, call resetZK at lines #53 if it makes sense.
Good stuff.
> ZooKeeper errors: "Client has seen zxid 0xe our last zxid is 0xd"
> -----------------------------------------------------------------
>
> Key: HBASE-1491
> URL: https://issues.apache.org/jira/browse/HBASE-1491
> Project: Hadoop HBase
> Issue Type: Bug
> Components: test
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1491.patch
>
>
> We have been seeing a lot of these messages in tests:
> [junit] 2009-06-02 11:57:23,658 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(514): Client has seen zxid 0xe our last zxid is 0xd
> They usually repeat in a seemingly endless loop, such as:
> [junit] 2009-06-02 13:27:54,083 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(737): Priming connection to java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:56511 remote=localhost/0:0:0:0:0:0:0:1:21810]
> [junit] 2009-06-02 13:27:54,084 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(889): Server connection successful
> [junit] 2009-06-02 13:27:54,093 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(532): Connected to /0:0:0:0:0:0:0:1%0:56511 lastZxid 16
> [junit] 2009-06-02 13:27:54,094 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(543): Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 WARN [NIOServerCxn.Factory:21810] server.NIOServerCnxn(444): Exception causing close of session 0x0 due to java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 DEBUG [NIOServerCxn.Facto777ry:21810] server.NIOServerCnxn(447): IOException stack trace
> [junit] java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:544)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:331)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:176)
> [junit] 2009-06-02 13:27:54,094 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(777): closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:21810 remote=/0:0:0:0:0:0:0:1%0:56511]
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(919): Exception closing session 0x121a2a7c43a0002 to sun.nio.ch.SelectionKeyImpl@2c662b4e
> [junit] java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(953): Ignoring exception during shutdown input
> [junit] java.net.SocketException: Socket is not connected
> [junit] at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> [junit] at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> [junit] at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1491) ZooKeeper errors: "Client has seen
zxid 0xe our last zxid is 0xd"
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717116#action_12717116 ]
stack commented on HBASE-1491:
------------------------------
Why not call resetZooKeeper at lines #53 in HCM rather than duplicate a bit of code (test for null and if not, close). Otherwise patch looks good. I was going to commit but maybe JGray wants to play some more?
> ZooKeeper errors: "Client has seen zxid 0xe our last zxid is 0xd"
> -----------------------------------------------------------------
>
> Key: HBASE-1491
> URL: https://issues.apache.org/jira/browse/HBASE-1491
> Project: Hadoop HBase
> Issue Type: Bug
> Components: test
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1491.patch
>
>
> We have been seeing a lot of these messages in tests:
> [junit] 2009-06-02 11:57:23,658 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(514): Client has seen zxid 0xe our last zxid is 0xd
> They usually repeat in a seemingly endless loop, such as:
> [junit] 2009-06-02 13:27:54,083 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(737): Priming connection to java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:56511 remote=localhost/0:0:0:0:0:0:0:1:21810]
> [junit] 2009-06-02 13:27:54,084 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(889): Server connection successful
> [junit] 2009-06-02 13:27:54,093 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(532): Connected to /0:0:0:0:0:0:0:1%0:56511 lastZxid 16
> [junit] 2009-06-02 13:27:54,094 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(543): Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 WARN [NIOServerCxn.Factory:21810] server.NIOServerCnxn(444): Exception causing close of session 0x0 due to java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 DEBUG [NIOServerCxn.Facto777ry:21810] server.NIOServerCnxn(447): IOException stack trace
> [junit] java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:544)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:331)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:176)
> [junit] 2009-06-02 13:27:54,094 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(777): closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:21810 remote=/0:0:0:0:0:0:0:1%0:56511]
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(919): Exception closing session 0x121a2a7c43a0002 to sun.nio.ch.SelectionKeyImpl@2c662b4e
> [junit] java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(953): Ignoring exception during shutdown input
> [junit] java.net.SocketException: Socket is not connected
> [junit] at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> [junit] at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> [junit] at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1491) ZooKeeper errors: "Client has seen
zxid 0xe our last zxid is 0xd"
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nitay Joffe updated HBASE-1491:
-------------------------------
Attachment: hbase-1491.patch
Here is my solution for this. I ran this with TestGetRowVersions and am seeing none of these messages.
In this patch:
- Close ZooKeeperWrapper in Master, RegionServer, and TableServers when they're finishing.
- Add method to HConnectionManager to delete all connections.
- On MiniHBaseCluster shutdown, after LocalHBaseCluster has finished shutting down, delete all connections in HConnectionManager.
Is the last item safe to do? I think it should be. Note that without it we get some of these zkid messages.
> ZooKeeper errors: "Client has seen zxid 0xe our last zxid is 0xd"
> -----------------------------------------------------------------
>
> Key: HBASE-1491
> URL: https://issues.apache.org/jira/browse/HBASE-1491
> Project: Hadoop HBase
> Issue Type: Bug
> Components: test
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1491.patch
>
>
> We have been seeing a lot of these messages in tests:
> [junit] 2009-06-02 11:57:23,658 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(514): Client has seen zxid 0xe our last zxid is 0xd
> They usually repeat in a seemingly endless loop, such as:
> [junit] 2009-06-02 13:27:54,083 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(737): Priming connection to java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:56511 remote=localhost/0:0:0:0:0:0:0:1:21810]
> [junit] 2009-06-02 13:27:54,084 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(889): Server connection successful
> [junit] 2009-06-02 13:27:54,093 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(532): Connected to /0:0:0:0:0:0:0:1%0:56511 lastZxid 16
> [junit] 2009-06-02 13:27:54,094 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(543): Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 WARN [NIOServerCxn.Factory:21810] server.NIOServerCnxn(444): Exception causing close of session 0x0 due to java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 DEBUG [NIOServerCxn.Facto777ry:21810] server.NIOServerCnxn(447): IOException stack trace
> [junit] java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:544)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:331)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:176)
> [junit] 2009-06-02 13:27:54,094 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(777): closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:21810 remote=/0:0:0:0:0:0:0:1%0:56511]
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(919): Exception closing session 0x121a2a7c43a0002 to sun.nio.ch.SelectionKeyImpl@2c662b4e
> [junit] java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(953): Ignoring exception during shutdown input
> [junit] java.net.SocketException: Socket is not connected
> [junit] at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> [junit] at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> [junit] at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1491) ZooKeeper errors: "Client has seen
zxid 0xe our last zxid is 0xd"
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716800#action_12716800 ]
Nitay Joffe commented on HBASE-1491:
------------------------------------
I've been talking with ZooKeeper folks about this. Here's a link to the conversation on zookeeper-user:
http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/200906.mbox/%3C82b0992a0906021412p6b840baci1bb4e4d44e12223e@mail.gmail.com%3E
> ZooKeeper errors: "Client has seen zxid 0xe our last zxid is 0xd"
> -----------------------------------------------------------------
>
> Key: HBASE-1491
> URL: https://issues.apache.org/jira/browse/HBASE-1491
> Project: Hadoop HBase
> Issue Type: Bug
> Components: test
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
>
> We have been seeing a lot of these messages in tests:
> [junit] 2009-06-02 11:57:23,658 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(514): Client has seen zxid 0xe our last zxid is 0xd
> They usually repeat in a seemingly endless loop, such as:
> [junit] 2009-06-02 13:27:54,083 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(737): Priming connection to java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:56511 remote=localhost/0:0:0:0:0:0:0:1:21810]
> [junit] 2009-06-02 13:27:54,084 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(889): Server connection successful
> [junit] 2009-06-02 13:27:54,093 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(532): Connected to /0:0:0:0:0:0:0:1%0:56511 lastZxid 16
> [junit] 2009-06-02 13:27:54,094 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(543): Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 WARN [NIOServerCxn.Factory:21810] server.NIOServerCnxn(444): Exception causing close of session 0x0 due to java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 DEBUG [NIOServerCxn.Facto777ry:21810] server.NIOServerCnxn(447): IOException stack trace
> [junit] java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:544)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:331)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:176)
> [junit] 2009-06-02 13:27:54,094 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(777): closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:21810 remote=/0:0:0:0:0:0:0:1%0:56511]
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(919): Exception closing session 0x121a2a7c43a0002 to sun.nio.ch.SelectionKeyImpl@2c662b4e
> [junit] java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(953): Ignoring exception during shutdown input
> [junit] java.net.SocketException: Socket is not connected
> [junit] at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> [junit] at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> [junit] at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1491) ZooKeeper errors: "Client has seen
zxid 0xe our last zxid is 0xd"
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nitay Joffe updated HBASE-1491:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed, with change Stack suggested.
> ZooKeeper errors: "Client has seen zxid 0xe our last zxid is 0xd"
> -----------------------------------------------------------------
>
> Key: HBASE-1491
> URL: https://issues.apache.org/jira/browse/HBASE-1491
> Project: Hadoop HBase
> Issue Type: Bug
> Components: test
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1491.patch
>
>
> We have been seeing a lot of these messages in tests:
> [junit] 2009-06-02 11:57:23,658 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(514): Client has seen zxid 0xe our last zxid is 0xd
> They usually repeat in a seemingly endless loop, such as:
> [junit] 2009-06-02 13:27:54,083 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(737): Priming connection to java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:56511 remote=localhost/0:0:0:0:0:0:0:1:21810]
> [junit] 2009-06-02 13:27:54,084 INFO [main-SendThread] zookeeper.ClientCnxn$SendThread(889): Server connection successful
> [junit] 2009-06-02 13:27:54,093 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(532): Connected to /0:0:0:0:0:0:0:1%0:56511 lastZxid 16
> [junit] 2009-06-02 13:27:54,094 ERROR [NIOServerCxn.Factory:21810] server.NIOServerCnxn(543): Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 WARN [NIOServerCxn.Factory:21810] server.NIOServerCnxn(444): Exception causing close of session 0x0 due to java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] 2009-06-02 13:27:54,094 DEBUG [NIOServerCxn.Facto777ry:21810] server.NIOServerCnxn(447): IOException stack trace
> [junit] java.io.IOException: Client has seen zxid 0x10 our last zxid is 0x4
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.readConnectRequest(NIOServerCnxn.java:544)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:331)
> [junit] at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:176)
> [junit] 2009-06-02 13:27:54,094 INFO [NIOServerCxn.Factory:21810] server.NIOServerCnxn(777): closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:21810 remote=/0:0:0:0:0:0:0:1%0:56511]
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(919): Exception closing session 0x121a2a7c43a0002 to sun.nio.ch.SelectionKeyImpl@2c662b4e
> [junit] java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> [junit] 2009-06-02 13:27:54,097 WARN [main-SendThread] zookeeper.ClientCnxn$SendThread(953): Ignoring exception during shutdown input
> [junit] java.net.SocketException: Socket is not connected
> [junit] at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
> [junit] at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
> [junit] at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
> [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.