You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Rakesh R <ra...@huawei.com> on 2014/11/26 16:23:04 UTC
ZooKeeper client is getting "java.io.IOException: Nothing in the
queue, but got "
Hi,
Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
Client & server is using kerberoes based sasl authentication mechanism.
I'm using 3.4.6 release.
2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)] Client will use GSSAPI as SASL mechanism. org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClient.java:285)
2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)] Opening socket connection to server vm-01/**.**.**.**:24002. Will attempt to SASL-authenticate using Login Context section 'Client' org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1000)
2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)] Socket connection established to vm-216/**.**.**.**:24002, initiating session org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:855)
2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)] Session establishment complete on server vm-216/**.**.**.**:24002, sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000 org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1260)
2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)] Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002, unexpected error, closing socket connection and attempting reconnect org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
java.io.IOException: Nothing in the queue, but got 425467
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
This exception has occured in the following logic.
CilentCnxn#readResponse()
synchronized (pendingQueue) {
if (pendingQueue.size() == 0) {
throw new IOException("Nothing in the queue, but got "
+ replyHdr.getXid());
}
packet = pendingQueue.remove();
}
I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
ClientCnxnSocketNIO.java
sock.write(p.bb);
if (!p.bb.hasRemaining()) {
.....
.....
.....
pendingQueue.add(p);
}
Does anyone phase similar issue and would appreciate any help. Thanks!
Thanks & Regards,
Rakesh
RE: Re:RE: Re:ZooKeeper client is getting "java.io.IOException:
Nothing in the queue, but got "
Posted by Rakesh R <ra...@huawei.com>.
It was coming in one of the lab environment. Its a long run scenario and now its started running with my patch mentioned earlier. Probably I'll add log when I get another chance.
It looks like the packet was not getting added to the 'pendingQueue' due to the following condition.
sock.write(p.bb);
if (!p.bb.hasRemaining()) {
.....
.....
.....
pendingQueue.add(p);
}
-Rakesh
-----Original Message-----
From: Robin [mailto:rchzzjcn@163.com]
Sent: 28 November 2014 07:13
To: user@zookeeper.apache.org
Cc: Zookeeper; zookeeper-user
Subject: Re:RE: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
Rakesh,
I believe your problem is a little bit different than the bug I fired(ZOOKEEPER-2091). Your problem is because the SASL Packe is directly sent out and not put into the pendingQueue, so when the response Packet arrives, it will not find the request Packet in the pendingQueue.
BTW, could you add a log so that we can get more detail about the socket behavior ?
sock.write(p.bb);
while (p.bb.hasRemaining()) {
logger.info(...)
sock.write(p.bb);
}
Thanks,
- Robin
At 2014-11-27 20:09:35, "Rakesh R" <ra...@huawei.com> wrote:
>Yes exactly. I'm also thinking the same way. I have modified the code by iterating over the buffer and write fully to the socket.
>It looks fine now. Let me run the environment for some more time.
>
> sock.write(p.bb);
> while (p.bb.hasRemaining()) {
> sock.write(p.bb);
> }
>
>Regards,
>Rakesh
>-----Original Message-----
>From: Robin [mailto:rchzzjcn@163.com]
>Sent: 27 November 2014 17:34
>To: user@zookeeper.apache.org
>Cc: Zookeeper; zookeeper-user
>Subject: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
>
>Hi Rakesh,
>
>By the following call flow:
> SendThread#run()
> -> ZooKeeperSaslClient#initialize()
> -> ZooKeeperSaslClient#sendSaslPacket()
> -> ClientCnxn#sendPacket()
> -> SendThread#sendPacket()
> -> ClientCnxnSocketNIO#sendPacket() It seems that the Sasl Packet has not been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL response comes, it will not find the corresponding Packet in the pendingQueue.
>
>- Robin
>
>
>At 2014-11-26 23:23:04, "Rakesh R" <ra...@huawei.com> wrote:
>>Hi,
>>
>>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
>>Client & server is using kerberoes based sasl authentication mechanism.
>>
>>I'm using 3.4.6 release.
>>
>>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>>Client will use GSSAPI as SASL mechanism.
>>org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslCli
>>e
>>nt.java:285)
>>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>>Opening socket connection to server vm-01/**.**.**.**:24002. Will
>>attempt to SASL-authenticate using Login Context section 'Client'
>>org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.
>>j
>>ava:1000)
>>2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)]
>>Socket connection established to vm-216/**.**.**.**:24002, initiating
>>session
>>org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.
>>j
>>ava:855)
>>2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)]
>>Session establishment complete on server vm-216/**.**.**.**:24002,
>>sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000
>>org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:
>>1260)
>>2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)]
>>Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002,
>>unexpected error, closing socket connection and attempting reconnect
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>>java.io.IOException: Nothing in the queue, but got 425467
>> at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
>> at
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>>
>>
>>This exception has occured in the following logic.
>>
>>CilentCnxn#readResponse()
>>
>> synchronized (pendingQueue) {
>> if (pendingQueue.size() == 0) {
>> throw new IOException("Nothing in the queue, but got "
>> + replyHdr.getXid());
>> }
>> packet = pendingQueue.remove();
>> }
>>
>>
>>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
>>
>>ClientCnxnSocketNIO.java
>>
>> sock.write(p.bb);
>> if (!p.bb.hasRemaining()) {
>> .....
>> .....
>> .....
>> pendingQueue.add(p);
>> }
>>
>>Does anyone phase similar issue and would appreciate any help. Thanks!
>>
>>Thanks & Regards,
>>Rakesh
RE: Re:RE: Re:ZooKeeper client is getting "java.io.IOException:
Nothing in the queue, but got "
Posted by Rakesh R <ra...@huawei.com>.
It was coming in one of the lab environment. Its a long run scenario and now its started running with my patch mentioned earlier. Probably I'll add log when I get another chance.
It looks like the packet was not getting added to the 'pendingQueue' due to the following condition.
sock.write(p.bb);
if (!p.bb.hasRemaining()) {
.....
.....
.....
pendingQueue.add(p);
}
-Rakesh
-----Original Message-----
From: Robin [mailto:rchzzjcn@163.com]
Sent: 28 November 2014 07:13
To: user@zookeeper.apache.org
Cc: Zookeeper; zookeeper-user
Subject: Re:RE: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
Rakesh,
I believe your problem is a little bit different than the bug I fired(ZOOKEEPER-2091). Your problem is because the SASL Packe is directly sent out and not put into the pendingQueue, so when the response Packet arrives, it will not find the request Packet in the pendingQueue.
BTW, could you add a log so that we can get more detail about the socket behavior ?
sock.write(p.bb);
while (p.bb.hasRemaining()) {
logger.info(...)
sock.write(p.bb);
}
Thanks,
- Robin
At 2014-11-27 20:09:35, "Rakesh R" <ra...@huawei.com> wrote:
>Yes exactly. I'm also thinking the same way. I have modified the code by iterating over the buffer and write fully to the socket.
>It looks fine now. Let me run the environment for some more time.
>
> sock.write(p.bb);
> while (p.bb.hasRemaining()) {
> sock.write(p.bb);
> }
>
>Regards,
>Rakesh
>-----Original Message-----
>From: Robin [mailto:rchzzjcn@163.com]
>Sent: 27 November 2014 17:34
>To: user@zookeeper.apache.org
>Cc: Zookeeper; zookeeper-user
>Subject: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
>
>Hi Rakesh,
>
>By the following call flow:
> SendThread#run()
> -> ZooKeeperSaslClient#initialize()
> -> ZooKeeperSaslClient#sendSaslPacket()
> -> ClientCnxn#sendPacket()
> -> SendThread#sendPacket()
> -> ClientCnxnSocketNIO#sendPacket() It seems that the Sasl Packet has not been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL response comes, it will not find the corresponding Packet in the pendingQueue.
>
>- Robin
>
>
>At 2014-11-26 23:23:04, "Rakesh R" <ra...@huawei.com> wrote:
>>Hi,
>>
>>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
>>Client & server is using kerberoes based sasl authentication mechanism.
>>
>>I'm using 3.4.6 release.
>>
>>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>>Client will use GSSAPI as SASL mechanism.
>>org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslCli
>>e
>>nt.java:285)
>>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>>Opening socket connection to server vm-01/**.**.**.**:24002. Will
>>attempt to SASL-authenticate using Login Context section 'Client'
>>org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.
>>j
>>ava:1000)
>>2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)]
>>Socket connection established to vm-216/**.**.**.**:24002, initiating
>>session
>>org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.
>>j
>>ava:855)
>>2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)]
>>Session establishment complete on server vm-216/**.**.**.**:24002,
>>sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000
>>org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:
>>1260)
>>2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)]
>>Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002,
>>unexpected error, closing socket connection and attempting reconnect
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>>java.io.IOException: Nothing in the queue, but got 425467
>> at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
>> at
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>>
>>
>>This exception has occured in the following logic.
>>
>>CilentCnxn#readResponse()
>>
>> synchronized (pendingQueue) {
>> if (pendingQueue.size() == 0) {
>> throw new IOException("Nothing in the queue, but got "
>> + replyHdr.getXid());
>> }
>> packet = pendingQueue.remove();
>> }
>>
>>
>>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
>>
>>ClientCnxnSocketNIO.java
>>
>> sock.write(p.bb);
>> if (!p.bb.hasRemaining()) {
>> .....
>> .....
>> .....
>> pendingQueue.add(p);
>> }
>>
>>Does anyone phase similar issue and would appreciate any help. Thanks!
>>
>>Thanks & Regards,
>>Rakesh
Re:RE: Re:ZooKeeper client is getting "java.io.IOException: Nothing
in the queue, but got "
Posted by Robin <rc...@163.com>.
Rakesh,
I believe your problem is a little bit different than the bug I fired(ZOOKEEPER-2091). Your problem is because the SASL Packe is directly sent out and not put into the pendingQueue, so when the response Packet arrives, it will not find the request Packet in the pendingQueue.
BTW, could you add a log so that we can get more detail about the socket behavior ?
sock.write(p.bb);
while (p.bb.hasRemaining()) {
logger.info(...)
sock.write(p.bb);
}
Thanks,
- Robin
At 2014-11-27 20:09:35, "Rakesh R" <ra...@huawei.com> wrote:
>Yes exactly. I'm also thinking the same way. I have modified the code by iterating over the buffer and write fully to the socket.
>It looks fine now. Let me run the environment for some more time.
>
> sock.write(p.bb);
> while (p.bb.hasRemaining()) {
> sock.write(p.bb);
> }
>
>Regards,
>Rakesh
>-----Original Message-----
>From: Robin [mailto:rchzzjcn@163.com]
>Sent: 27 November 2014 17:34
>To: user@zookeeper.apache.org
>Cc: Zookeeper; zookeeper-user
>Subject: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
>
>Hi Rakesh,
>
>By the following call flow:
> SendThread#run()
> -> ZooKeeperSaslClient#initialize()
> -> ZooKeeperSaslClient#sendSaslPacket()
> -> ClientCnxn#sendPacket()
> -> SendThread#sendPacket()
> -> ClientCnxnSocketNIO#sendPacket() It seems that the Sasl Packet has not been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL response comes, it will not find the corresponding Packet in the pendingQueue.
>
>- Robin
>
>
>At 2014-11-26 23:23:04, "Rakesh R" <ra...@huawei.com> wrote:
>>Hi,
>>
>>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
>>Client & server is using kerberoes based sasl authentication mechanism.
>>
>>I'm using 3.4.6 release.
>>
>>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>>Client will use GSSAPI as SASL mechanism.
>>org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClie
>>nt.java:285)
>>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>>Opening socket connection to server vm-01/**.**.**.**:24002. Will
>>attempt to SASL-authenticate using Login Context section 'Client'
>>org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.j
>>ava:1000)
>>2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)]
>>Socket connection established to vm-216/**.**.**.**:24002, initiating
>>session
>>org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.j
>>ava:855)
>>2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)]
>>Session establishment complete on server vm-216/**.**.**.**:24002,
>>sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000
>>org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:
>>1260)
>>2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)]
>>Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002,
>>unexpected error, closing socket connection and attempting reconnect
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>>java.io.IOException: Nothing in the queue, but got 425467
>> at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
>> at
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>>
>>
>>This exception has occured in the following logic.
>>
>>CilentCnxn#readResponse()
>>
>> synchronized (pendingQueue) {
>> if (pendingQueue.size() == 0) {
>> throw new IOException("Nothing in the queue, but got "
>> + replyHdr.getXid());
>> }
>> packet = pendingQueue.remove();
>> }
>>
>>
>>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
>>
>>ClientCnxnSocketNIO.java
>>
>> sock.write(p.bb);
>> if (!p.bb.hasRemaining()) {
>> .....
>> .....
>> .....
>> pendingQueue.add(p);
>> }
>>
>>Does anyone phase similar issue and would appreciate any help. Thanks!
>>
>>Thanks & Regards,
>>Rakesh
Re:RE: Re:ZooKeeper client is getting "java.io.IOException: Nothing
in the queue, but got "
Posted by Robin <rc...@163.com>.
Rakesh,
I believe your problem is a little bit different than the bug I fired(ZOOKEEPER-2091). Your problem is because the SASL Packe is directly sent out and not put into the pendingQueue, so when the response Packet arrives, it will not find the request Packet in the pendingQueue.
BTW, could you add a log so that we can get more detail about the socket behavior ?
sock.write(p.bb);
while (p.bb.hasRemaining()) {
logger.info(...)
sock.write(p.bb);
}
Thanks,
- Robin
At 2014-11-27 20:09:35, "Rakesh R" <ra...@huawei.com> wrote:
>Yes exactly. I'm also thinking the same way. I have modified the code by iterating over the buffer and write fully to the socket.
>It looks fine now. Let me run the environment for some more time.
>
> sock.write(p.bb);
> while (p.bb.hasRemaining()) {
> sock.write(p.bb);
> }
>
>Regards,
>Rakesh
>-----Original Message-----
>From: Robin [mailto:rchzzjcn@163.com]
>Sent: 27 November 2014 17:34
>To: user@zookeeper.apache.org
>Cc: Zookeeper; zookeeper-user
>Subject: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
>
>Hi Rakesh,
>
>By the following call flow:
> SendThread#run()
> -> ZooKeeperSaslClient#initialize()
> -> ZooKeeperSaslClient#sendSaslPacket()
> -> ClientCnxn#sendPacket()
> -> SendThread#sendPacket()
> -> ClientCnxnSocketNIO#sendPacket() It seems that the Sasl Packet has not been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL response comes, it will not find the corresponding Packet in the pendingQueue.
>
>- Robin
>
>
>At 2014-11-26 23:23:04, "Rakesh R" <ra...@huawei.com> wrote:
>>Hi,
>>
>>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
>>Client & server is using kerberoes based sasl authentication mechanism.
>>
>>I'm using 3.4.6 release.
>>
>>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>>Client will use GSSAPI as SASL mechanism.
>>org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClie
>>nt.java:285)
>>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>>Opening socket connection to server vm-01/**.**.**.**:24002. Will
>>attempt to SASL-authenticate using Login Context section 'Client'
>>org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.j
>>ava:1000)
>>2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)]
>>Socket connection established to vm-216/**.**.**.**:24002, initiating
>>session
>>org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.j
>>ava:855)
>>2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)]
>>Session establishment complete on server vm-216/**.**.**.**:24002,
>>sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000
>>org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:
>>1260)
>>2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)]
>>Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002,
>>unexpected error, closing socket connection and attempting reconnect
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>>java.io.IOException: Nothing in the queue, but got 425467
>> at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
>> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
>> at
>>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>>
>>
>>This exception has occured in the following logic.
>>
>>CilentCnxn#readResponse()
>>
>> synchronized (pendingQueue) {
>> if (pendingQueue.size() == 0) {
>> throw new IOException("Nothing in the queue, but got "
>> + replyHdr.getXid());
>> }
>> packet = pendingQueue.remove();
>> }
>>
>>
>>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
>>
>>ClientCnxnSocketNIO.java
>>
>> sock.write(p.bb);
>> if (!p.bb.hasRemaining()) {
>> .....
>> .....
>> .....
>> pendingQueue.add(p);
>> }
>>
>>Does anyone phase similar issue and would appreciate any help. Thanks!
>>
>>Thanks & Regards,
>>Rakesh
RE: Re:ZooKeeper client is getting "java.io.IOException: Nothing in
the queue, but got "
Posted by Rakesh R <ra...@huawei.com>.
Yes exactly. I'm also thinking the same way. I have modified the code by iterating over the buffer and write fully to the socket.
It looks fine now. Let me run the environment for some more time.
sock.write(p.bb);
while (p.bb.hasRemaining()) {
sock.write(p.bb);
}
Regards,
Rakesh
-----Original Message-----
From: Robin [mailto:rchzzjcn@163.com]
Sent: 27 November 2014 17:34
To: user@zookeeper.apache.org
Cc: Zookeeper; zookeeper-user
Subject: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
Hi Rakesh,
By the following call flow:
SendThread#run()
-> ZooKeeperSaslClient#initialize()
-> ZooKeeperSaslClient#sendSaslPacket()
-> ClientCnxn#sendPacket()
-> SendThread#sendPacket()
-> ClientCnxnSocketNIO#sendPacket() It seems that the Sasl Packet has not been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL response comes, it will not find the corresponding Packet in the pendingQueue.
- Robin
At 2014-11-26 23:23:04, "Rakesh R" <ra...@huawei.com> wrote:
>Hi,
>
>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
>Client & server is using kerberoes based sasl authentication mechanism.
>
>I'm using 3.4.6 release.
>
>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>Client will use GSSAPI as SASL mechanism.
>org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClie
>nt.java:285)
>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>Opening socket connection to server vm-01/**.**.**.**:24002. Will
>attempt to SASL-authenticate using Login Context section 'Client'
>org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.j
>ava:1000)
>2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)]
>Socket connection established to vm-216/**.**.**.**:24002, initiating
>session
>org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.j
>ava:855)
>2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)]
>Session establishment complete on server vm-216/**.**.**.**:24002,
>sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000
>org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:
>1260)
>2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)]
>Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002,
>unexpected error, closing socket connection and attempting reconnect
>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>java.io.IOException: Nothing in the queue, but got 425467
> at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> at
>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>
>
>This exception has occured in the following logic.
>
>CilentCnxn#readResponse()
>
> synchronized (pendingQueue) {
> if (pendingQueue.size() == 0) {
> throw new IOException("Nothing in the queue, but got "
> + replyHdr.getXid());
> }
> packet = pendingQueue.remove();
> }
>
>
>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
>
>ClientCnxnSocketNIO.java
>
> sock.write(p.bb);
> if (!p.bb.hasRemaining()) {
> .....
> .....
> .....
> pendingQueue.add(p);
> }
>
>Does anyone phase similar issue and would appreciate any help. Thanks!
>
>Thanks & Regards,
>Rakesh
RE: Re:ZooKeeper client is getting "java.io.IOException: Nothing in
the queue, but got "
Posted by Rakesh R <ra...@huawei.com>.
Yes exactly. I'm also thinking the same way. I have modified the code by iterating over the buffer and write fully to the socket.
It looks fine now. Let me run the environment for some more time.
sock.write(p.bb);
while (p.bb.hasRemaining()) {
sock.write(p.bb);
}
Regards,
Rakesh
-----Original Message-----
From: Robin [mailto:rchzzjcn@163.com]
Sent: 27 November 2014 17:34
To: user@zookeeper.apache.org
Cc: Zookeeper; zookeeper-user
Subject: Re:ZooKeeper client is getting "java.io.IOException: Nothing in the queue, but got "
Hi Rakesh,
By the following call flow:
SendThread#run()
-> ZooKeeperSaslClient#initialize()
-> ZooKeeperSaslClient#sendSaslPacket()
-> ClientCnxn#sendPacket()
-> SendThread#sendPacket()
-> ClientCnxnSocketNIO#sendPacket() It seems that the Sasl Packet has not been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL response comes, it will not find the corresponding Packet in the pendingQueue.
- Robin
At 2014-11-26 23:23:04, "Rakesh R" <ra...@huawei.com> wrote:
>Hi,
>
>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
>Client & server is using kerberoes based sasl authentication mechanism.
>
>I'm using 3.4.6 release.
>
>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>Client will use GSSAPI as SASL mechanism.
>org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClie
>nt.java:285)
>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)]
>Opening socket connection to server vm-01/**.**.**.**:24002. Will
>attempt to SASL-authenticate using Login Context section 'Client'
>org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.j
>ava:1000)
>2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)]
>Socket connection established to vm-216/**.**.**.**:24002, initiating
>session
>org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.j
>ava:855)
>2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)]
>Session establishment complete on server vm-216/**.**.**.**:24002,
>sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000
>org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:
>1260)
>2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)]
>Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002,
>unexpected error, closing socket connection and attempting reconnect
>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>java.io.IOException: Nothing in the queue, but got 425467
> at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> at
>org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>
>
>This exception has occured in the following logic.
>
>CilentCnxn#readResponse()
>
> synchronized (pendingQueue) {
> if (pendingQueue.size() == 0) {
> throw new IOException("Nothing in the queue, but got "
> + replyHdr.getXid());
> }
> packet = pendingQueue.remove();
> }
>
>
>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
>
>ClientCnxnSocketNIO.java
>
> sock.write(p.bb);
> if (!p.bb.hasRemaining()) {
> .....
> .....
> .....
> pendingQueue.add(p);
> }
>
>Does anyone phase similar issue and would appreciate any help. Thanks!
>
>Thanks & Regards,
>Rakesh
Re:ZooKeeper client is getting "java.io.IOException: Nothing in the
queue, but got "
Posted by Robin <rc...@163.com>.
Hi Rakesh,
By the following call flow:
SendThread#run()
-> ZooKeeperSaslClient#initialize()
-> ZooKeeperSaslClient#sendSaslPacket()
-> ClientCnxn#sendPacket()
-> SendThread#sendPacket()
-> ClientCnxnSocketNIO#sendPacket()
It seems that the Sasl Packet has not been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL response comes, it will not find the corresponding Packet in the pendingQueue.
- Robin
At 2014-11-26 23:23:04, "Rakesh R" <ra...@huawei.com> wrote:
>Hi,
>
>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
>Client & server is using kerberoes based sasl authentication mechanism.
>
>I'm using 3.4.6 release.
>
>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)] Client will use GSSAPI as SASL mechanism. org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClient.java:285)
>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)] Opening socket connection to server vm-01/**.**.**.**:24002. Will attempt to SASL-authenticate using Login Context section 'Client' org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1000)
>2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)] Socket connection established to vm-216/**.**.**.**:24002, initiating session org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:855)
>2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)] Session establishment complete on server vm-216/**.**.**.**:24002, sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000 org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1260)
>2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)] Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002, unexpected error, closing socket connection and attempting reconnect org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>java.io.IOException: Nothing in the queue, but got 425467
> at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>
>
>This exception has occured in the following logic.
>
>CilentCnxn#readResponse()
>
> synchronized (pendingQueue) {
> if (pendingQueue.size() == 0) {
> throw new IOException("Nothing in the queue, but got "
> + replyHdr.getXid());
> }
> packet = pendingQueue.remove();
> }
>
>
>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
>
>ClientCnxnSocketNIO.java
>
> sock.write(p.bb);
> if (!p.bb.hasRemaining()) {
> .....
> .....
> .....
> pendingQueue.add(p);
> }
>
>Does anyone phase similar issue and would appreciate any help. Thanks!
>
>Thanks & Regards,
>Rakesh
Re:ZooKeeper client is getting "java.io.IOException: Nothing in the
queue, but got "
Posted by Robin <rc...@163.com>.
Hi Rakesh,
By the following call flow:
SendThread#run()
-> ZooKeeperSaslClient#initialize()
-> ZooKeeperSaslClient#sendSaslPacket()
-> ClientCnxn#sendPacket()
-> SendThread#sendPacket()
-> ClientCnxnSocketNIO#sendPacket()
It seems that the Sasl Packet has not been put into the pendingQueue after it is sent to the ZooKeeper Server. So when the SASL response comes, it will not find the corresponding Packet in the pendingQueue.
- Robin
At 2014-11-26 23:23:04, "Rakesh R" <ra...@huawei.com> wrote:
>Hi,
>
>Following exception is coming always for the client session 0xf49d7f31e4c0000 in zookeeper client logs.
>Client & server is using kerberoes based sasl authentication mechanism.
>
>I'm using 3.4.6 release.
>
>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)] Client will use GSSAPI as SASL mechanism. org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClient.java:285)
>2014-11-25 22:06:19,114 INFO [Thread-1-SendThread(vm-216:24002)] Opening socket connection to server vm-01/**.**.**.**:24002. Will attempt to SASL-authenticate using Login Context section 'Client' org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:1000)
>2014-11-25 22:06:19,115 INFO [Thread-1-SendThread(vm-216:24002)] Socket connection established to vm-216/**.**.**.**:24002, initiating session org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:855)
>2014-11-25 22:06:19,116 INFO [Thread-1-SendThread(vm-216:24002)] Session establishment complete on server vm-216/**.**.**.**:24002, sessionid = 0xf49d7f31e4c0000, negotiated timeout = 90000 org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1260)
>2014-11-25 22:06:19,118 WARN [Thread-1-SendThread(vm-216:24002)] Session 0xf49d7f31e4c0000 for server vm-216/**.**.**.**:24002, unexpected error, closing socket connection and attempting reconnect org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1127)
>java.io.IOException: Nothing in the queue, but got 425467
> at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:788)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1106)
>
>
>This exception has occured in the following logic.
>
>CilentCnxn#readResponse()
>
> synchronized (pendingQueue) {
> if (pendingQueue.size() == 0) {
> throw new IOException("Nothing in the queue, but got "
> + replyHdr.getXid());
> }
> packet = pendingQueue.remove();
> }
>
>
>I failed to find the reason why 'pendingQueue' becomes empty. Only possibility I could see is the following condition fails and missed to add it to the "pendingQueue"
>
>ClientCnxnSocketNIO.java
>
> sock.write(p.bb);
> if (!p.bb.hasRemaining()) {
> .....
> .....
> .....
> pendingQueue.add(p);
> }
>
>Does anyone phase similar issue and would appreciate any help. Thanks!
>
>Thanks & Regards,
>Rakesh