[jira] [Created] (ZOOKEEPER-4378) org.apache.zookeeper.server.NettyServerCnxn does not increase outstandingCount properly, cause outstandingCount to overflow.

["linmaolin (Jira)" <ji...@apache.org>]

linmaolin created ZOOKEEPER-4378:
------------------------------------

             Summary: org.apache.zookeeper.server.NettyServerCnxn does not increase outstandingCount properly, cause outstandingCount to overflow.
                 Key: ZOOKEEPER-4378
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4378
             Project: ZooKeeper
          Issue Type: Bug
    Affects Versions: 3.5.9
         Environment: 3.5.9.

The connection status response gives a very large outstanding_requests.

 
{quote}# curl -s http://127.0.0.1:18080/commands/stats | jq .secure_connections
[
 {
 "remote_socket_address": "10.103.249.25:50284",
 "interest_ops": 1,
 "outstanding_requests": 16587,
 "packets_received": 72288,
 "packets_sent": 72288
 },
 {
 "remote_socket_address": "10.103.249.25:50232",
 "interest_ops": 1,
 "outstanding_requests": 10369,
 "packets_received": 18197,
 "packets_sent": 18198
 },
 {
 "remote_socket_address": "10.103.249.25:53490",
 "interest_ops": 1,
 "outstanding_requests": 2867,
 "packets_received": 3179,
 "packets_sent": 3180
 }
]
{quote}
            Reporter: linmaolin
             Fix For: 3.6.0


In the receiveMessage method the count is increased right after processPacket without judges anything as below.
{quote}if (initialized) {
 // TODO: if zks.processPacket() is changed to take a ByteBuffer[],
 // we could implement zero-copy queueing.
 zks.processPacket(this, bb);

if (zks.shouldThrottle(outstandingCount.incrementAndGet())) {
 disableRecvNoWait();
 }
 }
{quote}
But after the request is handled, the decrease operation is taken only when xid is larger than 0.
{quote}@Override
 public void sendResponse(ReplyHeader h, Record r, String tag)
 throws IOException {
 if (closingChannel || !channel.isOpen()) {
 return;
 }
 super.sendResponse(h, r, tag);
 if (h.getXid() > 0) {
 // zks cannot be null otherwise we would not have gotten here!
 if (!zkServer.shouldThrottle(outstandingCount.decrementAndGet())) {
 enableRecv();
 }
 }
 }
{quote}
 

So the bultin xids like "PING", "AUTH", will make outstandingCount larger and larger, until it hit the limit; All the request on that connection will be refused.

 

I see the problem is solved in 3.6.0 version, should there be a patch for 3.5.9? Looking forward for your reply, Thantks!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)