You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jaydeepkumar Chovatia (JIRA)" <ji...@apache.org> on 2018/04/24 20:58:08 UTC
[jira] [Comment Edited] (CASSANDRA-11551) Incorrect counting of
pending messages in OutboundTcpConnection
[ https://issues.apache.org/jira/browse/CASSANDRA-11551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450341#comment-16450341 ]
Jaydeepkumar Chovatia edited comment on CASSANDRA-11551 at 4/24/18 8:57 PM:
----------------------------------------------------------------------------
I think I've found the root cause for this issue, one possible reason is once we break at [this point|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L270] then {{currentMsgBufferCount}} doesn't get reset and at that time {{backlog.size()}} is 0 but {{currentMsgBufferCount}} could be > 0 and call to {{getPendingMessages}} at this time will give > 0 pending message even though in reality it is none.
This problem will not happen most likely on {{trunk}} because it doesn't use {{currentMsgBufferCount}} instead it directly deals with {{backlog.size()}} in [getPendingMessages|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/async/OutboundMessagingConnection.java#L659]
I've fixed this for older branches, please find fix here:
||3.11||3.0||2.2||
|[diff |https://github.com/apache/cassandra/compare/cassandra-3.11...jaydeepkumar1984:11551-3.11?expand=1]|[diff |https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:11551-3.0?expand=1]|[diff|https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:11551-2.2?expand=1]|
|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-3.11.svg?style=svg! |https://circleci.com/gh/jaydeepkumar1984/cassandra/63]|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-3.0.svg?style=svg! |https://circleci.com/gh/jaydeepkumar1984/cassandra/65]|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-2.2.svg?style=svg! |https://circleci.com/gh/jaydeepkumar1984/cassandra/69]|
Jaydeep
was (Author: chovatia.jaydeep@gmail.com):
I think I've found the root cause for this issue, one possible reason is once we break at [this point|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L270] then {{currentMsgBufferCount}} doesn't get reset and at that time {{backlog.size()}} is 0 but {{currentMsgBufferCount}} could be > 0 and call to {{getPendingMessages}} at this time will give > 0 pending message even though in reality it is none.
This problem will not happen most likely on {{trunk}} because it doesn't use {{currentMsgBufferCount}} instead it directly deals with {{backlog.size()}} in [getPendingMessages|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/async/OutboundMessagingConnection.java#L659]
I've fixed this for older branches, please find fix here:
||3.11||3.0||2.2||
|[diff |https://github.com/apache/cassandra/compare/cassandra-3.11...jaydeepkumar1984:11551-3.11?expand=1]|[diff |https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:11551-3.0?expand=1]|[diff|https://github.com/apache/cassandra/compare/trunk...jaydeepkumar1984:11551-2.2?expand=1]|
|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-3.11.svg?style=svg! |https://circleci.com/gh/jaydeepkumar1984/cassandra/63]|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-3.0.svg?style=svg! |https://circleci.com/gh/jaydeepkumar1984/cassandra/65]|[!https://circleci.com/gh/jaydeepkumar1984/cassandra/tree/11551-2.2.svg?style=svg! |https://circleci.com/gh/jaydeepkumar1984/cassandra/66]|
Jaydeep
> Incorrect counting of pending messages in OutboundTcpConnection
> ---------------------------------------------------------------
>
> Key: CASSANDRA-11551
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11551
> Project: Cassandra
> Issue Type: Bug
> Reporter: Robert Stupp
> Assignee: Jaydeepkumar Chovatia
> Priority: Minor
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
>
> Somehow {{OutboundTcpConnection.getPendingMessages()}} seems to return a wrong number.
> {code}
> nodetool netstats
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 1655
> Mismatch (Blocking): 0
> Mismatch (Background): 2
> Pool Name Active Pending Completed
> Large messages n/a 5 0
> Small messages n/a 0 31534100
> Gossip messages n/a 0 520393
> {code}
> Inspection of the heap dump of that node unveiled that all instances of {{OutboundTcpConnection.backlog}} are empty but {{currentMsgBufferCount}} is {{1}} for 5 instances of {{OutboundTcpConnection}}.
> Maybe the cause is in {{OutboundTcpConnection.run()}} where {{drainedMessages.size()}} is called twice but assumed that these are equal.
> /cc [~aweisberg]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org