You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by sat <sa...@gmail.com> on 2017/12/22 23:28:53 UTC
Discrepancy in nodetool status
Hi,
We have 3 nodes in cluster, we rebooted one of the cassandra VM, we noticed
nodetool status returning "UN" for itself and "DN" for other node, although
we observe gossip sync and ack messages being shared between these nodes.
*Issue in Detail*
*Nodes in cluster*
Node1
Node 2
Node 3
All above nodes formed cluster and nodetool status in all 3 machines were
"UN"
We rebooted Node 1 and restarted cassandra on node1, then we ran nodetool
status and observed
Node 1 - UN
Node 2 - DN
Node 3 - DN
However when we run nodetool status on other 2 nodes (Node2, and Node 3)
they claim all 3 nodes are "UN"
We enabled "Trace" level and checked Gossip messages and noticed "SYN",
"ACK" and "ACK2" initiated and received messages in Node 1 for other 2
nodes, but still nodetool status marks other 2 nodes as down.
Please let us know how nodetool detects other nodes as "DOWN". Any help is
highly appreciated.
Thanks
A.SathishKumar
Re: Discrepancy in nodetool status
Posted by sat <sa...@gmail.com>.
Hi,
We guess the issue we are facing is related to
https://issues.apache.org/jira/browse/CASSANDRA-9630.
Will it be fixed in 3.11 release.
Thanks and Regards
A.SathishKumar
On Fri, Dec 22, 2017 at 6:15 PM, sat <sa...@gmail.com> wrote:
> Hi,
>
> We tried rebooting again Node 1 and this time we observed nodetool status
> displaying "UN" for all 3 nodes in node1.
>
> Executing nodetool status on "Node 3" displays "UN" for all the nodes.
>
> Executing nodetool status on "Node 2" displays "DN" for node 1 (rebooted
> node) and "UN" for other 2 nodes. We also observed "Node 2" sending "Syn"
> message to "Node 1", but no "Ack" received from "Node 1" initially for
> 15min and then it is started receiving it.
>
> Please let us know the reason why "Node 2" is not receiving any "Ack"
> message from "Node 1" for 10 -15 minutes and how it suddenly able to
> receive "Ac" messages.
>
> Inter node communication port is 7000.
>
> 3 node cluster, and we have all 3 nodes as seed ips.
>
> Thanks and Regards
> A.SathishKumar
>
>
>
> On Fri, Dec 22, 2017 at 5:15 PM, sat <sa...@gmail.com> wrote:
>
>> Hi,
>>
>> We checked and we were able to telnet to port 7000.
>>
>> Thanks and Regards
>> A.SathishKumar
>>
>> On Fri, Dec 22, 2017 at 3:43 PM, Nitan Kainth <ni...@gmail.com>
>> wrote:
>>
>>> Try telnet on your listen port. It must be network issue due to port or
>>> firewall issue.
>>>
>>> Sent from my iPhone
>>>
>>> On Dec 22, 2017, at 5:28 PM, sat <sa...@gmail.com> wrote:
>>>
>>>
>>>
>>> Hi,
>>>
>>> We have 3 nodes in cluster, we rebooted one of the cassandra VM, we
>>> noticed nodetool status returning "UN" for itself and "DN" for other node,
>>> although we observe gossip sync and ack messages being shared between these
>>> nodes.
>>>
>>> *Issue in Detail*
>>>
>>> *Nodes in cluster*
>>> Node1
>>> Node 2
>>> Node 3
>>>
>>> All above nodes formed cluster and nodetool status in all 3 machines
>>> were "UN"
>>>
>>> We rebooted Node 1 and restarted cassandra on node1, then we ran
>>> nodetool status and observed
>>>
>>> Node 1 - UN
>>> Node 2 - DN
>>> Node 3 - DN
>>>
>>> However when we run nodetool status on other 2 nodes (Node2, and Node 3)
>>> they claim all 3 nodes are "UN"
>>>
>>> We enabled "Trace" level and checked Gossip messages and noticed "SYN",
>>> "ACK" and "ACK2" initiated and received messages in Node 1 for other 2
>>> nodes, but still nodetool status marks other 2 nodes as down.
>>>
>>> Please let us know how nodetool detects other nodes as "DOWN". Any help
>>> is highly appreciated.
>>>
>>> Thanks
>>> A.SathishKumar
>>>
>>>
>>
>>
>> --
>> A.SathishKumar
>> 044-24735023
>>
>
>
>
> --
> A.SathishKumar
> 044-24735023
>
--
A.SathishKumar
044-24735023
Re: Discrepancy in nodetool status
Posted by sat <sa...@gmail.com>.
Hi,
We tried rebooting again Node 1 and this time we observed nodetool status
displaying "UN" for all 3 nodes in node1.
Executing nodetool status on "Node 3" displays "UN" for all the nodes.
Executing nodetool status on "Node 2" displays "DN" for node 1 (rebooted
node) and "UN" for other 2 nodes. We also observed "Node 2" sending "Syn"
message to "Node 1", but no "Ack" received from "Node 1" initially for
15min and then it is started receiving it.
Please let us know the reason why "Node 2" is not receiving any "Ack"
message from "Node 1" for 10 -15 minutes and how it suddenly able to
receive "Ac" messages.
Inter node communication port is 7000.
3 node cluster, and we have all 3 nodes as seed ips.
Thanks and Regards
A.SathishKumar
On Fri, Dec 22, 2017 at 5:15 PM, sat <sa...@gmail.com> wrote:
> Hi,
>
> We checked and we were able to telnet to port 7000.
>
> Thanks and Regards
> A.SathishKumar
>
> On Fri, Dec 22, 2017 at 3:43 PM, Nitan Kainth <ni...@gmail.com>
> wrote:
>
>> Try telnet on your listen port. It must be network issue due to port or
>> firewall issue.
>>
>> Sent from my iPhone
>>
>> On Dec 22, 2017, at 5:28 PM, sat <sa...@gmail.com> wrote:
>>
>>
>>
>> Hi,
>>
>> We have 3 nodes in cluster, we rebooted one of the cassandra VM, we
>> noticed nodetool status returning "UN" for itself and "DN" for other node,
>> although we observe gossip sync and ack messages being shared between these
>> nodes.
>>
>> *Issue in Detail*
>>
>> *Nodes in cluster*
>> Node1
>> Node 2
>> Node 3
>>
>> All above nodes formed cluster and nodetool status in all 3 machines were
>> "UN"
>>
>> We rebooted Node 1 and restarted cassandra on node1, then we ran nodetool
>> status and observed
>>
>> Node 1 - UN
>> Node 2 - DN
>> Node 3 - DN
>>
>> However when we run nodetool status on other 2 nodes (Node2, and Node 3)
>> they claim all 3 nodes are "UN"
>>
>> We enabled "Trace" level and checked Gossip messages and noticed "SYN",
>> "ACK" and "ACK2" initiated and received messages in Node 1 for other 2
>> nodes, but still nodetool status marks other 2 nodes as down.
>>
>> Please let us know how nodetool detects other nodes as "DOWN". Any help
>> is highly appreciated.
>>
>> Thanks
>> A.SathishKumar
>>
>>
>
>
> --
> A.SathishKumar
> 044-24735023
>
--
A.SathishKumar
044-24735023
Re: Discrepancy in nodetool status
Posted by sat <sa...@gmail.com>.
Hi,
We checked and we were able to telnet to port 7000.
Thanks and Regards
A.SathishKumar
On Fri, Dec 22, 2017 at 3:43 PM, Nitan Kainth <ni...@gmail.com> wrote:
> Try telnet on your listen port. It must be network issue due to port or
> firewall issue.
>
> Sent from my iPhone
>
> On Dec 22, 2017, at 5:28 PM, sat <sa...@gmail.com> wrote:
>
>
>
> Hi,
>
> We have 3 nodes in cluster, we rebooted one of the cassandra VM, we
> noticed nodetool status returning "UN" for itself and "DN" for other node,
> although we observe gossip sync and ack messages being shared between these
> nodes.
>
> *Issue in Detail*
>
> *Nodes in cluster*
> Node1
> Node 2
> Node 3
>
> All above nodes formed cluster and nodetool status in all 3 machines were
> "UN"
>
> We rebooted Node 1 and restarted cassandra on node1, then we ran nodetool
> status and observed
>
> Node 1 - UN
> Node 2 - DN
> Node 3 - DN
>
> However when we run nodetool status on other 2 nodes (Node2, and Node 3)
> they claim all 3 nodes are "UN"
>
> We enabled "Trace" level and checked Gossip messages and noticed "SYN",
> "ACK" and "ACK2" initiated and received messages in Node 1 for other 2
> nodes, but still nodetool status marks other 2 nodes as down.
>
> Please let us know how nodetool detects other nodes as "DOWN". Any help is
> highly appreciated.
>
> Thanks
> A.SathishKumar
>
>
--
A.SathishKumar
044-24735023
Re: Discrepancy in nodetool status
Posted by Nitan Kainth <ni...@gmail.com>.
Try telnet on your listen port. It must be network issue due to port or firewall issue.
Sent from my iPhone
> On Dec 22, 2017, at 5:28 PM, sat <sa...@gmail.com> wrote:
>
>
>
> Hi,
>
> We have 3 nodes in cluster, we rebooted one of the cassandra VM, we noticed nodetool status returning "UN" for itself and "DN" for other node, although we observe gossip sync and ack messages being shared between these nodes.
>
> Issue in Detail
>
> Nodes in cluster
> Node1
> Node 2
> Node 3
>
> All above nodes formed cluster and nodetool status in all 3 machines were "UN"
>
> We rebooted Node 1 and restarted cassandra on node1, then we ran nodetool status and observed
>
> Node 1 - UN
> Node 2 - DN
> Node 3 - DN
>
> However when we run nodetool status on other 2 nodes (Node2, and Node 3) they claim all 3 nodes are "UN"
>
> We enabled "Trace" level and checked Gossip messages and noticed "SYN", "ACK" and "ACK2" initiated and received messages in Node 1 for other 2 nodes, but still nodetool status marks other 2 nodes as down.
>
> Please let us know how nodetool detects other nodes as "DOWN". Any help is highly appreciated.
>
> Thanks
> A.SathishKumar