You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rahul Reddy <ra...@gmail.com> on 2019/07/19 15:48:44 UTC
Rebooting one Cassandra node caused all the application nodes go down
Here ,
We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have
RF 3 and cl set to local quorum. And gossip snitch. All our instance are
c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
instance type had a bug which aws asked to set nvme_timeout to higher
number in etc/grub.conf. after setting the parameter and did run nodetool
drain and reboot the node in east
Instance cameup but Cassandra didn't come up normal had to start the
Cassandra. Cassandra cameup but it shows other instances down. Even though
didn't reboot the other node down same was observed in one other node. How
could that happen and don't any errors in system.log which is set to info.
Without any intervention gossip settled in 10 mins entire cluster became
normal.
Tried same thing West it happened again
I'm concerned how to check what caused it and if a reboot happens again how
to avoid this.
If I just STOP Cassandra instead of reboot I don't see this issue.
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Rahul Reddy <ra...@gmail.com>.
Sorry no corruption errors.
Thanks Jeff,
Anything specific to look into if this happens again
On Fri, Jul 19, 2019, 2:40 PM Nitan Kainth <ni...@gmail.com> wrote:
> You no corruption error or you see corruption error?
>
>
> Regards,
>
> Nitan
>
> Cell: 510 449 9629
>
> On Jul 19, 2019, at 1:52 PM, Rahul Reddy <ra...@gmail.com> wrote:
>
> Schema matches and corruption errors in system.log
>
> On Fri, Jul 19, 2019, 1:33 PM Nitan Kainth <ni...@gmail.com> wrote:
>
>> Do you see schemat in sync? Nodetool describecluster.
>>
>> Check system log for any corruption.
>>
>>
>> Regards,
>>
>> Nitan
>>
>> Cell: 510 449 9629
>>
>> On Jul 19, 2019, at 12:32 PM, ZAIDI, ASAD A <az...@att.com> wrote:
>>
>> “aws asked to set nvme_timeout to higher number in etc/grub.conf.”
>>
>>
>>
>> Did you ask AWS if setting higher value is real solution to bug - Is
>> there not any patch available to address the bug? - just curios to know
>>
>>
>>
>> *From:* Rahul Reddy [mailto:rahulreddy1234@gmail.com
>> <ra...@gmail.com>]
>> *Sent:* Friday, July 19, 2019 10:49 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Rebooting one Cassandra node caused all the application nodes
>> go down
>>
>>
>>
>> Here ,
>>
>>
>>
>> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have
>> RF 3 and cl set to local quorum. And gossip snitch. All our instance are
>> c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
>> instance type had a bug which aws asked to set nvme_timeout to higher
>> number in etc/grub.conf. after setting the parameter and did run nodetool
>> drain and reboot the node in east
>>
>>
>>
>> Instance cameup but Cassandra didn't come up normal had to start the
>> Cassandra. Cassandra cameup but it shows other instances down. Even though
>> didn't reboot the other node down same was observed in one other node. How
>> could that happen and don't any errors in system.log which is set to info.
>>
>> Without any intervention gossip settled in 10 mins entire cluster became
>> normal.
>>
>>
>>
>> Tried same thing West it happened again
>>
>>
>>
>>
>>
>>
>>
>> I'm concerned how to check what caused it and if a reboot happens again
>> how to avoid this.
>>
>> If I just STOP Cassandra instead of reboot I don't see this issue.
>>
>>
>>
>>
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Nitan Kainth <ni...@gmail.com>.
You no corruption error or you see corruption error?
Regards,
Nitan
Cell: 510 449 9629
> On Jul 19, 2019, at 1:52 PM, Rahul Reddy <ra...@gmail.com> wrote:
>
> Schema matches and corruption errors in system.log
>
>> On Fri, Jul 19, 2019, 1:33 PM Nitan Kainth <ni...@gmail.com> wrote:
>> Do you see schemat in sync? Nodetool describecluster.
>>
>> Check system log for any corruption.
>>
>>
>> Regards,
>> Nitan
>> Cell: 510 449 9629
>>
>>> On Jul 19, 2019, at 12:32 PM, ZAIDI, ASAD A <az...@att.com> wrote:
>>>
>>> “aws asked to set nvme_timeout to higher number in etc/grub.conf.”
>>>
>>>
>>>
>>> Did you ask AWS if setting higher value is real solution to bug - Is there not any patch available to address the bug? - just curios to know
>>>
>>>
>>>
>>> From: Rahul Reddy [mailto:rahulreddy1234@gmail.com]
>>> Sent: Friday, July 19, 2019 10:49 AM
>>> To: user@cassandra.apache.org
>>> Subject: Rebooting one Cassandra node caused all the application nodes go down
>>>
>>>
>>>
>>> Here ,
>>>
>>>
>>>
>>> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have RF 3 and cl set to local quorum. And gossip snitch. All our instance are c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5 instance type had a bug which aws asked to set nvme_timeout to higher number in etc/grub.conf. after setting the parameter and did run nodetool drain and reboot the node in east
>>>
>>>
>>>
>>> Instance cameup but Cassandra didn't come up normal had to start the Cassandra. Cassandra cameup but it shows other instances down. Even though didn't reboot the other node down same was observed in one other node. How could that happen and don't any errors in system.log which is set to info.
>>>
>>> Without any intervention gossip settled in 10 mins entire cluster became normal.
>>>
>>>
>>>
>>> Tried same thing West it happened again
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> I'm concerned how to check what caused it and if a reboot happens again how to avoid this.
>>>
>>> If I just STOP Cassandra instead of reboot I don't see this issue.
>>>
>>>
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Rahul Reddy <ra...@gmail.com>.
Schema matches and corruption errors in system.log
On Fri, Jul 19, 2019, 1:33 PM Nitan Kainth <ni...@gmail.com> wrote:
> Do you see schemat in sync? Nodetool describecluster.
>
> Check system log for any corruption.
>
>
> Regards,
>
> Nitan
>
> Cell: 510 449 9629
>
> On Jul 19, 2019, at 12:32 PM, ZAIDI, ASAD A <az...@att.com> wrote:
>
> “aws asked to set nvme_timeout to higher number in etc/grub.conf.”
>
>
>
> Did you ask AWS if setting higher value is real solution to bug - Is there
> not any patch available to address the bug? - just curios to know
>
>
>
> *From:* Rahul Reddy [mailto:rahulreddy1234@gmail.com
> <ra...@gmail.com>]
> *Sent:* Friday, July 19, 2019 10:49 AM
> *To:* user@cassandra.apache.org
> *Subject:* Rebooting one Cassandra node caused all the application nodes
> go down
>
>
>
> Here ,
>
>
>
> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have
> RF 3 and cl set to local quorum. And gossip snitch. All our instance are
> c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
> instance type had a bug which aws asked to set nvme_timeout to higher
> number in etc/grub.conf. after setting the parameter and did run nodetool
> drain and reboot the node in east
>
>
>
> Instance cameup but Cassandra didn't come up normal had to start the
> Cassandra. Cassandra cameup but it shows other instances down. Even though
> didn't reboot the other node down same was observed in one other node. How
> could that happen and don't any errors in system.log which is set to info.
>
> Without any intervention gossip settled in 10 mins entire cluster became
> normal.
>
>
>
> Tried same thing West it happened again
>
>
>
>
>
>
>
> I'm concerned how to check what caused it and if a reboot happens again
> how to avoid this.
>
> If I just STOP Cassandra instead of reboot I don't see this issue.
>
>
>
>
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Nitan Kainth <ni...@gmail.com>.
Do you see schemat in sync? Nodetool describecluster.
Check system log for any corruption.
Regards,
Nitan
Cell: 510 449 9629
> On Jul 19, 2019, at 12:32 PM, ZAIDI, ASAD A <az...@att.com> wrote:
>
> “aws asked to set nvme_timeout to higher number in etc/grub.conf.”
>
> Did you ask AWS if setting higher value is real solution to bug - Is there not any patch available to address the bug? - just curios to know
>
> From: Rahul Reddy [mailto:rahulreddy1234@gmail.com]
> Sent: Friday, July 19, 2019 10:49 AM
> To: user@cassandra.apache.org
> Subject: Rebooting one Cassandra node caused all the application nodes go down
>
> Here ,
>
> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have RF 3 and cl set to local quorum. And gossip snitch. All our instance are c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5 instance type had a bug which aws asked to set nvme_timeout to higher number in etc/grub.conf. after setting the parameter and did run nodetool drain and reboot the node in east
>
> Instance cameup but Cassandra didn't come up normal had to start the Cassandra. Cassandra cameup but it shows other instances down. Even though didn't reboot the other node down same was observed in one other node. How could that happen and don't any errors in system.log which is set to info.
> Without any intervention gossip settled in 10 mins entire cluster became normal.
>
> Tried same thing West it happened again
>
>
>
> I'm concerned how to check what caused it and if a reboot happens again how to avoid this.
> If I just STOP Cassandra instead of reboot I don't see this issue.
>
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Rajsekhar Mallick <ra...@gmail.com>.
Hello Rahul,
As per your description, Cassandra process is up and running as you
verified from the logs.
But nodetool and grafana arnt fetching data.
This points to the suspect being jmx port 7199.
Do run and check 'netstat -anp | egrep"7199|9042|7070" ' on the impacted
and other hosts in the cluster.
There has to be some difference . Observe The ip address to which the jmx
port 7199 is binding to. Is it the same as it was prior to reboot.
Thanks
On Fri, 19 Jul, 2019, 10:28 PM Rahul Reddy, <ra...@gmail.com>
wrote:
> Raj,
>
> No that was not the case in system.log I see the started listening to call
> client at 16:42 but some how it still unreachable to 16:50 below grafana
> dashboard shows it. Once everything up in logs why would it still show down
> in nodetool status and grafana.
>
> Zaidi,
>
> In latest aws Linux Ami they took care of this bug . And also changing the
> Ami needs rebuild of all the nodes so didn't took that route.
>
> On Fri, Jul 19, 2019, 12:32 PM ZAIDI, ASAD A <az...@att.com> wrote:
>
>> “aws asked to set nvme_timeout to higher number in etc/grub.conf.”
>>
>>
>>
>> Did you ask AWS if setting higher value is real solution to bug - Is
>> there not any patch available to address the bug? - just curios to know
>>
>>
>>
>> *From:* Rahul Reddy [mailto:rahulreddy1234@gmail.com]
>> *Sent:* Friday, July 19, 2019 10:49 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Rebooting one Cassandra node caused all the application nodes
>> go down
>>
>>
>>
>> Here ,
>>
>>
>>
>> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have
>> RF 3 and cl set to local quorum. And gossip snitch. All our instance are
>> c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
>> instance type had a bug which aws asked to set nvme_timeout to higher
>> number in etc/grub.conf. after setting the parameter and did run nodetool
>> drain and reboot the node in east
>>
>>
>>
>> Instance cameup but Cassandra didn't come up normal had to start the
>> Cassandra. Cassandra cameup but it shows other instances down. Even though
>> didn't reboot the other node down same was observed in one other node. How
>> could that happen and don't any errors in system.log which is set to info.
>>
>> Without any intervention gossip settled in 10 mins entire cluster became
>> normal.
>>
>>
>>
>> Tried same thing West it happened again
>>
>>
>>
>>
>>
>>
>>
>> I'm concerned how to check what caused it and if a reboot happens again
>> how to avoid this.
>>
>> If I just STOP Cassandra instead of reboot I don't see this issue.
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
RE: Rebooting one Cassandra node caused all the application nodes
go down
Posted by "ZAIDI, ASAD A" <az...@att.com>.
https://lore.kernel.org/patchwork/patch/884501/
according to this link , it sound like having patched kernel is real solution to bug
From: Rahul Reddy [mailto:rahulreddy1234@gmail.com]
Sent: Friday, July 19, 2019 11:48 AM
To: user@cassandra.apache.org
Subject: Re: Rebooting one Cassandra node caused all the application nodes go down
Raj,
No that was not the case in system.log I see the started listening to call client at 16:42 but some how it still unreachable to 16:50 below grafana dashboard shows it. Once everything up in logs why would it still show down in nodetool status and grafana.
Zaidi,
In latest aws Linux Ami they took care of this bug . And also changing the Ami needs rebuild of all the nodes so didn't took that route.
On Fri, Jul 19, 2019, 12:32 PM ZAIDI, ASAD A <az...@att.com>> wrote:
“aws asked to set nvme_timeout to higher number in etc/grub.conf.”
Did you ask AWS if setting higher value is real solution to bug - Is there not any patch available to address the bug? - just curios to know
From: Rahul Reddy [mailto:rahulreddy1234@gmail.com<ma...@gmail.com>]
Sent: Friday, July 19, 2019 10:49 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Rebooting one Cassandra node caused all the application nodes go down
Here ,
We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have RF 3 and cl set to local quorum. And gossip snitch. All our instance are c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5 instance type had a bug which aws asked to set nvme_timeout to higher number in etc/grub.conf. after setting the parameter and did run nodetool drain and reboot the node in east
Instance cameup but Cassandra didn't come up normal had to start the Cassandra. Cassandra cameup but it shows other instances down. Even though didn't reboot the other node down same was observed in one other node. How could that happen and don't any errors in system.log which is set to info.
Without any intervention gossip settled in 10 mins entire cluster became normal.
Tried same thing West it happened again
I'm concerned how to check what caused it and if a reboot happens again how to avoid this.
If I just STOP Cassandra instead of reboot I don't see this issue.
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Rahul Reddy <ra...@gmail.com>.
Raj,
No that was not the case in system.log I see the started listening to call
client at 16:42 but some how it still unreachable to 16:50 below grafana
dashboard shows it. Once everything up in logs why would it still show down
in nodetool status and grafana.
Zaidi,
In latest aws Linux Ami they took care of this bug . And also changing the
Ami needs rebuild of all the nodes so didn't took that route.
On Fri, Jul 19, 2019, 12:32 PM ZAIDI, ASAD A <az...@att.com> wrote:
> “aws asked to set nvme_timeout to higher number in etc/grub.conf.”
>
>
>
> Did you ask AWS if setting higher value is real solution to bug - Is there
> not any patch available to address the bug? - just curios to know
>
>
>
> *From:* Rahul Reddy [mailto:rahulreddy1234@gmail.com]
> *Sent:* Friday, July 19, 2019 10:49 AM
> *To:* user@cassandra.apache.org
> *Subject:* Rebooting one Cassandra node caused all the application nodes
> go down
>
>
>
> Here ,
>
>
>
> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have
> RF 3 and cl set to local quorum. And gossip snitch. All our instance are
> c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
> instance type had a bug which aws asked to set nvme_timeout to higher
> number in etc/grub.conf. after setting the parameter and did run nodetool
> drain and reboot the node in east
>
>
>
> Instance cameup but Cassandra didn't come up normal had to start the
> Cassandra. Cassandra cameup but it shows other instances down. Even though
> didn't reboot the other node down same was observed in one other node. How
> could that happen and don't any errors in system.log which is set to info.
>
> Without any intervention gossip settled in 10 mins entire cluster became
> normal.
>
>
>
> Tried same thing West it happened again
>
>
>
>
>
>
>
> I'm concerned how to check what caused it and if a reboot happens again
> how to avoid this.
>
> If I just STOP Cassandra instead of reboot I don't see this issue.
>
>
>
RE: Rebooting one Cassandra node caused all the application nodes
go down
Posted by "ZAIDI, ASAD A" <az...@att.com>.
“aws asked to set nvme_timeout to higher number in etc/grub.conf.”
Did you ask AWS if setting higher value is real solution to bug - Is there not any patch available to address the bug? - just curios to know
From: Rahul Reddy [mailto:rahulreddy1234@gmail.com]
Sent: Friday, July 19, 2019 10:49 AM
To: user@cassandra.apache.org
Subject: Rebooting one Cassandra node caused all the application nodes go down
Here ,
We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have RF 3 and cl set to local quorum. And gossip snitch. All our instance are c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5 instance type had a bug which aws asked to set nvme_timeout to higher number in etc/grub.conf. after setting the parameter and did run nodetool drain and reboot the node in east
Instance cameup but Cassandra didn't come up normal had to start the Cassandra. Cassandra cameup but it shows other instances down. Even though didn't reboot the other node down same was observed in one other node. How could that happen and don't any errors in system.log which is set to info.
Without any intervention gossip settled in 10 mins entire cluster became normal.
Tried same thing West it happened again
I'm concerned how to check what caused it and if a reboot happens again how to avoid this.
If I just STOP Cassandra instead of reboot I don't see this issue.
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Jeff Jirsa <jj...@gmail.com>.
Could be something like
https://issues.apache.org/jira/browse/CASSANDRA-14358
Hard to say after the fact.
On Fri, Jul 19, 2019 at 8:49 AM Rahul Reddy <ra...@gmail.com>
wrote:
> Here ,
>
> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have
> RF 3 and cl set to local quorum. And gossip snitch. All our instance are
> c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
> instance type had a bug which aws asked to set nvme_timeout to higher
> number in etc/grub.conf. after setting the parameter and did run nodetool
> drain and reboot the node in east
>
> Instance cameup but Cassandra didn't come up normal had to start the
> Cassandra. Cassandra cameup but it shows other instances down. Even though
> didn't reboot the other node down same was observed in one other node. How
> could that happen and don't any errors in system.log which is set to info.
> Without any intervention gossip settled in 10 mins entire cluster became
> normal.
>
> Tried same thing West it happened again
>
>
>
> I'm concerned how to check what caused it and if a reboot happens again
> how to avoid this.
> If I just STOP Cassandra instead of reboot I don't see this issue.
>
>
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Rajsekhar Mallick <ra...@gmail.com>.
Hello Rahul,
Basically the issue is running nodetool status on the host rebooted node,
shows itself as UN and all other nodes in the cluster as DN.
And running nodetool status on any other node in the cluster shows this
rebooted node as DN.
Correct me if I am wrong. Is this the issue.
Also attach screenshot of the observation you are talking about. You may
choose to replace the ip address of the hosts
Thanks
On Fri, 19 Jul, 2019, 9:36 PM Rahul Reddy, <ra...@gmail.com> wrote:
> Thanks for quick response rajshekar.
>
> Correct same cassandra.yml and same java
>
> On Fri, Jul 19, 2019, 11:56 AM Rajsekhar Mallick <ra...@gmail.com>
> wrote:
>
>> Hello Rahul,
>>
>> May you please confirm on below things.
>>
>> 1. Cassandra.yaml file of the node which was started after the machine
>> reboot is same as that of rest of the nodes in the cluster.
>> 2. Java version is consistent across all nodes in the cluster.
>>
>> Do check and revert
>>
>> Thanks
>>
>> On Fri, 19 Jul, 2019, 9:19 PM Rahul Reddy, <ra...@gmail.com>
>> wrote:
>>
>>> Here ,
>>>
>>> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We
>>> have RF 3 and cl set to local quorum. And gossip snitch. All our instance
>>> are c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
>>> instance type had a bug which aws asked to set nvme_timeout to higher
>>> number in etc/grub.conf. after setting the parameter and did run nodetool
>>> drain and reboot the node in east
>>>
>>> Instance cameup but Cassandra didn't come up normal had to start the
>>> Cassandra. Cassandra cameup but it shows other instances down. Even though
>>> didn't reboot the other node down same was observed in one other node. How
>>> could that happen and don't any errors in system.log which is set to info.
>>> Without any intervention gossip settled in 10 mins entire cluster became
>>> normal.
>>>
>>> Tried same thing West it happened again
>>>
>>>
>>>
>>> I'm concerned how to check what caused it and if a reboot happens again
>>> how to avoid this.
>>> If I just STOP Cassandra instead of reboot I don't see this issue.
>>>
>>>
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Rahul Reddy <ra...@gmail.com>.
Thanks for quick response rajshekar.
Correct same cassandra.yml and same java
On Fri, Jul 19, 2019, 11:56 AM Rajsekhar Mallick <ra...@gmail.com>
wrote:
> Hello Rahul,
>
> May you please confirm on below things.
>
> 1. Cassandra.yaml file of the node which was started after the machine
> reboot is same as that of rest of the nodes in the cluster.
> 2. Java version is consistent across all nodes in the cluster.
>
> Do check and revert
>
> Thanks
>
> On Fri, 19 Jul, 2019, 9:19 PM Rahul Reddy, <ra...@gmail.com>
> wrote:
>
>> Here ,
>>
>> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have
>> RF 3 and cl set to local quorum. And gossip snitch. All our instance are
>> c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
>> instance type had a bug which aws asked to set nvme_timeout to higher
>> number in etc/grub.conf. after setting the parameter and did run nodetool
>> drain and reboot the node in east
>>
>> Instance cameup but Cassandra didn't come up normal had to start the
>> Cassandra. Cassandra cameup but it shows other instances down. Even though
>> didn't reboot the other node down same was observed in one other node. How
>> could that happen and don't any errors in system.log which is set to info.
>> Without any intervention gossip settled in 10 mins entire cluster became
>> normal.
>>
>> Tried same thing West it happened again
>>
>>
>>
>> I'm concerned how to check what caused it and if a reboot happens again
>> how to avoid this.
>> If I just STOP Cassandra instead of reboot I don't see this issue.
>>
>>
Re: Rebooting one Cassandra node caused all the application nodes go down
Posted by Rajsekhar Mallick <ra...@gmail.com>.
Hello Rahul,
May you please confirm on below things.
1. Cassandra.yaml file of the node which was started after the machine
reboot is same as that of rest of the nodes in the cluster.
2. Java version is consistent across all nodes in the cluster.
Do check and revert
Thanks
On Fri, 19 Jul, 2019, 9:19 PM Rahul Reddy, <ra...@gmail.com> wrote:
> Here ,
>
> We have 6 nodes each in 2 data centers us-east-1 and us-west-2 . We have
> RF 3 and cl set to local quorum. And gossip snitch. All our instance are
> c5.2xlarge and data files and comit logs are stored in gp2 ebs. C5
> instance type had a bug which aws asked to set nvme_timeout to higher
> number in etc/grub.conf. after setting the parameter and did run nodetool
> drain and reboot the node in east
>
> Instance cameup but Cassandra didn't come up normal had to start the
> Cassandra. Cassandra cameup but it shows other instances down. Even though
> didn't reboot the other node down same was observed in one other node. How
> could that happen and don't any errors in system.log which is set to info.
> Without any intervention gossip settled in 10 mins entire cluster became
> normal.
>
> Tried same thing West it happened again
>
>
>
> I'm concerned how to check what caused it and if a reboot happens again
> how to avoid this.
> If I just STOP Cassandra instead of reboot I don't see this issue.
>
>