You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Peddi, Praveen" <pe...@amazon.com> on 2016/03/02 17:10:47 UTC
Removing Node causes bunch of HostUnavailableException
Hi all,,
We have few dead nodes in the cluster (Amazon ASG removed those thinking there is an issue with health). Now we are trying to remove those dead nodes from the cluster so that other nodes can take over. As soon as I execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions both on reads and writes. What I am not able to understand is, these are deadnodes and don't even physically exists. Why would removenode command cause any outage of nodes in Cassandra when we had no errors whatsoever before removing them. I could not really find a jira ticket for this.
We are using Cassandra 2.0.9. Any information would be greatly appreciated.
Thanks
Praveen
Re: Removing Node causes bunch of HostUnavailableException
Posted by "Peddi, Praveen" <pe...@amazon.com>.
Hi Alain,
Sorry I completely missed your email until my colleague pointed it out.
>From the testing we have done so far, We still have this issue when removing nodes on 2.0.9 but not on 2.2.4. We will be upgrading to 2.2.4 pretty soon so I am not too worried about errors in 2.0.9. Since we haven’t changed any code on our side between 2.0.9 and 2.2.4, I am guessing this is probably a bug in 2.0.9 and since 2.0.9 is not supported anymore, I didn’t create a jira ticket for it.
Thanks for following up though.
Praveen
From: Alain RODRIGUEZ <ar...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Thursday, March 10, 2016 at 5:30 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Removing Node causes bunch of HostUnavailableException
Hi Praveen, how is this going ?
I have been out for a while, did you manage to remove the nodes ? Do you need more help ? If so, I could use a status update and more information about the remaining issues.
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com<ma...@thelastpickle.com>
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-03-04 19:39 GMT+01:00 Peddi, Praveen <pe...@amazon.com>>:
Hi Jack,
My answers below…
What is the exact exception you are getting and where do you get it? Is it UnavailableException or NoHostAvailableException and does it occur on the client, using the Java driver?
We saw different types of exceptions. One I could quickly grep are:
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency SERIAL (2 replica were required but only 1 acknowledged the write)
com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)
QueryTimeoutException
What is your LoadBalancingPolicy?
new TokenAwarePolicy(new RoundRobinPolicy()))
What consistency level is the client using?
QUORUM for reads. For writes some APIs use SERIAL and some use QUORUM dependending if we want to do optimistic locking,
What retry policy is the client using?
Default Retry Policy
When you say that the failures don't last for more than a few minutes, you mean from the moment you perform the nodetool removenode? And is operation completely normal after those few minutes?
That is correct. All operations recover from failures after few minutes.
-- Jack Krupansky
On Thu, Mar 3, 2016 at 4:40 PM, Peddi, Praveen <pe...@amazon.com>> wrote:
Hi Jack,
Which node(s) were getting the HostNotAvailable errors - all nodes for every query, or just a small portion of the nodes on some queries?
Not all read/writes are failing with Unavalable or Timeout exception. Writes failures were around 10% of total calls. Reads were little worse (as worse as 35% of total calls).
It may take some time for the gossip state to propagate; maybe some of it is corrupted or needs a full refresh.
Were any of the seed nodes in the collection of nodes that were removed? How many seed nodes does each node typically have?
We currently use all hosts as seed hosts which I know is a very bad idea and we are going to fix that soon. The reason we use all hosts as seed hosts is because these hosts can get recycled for many reasons and we didn’t want to hard code the host names so we programmatically get host names (we wrote our own seed host provider). Could that be the reason for these failures? If a dead node is in the seed nodes list and we try to remove that node, could that lead to blip of failures. The failures don’t last for more than few minutes.
-- Jack Krupansky
On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen <pe...@amazon.com>> wrote:
Thanks Alain for quick and detailed response. My answers inline. One thing I want to clarify is, the nodes got recycled due to some automatic health check failure. This means old nodes are dead and new nodes got added w/o our intervention. So replacing nodes would not work for us since the new nodes were already added.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM)
Do you use at leat 3 distinct AZ ? If so, you should indeed be fine regarding data integrity. Also repair should then work for you. If you have less than 3 AZ, then you are in troubles...
Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8 nodes were recycled, there were absolutely no outage on Cassandra (other two nodes wtill satisfy quorum consistency)
About the unreachable errors, I believe it can be due to the overload due to the missing nodes. Pressure on the remaining node might be too strong.
It is certainly possible but we have beefed up cluster with <3% CPU, hardly any network I/o and disk usage. We have 162 nodes in the cluster and each node doesn’t have more than 80 to 100MB of data.
However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn’t make any sense because I am just removing a node that doesn’t even exist.
This probably added even more load, if you are using vnodes, all the remaining nodes probably started streaming data to each other node at the speed of "nodetool getstreamthroughput". AWS network isn't that good, and is probably saturated. Also have you the phi_convict_threshold configured to a high value at least 10 or 12 ? This would avoid nodes to be marked down that often.
We are using c3.2xlarge which has good network throughput (1GB/sec I think). We are using default value which is 200MB/sec in 2.0.9. We will play with it in future and see if this could make any difference but as I mentioned the data size on each node is not huge.
Regarding phi_convict_threshold, our Cassandra is not bringing itself down. There was a bug in health check from one of our internal tool and that tool is recycling the nodes. Nothing to do with Cassandra health. Again we will keep an eye on it in future.
What does "nodetool tpstats" outputs ?
Nodetool tpstats on which node? Any node?
Also you might try to monitor resources and see what happens (my guess is you should focus at iowait, disk usage and network, have an eye at cpu too).
We did monitor cpu, disk and network and they are all very low.
A quick fix would probably be to hardly throttle the network on all the nodes and see if it helps:
nodetool setstreamthroughput 2
We will play with this config. 2.0.9 defaults to 200MB/sec which I think is too high.
If this work, you could incrementally increase it and monitor, find the good tuning and put it the cassandra.yaml.
I opened a ticket a while ago about that issue: https://issues.apache.org/jira/browse/CASSANDRA-9509
I voted for this issue. Lets see if it gets picked up :).
I hope this will help you to go back to a healthy state allowing you a fast upgrade ;-).
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com<ma...@thelastpickle.com>
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>>:
Hi Robert,
Thanks for your response.
Replication factor is 3.
We are in the process of upgrading to 2.2.4. We have had too many performance issues with later versions of Cassandra (I have asked asked for help related to that in the forum). We are close to getting to similar performance now and hopefully upgrade in next few weeks. Lot of testing to do :(.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM). However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn’t make any sense because I am just removing a node that doesn’t even exist.
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, March 2, 2016 at 2:52 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Removing Node causes bunch of HostUnavailableException
On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com>> wrote:
We have few dead nodes in the cluster (Amazon ASG removed those thinking there is an issue with health). Now we are trying to remove those dead nodes from the cluster so that other nodes can take over. As soon as I execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions both on reads and writes. What I am not able to understand is, these are deadnodes and don’t even physically exists. Why would removenode command cause any outage of nodes in Cassandra when we had no errors whatsoever before removing them. I could not really find a jira ticket for this.
What is your replication factor?
Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
Also, removing multiple nodes with removenode means your consistency is pretty hosed. Repair ASAP, but there are potential cases where repair won't help.
=Rob
=Rob
Re: Removing Node causes bunch of HostUnavailableException
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi Praveen, how is this going ?
I have been out for a while, did you manage to remove the nodes ? Do you
need more help ? If so, I could use a status update and more information
about the remaining issues.
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-03-04 19:39 GMT+01:00 Peddi, Praveen <pe...@amazon.com>:
> Hi Jack,
> My answers below…
>
> What is the exact exception you are getting and where do you get it? Is it
> UnavailableException or NoHostAvailableException and does it occur on the
> client, using the Java driver?
>
> We saw different types of exceptions. One I could quickly grep are:
> com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra
> timeout during write query at consistency SERIAL (2 replica were required
> but only 1 acknowledged the write)
> com.datastax.driver.core.exceptions.UnavailableException: Not enough
> replica available for query at consistency QUORUM (2 required but only 1
> alive)
> QueryTimeoutException
>
>
>
> What is your LoadBalancingPolicy?
>
> new TokenAwarePolicy(new RoundRobinPolicy()))
>
>
> What consistency level is the client using?
>
> QUORUM for reads. For writes some APIs use SERIAL and some use QUORUM
> dependending if we want to do optimistic locking,
>
>
> What retry policy is the client using?
>
> Default Retry Policy
>
>
> When you say that the failures don't last for more than a few minutes, you
> mean from the moment you perform the nodetool removenode? And is operation
> completely normal after those few minutes?
>
> That is correct. All operations recover from failures after few minutes.
>
>
>
> -- Jack Krupansky
>
> On Thu, Mar 3, 2016 at 4:40 PM, Peddi, Praveen <pe...@amazon.com> wrote:
>
>> Hi Jack,
>>
>> Which node(s) were getting the HostNotAvailable errors - all nodes for
>> every query, or just a small portion of the nodes on some queries?
>>
>> Not all read/writes are failing with Unavalable or Timeout exception.
>> Writes failures were around 10% of total calls. Reads were little worse (as
>> worse as 35% of total calls).
>>
>>
>> It may take some time for the gossip state to propagate; maybe some of it
>> is corrupted or needs a full refresh.
>>
>> Were any of the seed nodes in the collection of nodes that were removed?
>> How many seed nodes does each node typically have?
>>
>> We currently use all hosts as seed hosts which I know is a very bad idea
>> and we are going to fix that soon. The reason we use all hosts as seed
>> hosts is because these hosts can get recycled for many reasons and we
>> didn’t want to hard code the host names so we programmatically get host
>> names (we wrote our own seed host provider). Could that be the reason for
>> these failures? If a dead node is in the seed nodes list and we try to
>> remove that node, could that lead to blip of failures. The failures don’t
>> last for more than few minutes.
>>
>>
>>
>> -- Jack Krupansky
>>
>> On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen <pe...@amazon.com> wrote:
>>
>>> Thanks Alain for quick and detailed response. My answers inline. One
>>> thing I want to clarify is, the nodes got recycled due to some automatic
>>> health check failure. This means old nodes are dead and new nodes got added
>>> w/o our intervention. So replacing nodes would not work for us since the
>>> new nodes were already added.
>>>
>>>
>>>
>>>> We are not removing multiple nodes at the same time. All dead nodes are
>>>> from same AZ so there were no errors when the nodes were down as expected
>>>> (because we use QUORUM)
>>>
>>>
>>> Do you use at leat 3 distinct AZ ? If so, you should indeed be fine
>>> regarding data integrity. Also repair should then work for you. If you have
>>> less than 3 AZ, then you are in troubles...
>>>
>>> Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8
>>> nodes were recycled, there were absolutely no outage on Cassandra (other
>>> two nodes wtill satisfy quorum consistency)
>>>
>>>
>>> About the unreachable errors, I believe it can be due to the overload
>>> due to the missing nodes. Pressure on the remaining node might be too
>>> strong.
>>>
>>> It is certainly possible but we have beefed up cluster with <3% CPU,
>>> hardly any network I/o and disk usage. We have 162 nodes in the cluster and
>>> each node doesn’t have more than 80 to 100MB of data.
>>>
>>>
>>>
>>> However, As soon as I started removing nodes one by one, every time time
>>>> we see lot of timeout and unavailable exceptions which doesn’t make any
>>>> sense because I am just removing a node that doesn’t even exist.
>>>>
>>>
>>> This probably added even more load, if you are using vnodes, all the
>>> remaining nodes probably started streaming data to each other node at the
>>> speed of "nodetool getstreamthroughput". AWS network isn't that good, and
>>> is probably saturated. Also have you the phi_convict_threshold configured
>>> to a high value at least 10 or 12 ? This would avoid nodes to be marked
>>> down that often.
>>>
>>> We are using c3.2xlarge which has good network throughput (1GB/sec I
>>> think). We are using default value which is 200MB/sec in 2.0.9. We will
>>> play with it in future and see if this could make any difference but as I
>>> mentioned the data size on each node is not huge.
>>> Regarding phi_convict_threshold, our Cassandra is not bringing itself
>>> down. There was a bug in health check from one of our internal tool and
>>> that tool is recycling the nodes. Nothing to do with Cassandra health.
>>> Again we will keep an eye on it in future.
>>>
>>>
>>> What does "nodetool tpstats" outputs ?
>>>
>>> Nodetool tpstats on which node? Any node?
>>>
>>>
>>> Also you might try to monitor resources and see what happens (my guess
>>> is you should focus at iowait, disk usage and network, have an eye at cpu
>>> too).
>>>
>>> We did monitor cpu, disk and network and they are all very low.
>>>
>>>
>>> A quick fix would probably be to hardly throttle the network on all the
>>> nodes and see if it helps:
>>>
>>> nodetool setstreamthroughput 2
>>>
>>> We will play with this config. 2.0.9 defaults to 200MB/sec which I think
>>> is too high.
>>>
>>>
>>> If this work, you could incrementally increase it and monitor, find the
>>> good tuning and put it the cassandra.yaml.
>>>
>>> I opened a ticket a while ago about that issue:
>>> https://issues.apache.org/jira/browse/CASSANDRA-9509
>>>
>>> I voted for this issue. Lets see if it gets picked up :).
>>>
>>>
>>> I hope this will help you to go back to a healthy state allowing you a
>>> fast upgrade ;-).
>>>
>>> C*heers,
>>> -----------------------
>>> Alain Rodriguez - alain@thelastpickle.com
>>> France
>>>
>>> The Last Pickle - Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>> 2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>:
>>>
>>>> Hi Robert,
>>>> Thanks for your response.
>>>>
>>>> Replication factor is 3.
>>>>
>>>> We are in the process of upgrading to 2.2.4. We have had too many
>>>> performance issues with later versions of Cassandra (I have asked asked for
>>>> help related to that in the forum). We are close to getting to similar
>>>> performance now and hopefully upgrade in next few weeks. Lot of testing to
>>>> do :(.
>>>>
>>>> We are not removing multiple nodes at the same time. All dead nodes are
>>>> from same AZ so there were no errors when the nodes were down as expected
>>>> (because we use QUORUM). However, As soon as I started removing nodes one
>>>> by one, every time time we see lot of timeout and unavailable exceptions
>>>> which doesn’t make any sense because I am just removing a node that doesn’t
>>>> even exist.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: Robert Coli <rc...@eventbrite.com>
>>>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>>>> Date: Wednesday, March 2, 2016 at 2:52 PM
>>>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>>>> Subject: Re: Removing Node causes bunch of HostUnavailableException
>>>>
>>>> On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com>
>>>> wrote:
>>>>
>>>>> We have few dead nodes in the cluster (Amazon ASG removed those
>>>>> thinking there is an issue with health). Now we are trying to remove those
>>>>> dead nodes from the cluster so that other nodes can take over. As soon as I
>>>>> execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions
>>>>> both on reads and writes. What I am not able to understand is, these are
>>>>> deadnodes and don’t even physically exists. Why would removenode command
>>>>> cause any outage of nodes in Cassandra when we had no errors whatsoever
>>>>> before removing them. I could not really find a jira ticket for this.
>>>>>
>>>>
>>>> What is your replication factor?
>>>>
>>>> Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
>>>>
>>>> Also, removing multiple nodes with removenode means your consistency is
>>>> pretty hosed. Repair ASAP, but there are potential cases where repair won't
>>>> help.
>>>>
>>>> =Rob
>>>>
>>>>
>>>> =Rob
>>>>
>>>>
>>>
>>>
>>
>
Re: Removing Node causes bunch of HostUnavailableException
Posted by "Peddi, Praveen" <pe...@amazon.com>.
Hi Jack,
My answers below…
What is the exact exception you are getting and where do you get it? Is it UnavailableException or NoHostAvailableException and does it occur on the client, using the Java driver?
We saw different types of exceptions. One I could quickly grep are:
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency SERIAL (2 replica were required but only 1 acknowledged the write)
com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency QUORUM (2 required but only 1 alive)
QueryTimeoutException
What is your LoadBalancingPolicy?
new TokenAwarePolicy(new RoundRobinPolicy()))
What consistency level is the client using?
QUORUM for reads. For writes some APIs use SERIAL and some use QUORUM dependending if we want to do optimistic locking,
What retry policy is the client using?
Default Retry Policy
When you say that the failures don't last for more than a few minutes, you mean from the moment you perform the nodetool removenode? And is operation completely normal after those few minutes?
That is correct. All operations recover from failures after few minutes.
-- Jack Krupansky
On Thu, Mar 3, 2016 at 4:40 PM, Peddi, Praveen <pe...@amazon.com>> wrote:
Hi Jack,
Which node(s) were getting the HostNotAvailable errors - all nodes for every query, or just a small portion of the nodes on some queries?
Not all read/writes are failing with Unavalable or Timeout exception. Writes failures were around 10% of total calls. Reads were little worse (as worse as 35% of total calls).
It may take some time for the gossip state to propagate; maybe some of it is corrupted or needs a full refresh.
Were any of the seed nodes in the collection of nodes that were removed? How many seed nodes does each node typically have?
We currently use all hosts as seed hosts which I know is a very bad idea and we are going to fix that soon. The reason we use all hosts as seed hosts is because these hosts can get recycled for many reasons and we didn’t want to hard code the host names so we programmatically get host names (we wrote our own seed host provider). Could that be the reason for these failures? If a dead node is in the seed nodes list and we try to remove that node, could that lead to blip of failures. The failures don’t last for more than few minutes.
-- Jack Krupansky
On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen <pe...@amazon.com>> wrote:
Thanks Alain for quick and detailed response. My answers inline. One thing I want to clarify is, the nodes got recycled due to some automatic health check failure. This means old nodes are dead and new nodes got added w/o our intervention. So replacing nodes would not work for us since the new nodes were already added.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM)
Do you use at leat 3 distinct AZ ? If so, you should indeed be fine regarding data integrity. Also repair should then work for you. If you have less than 3 AZ, then you are in troubles...
Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8 nodes were recycled, there were absolutely no outage on Cassandra (other two nodes wtill satisfy quorum consistency)
About the unreachable errors, I believe it can be due to the overload due to the missing nodes. Pressure on the remaining node might be too strong.
It is certainly possible but we have beefed up cluster with <3% CPU, hardly any network I/o and disk usage. We have 162 nodes in the cluster and each node doesn’t have more than 80 to 100MB of data.
However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn’t make any sense because I am just removing a node that doesn’t even exist.
This probably added even more load, if you are using vnodes, all the remaining nodes probably started streaming data to each other node at the speed of "nodetool getstreamthroughput". AWS network isn't that good, and is probably saturated. Also have you the phi_convict_threshold configured to a high value at least 10 or 12 ? This would avoid nodes to be marked down that often.
We are using c3.2xlarge which has good network throughput (1GB/sec I think). We are using default value which is 200MB/sec in 2.0.9. We will play with it in future and see if this could make any difference but as I mentioned the data size on each node is not huge.
Regarding phi_convict_threshold, our Cassandra is not bringing itself down. There was a bug in health check from one of our internal tool and that tool is recycling the nodes. Nothing to do with Cassandra health. Again we will keep an eye on it in future.
What does "nodetool tpstats" outputs ?
Nodetool tpstats on which node? Any node?
Also you might try to monitor resources and see what happens (my guess is you should focus at iowait, disk usage and network, have an eye at cpu too).
We did monitor cpu, disk and network and they are all very low.
A quick fix would probably be to hardly throttle the network on all the nodes and see if it helps:
nodetool setstreamthroughput 2
We will play with this config. 2.0.9 defaults to 200MB/sec which I think is too high.
If this work, you could incrementally increase it and monitor, find the good tuning and put it the cassandra.yaml.
I opened a ticket a while ago about that issue: https://issues.apache.org/jira/browse/CASSANDRA-9509
I voted for this issue. Lets see if it gets picked up :).
I hope this will help you to go back to a healthy state allowing you a fast upgrade ;-).
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com<ma...@thelastpickle.com>
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>>:
Hi Robert,
Thanks for your response.
Replication factor is 3.
We are in the process of upgrading to 2.2.4. We have had too many performance issues with later versions of Cassandra (I have asked asked for help related to that in the forum). We are close to getting to similar performance now and hopefully upgrade in next few weeks. Lot of testing to do :(.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM). However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn’t make any sense because I am just removing a node that doesn’t even exist.
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, March 2, 2016 at 2:52 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Removing Node causes bunch of HostUnavailableException
On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com>> wrote:
We have few dead nodes in the cluster (Amazon ASG removed those thinking there is an issue with health). Now we are trying to remove those dead nodes from the cluster so that other nodes can take over. As soon as I execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions both on reads and writes. What I am not able to understand is, these are deadnodes and don’t even physically exists. Why would removenode command cause any outage of nodes in Cassandra when we had no errors whatsoever before removing them. I could not really find a jira ticket for this.
What is your replication factor?
Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
Also, removing multiple nodes with removenode means your consistency is pretty hosed. Repair ASAP, but there are potential cases where repair won't help.
=Rob
=Rob
Re: Removing Node causes bunch of HostUnavailableException
Posted by Jack Krupansky <ja...@gmail.com>.
What is the exact exception you are getting and where do you get it? Is it
UnavailableException or NoHostAvailableException and does it occur on the
client, using the Java driver?
What is your LoadBalancingPolicy?
What consistency level is the client using?
What retry policy is the client using?
When you say that the failures don't last for more than a few minutes, you
mean from the moment you perform the nodetool removenode? And is operation
completely normal after those few minutes?
-- Jack Krupansky
On Thu, Mar 3, 2016 at 4:40 PM, Peddi, Praveen <pe...@amazon.com> wrote:
> Hi Jack,
>
> Which node(s) were getting the HostNotAvailable errors - all nodes for
> every query, or just a small portion of the nodes on some queries?
>
> Not all read/writes are failing with Unavalable or Timeout exception.
> Writes failures were around 10% of total calls. Reads were little worse (as
> worse as 35% of total calls).
>
>
> It may take some time for the gossip state to propagate; maybe some of it
> is corrupted or needs a full refresh.
>
> Were any of the seed nodes in the collection of nodes that were removed?
> How many seed nodes does each node typically have?
>
> We currently use all hosts as seed hosts which I know is a very bad idea
> and we are going to fix that soon. The reason we use all hosts as seed
> hosts is because these hosts can get recycled for many reasons and we
> didn’t want to hard code the host names so we programmatically get host
> names (we wrote our own seed host provider). Could that be the reason for
> these failures? If a dead node is in the seed nodes list and we try to
> remove that node, could that lead to blip of failures. The failures don’t
> last for more than few minutes.
>
>
>
> -- Jack Krupansky
>
> On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen <pe...@amazon.com> wrote:
>
>> Thanks Alain for quick and detailed response. My answers inline. One
>> thing I want to clarify is, the nodes got recycled due to some automatic
>> health check failure. This means old nodes are dead and new nodes got added
>> w/o our intervention. So replacing nodes would not work for us since the
>> new nodes were already added.
>>
>>
>>
>>> We are not removing multiple nodes at the same time. All dead nodes are
>>> from same AZ so there were no errors when the nodes were down as expected
>>> (because we use QUORUM)
>>
>>
>> Do you use at leat 3 distinct AZ ? If so, you should indeed be fine
>> regarding data integrity. Also repair should then work for you. If you have
>> less than 3 AZ, then you are in troubles...
>>
>> Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8
>> nodes were recycled, there were absolutely no outage on Cassandra (other
>> two nodes wtill satisfy quorum consistency)
>>
>>
>> About the unreachable errors, I believe it can be due to the overload due
>> to the missing nodes. Pressure on the remaining node might be too strong.
>>
>> It is certainly possible but we have beefed up cluster with <3% CPU,
>> hardly any network I/o and disk usage. We have 162 nodes in the cluster and
>> each node doesn’t have more than 80 to 100MB of data.
>>
>>
>>
>> However, As soon as I started removing nodes one by one, every time time
>>> we see lot of timeout and unavailable exceptions which doesn’t make any
>>> sense because I am just removing a node that doesn’t even exist.
>>>
>>
>> This probably added even more load, if you are using vnodes, all the
>> remaining nodes probably started streaming data to each other node at the
>> speed of "nodetool getstreamthroughput". AWS network isn't that good, and
>> is probably saturated. Also have you the phi_convict_threshold configured
>> to a high value at least 10 or 12 ? This would avoid nodes to be marked
>> down that often.
>>
>> We are using c3.2xlarge which has good network throughput (1GB/sec I
>> think). We are using default value which is 200MB/sec in 2.0.9. We will
>> play with it in future and see if this could make any difference but as I
>> mentioned the data size on each node is not huge.
>> Regarding phi_convict_threshold, our Cassandra is not bringing itself
>> down. There was a bug in health check from one of our internal tool and
>> that tool is recycling the nodes. Nothing to do with Cassandra health.
>> Again we will keep an eye on it in future.
>>
>>
>> What does "nodetool tpstats" outputs ?
>>
>> Nodetool tpstats on which node? Any node?
>>
>>
>> Also you might try to monitor resources and see what happens (my guess is
>> you should focus at iowait, disk usage and network, have an eye at cpu too).
>>
>> We did monitor cpu, disk and network and they are all very low.
>>
>>
>> A quick fix would probably be to hardly throttle the network on all the
>> nodes and see if it helps:
>>
>> nodetool setstreamthroughput 2
>>
>> We will play with this config. 2.0.9 defaults to 200MB/sec which I think
>> is too high.
>>
>>
>> If this work, you could incrementally increase it and monitor, find the
>> good tuning and put it the cassandra.yaml.
>>
>> I opened a ticket a while ago about that issue:
>> https://issues.apache.org/jira/browse/CASSANDRA-9509
>>
>> I voted for this issue. Lets see if it gets picked up :).
>>
>>
>> I hope this will help you to go back to a healthy state allowing you a
>> fast upgrade ;-).
>>
>> C*heers,
>> -----------------------
>> Alain Rodriguez - alain@thelastpickle.com
>> France
>>
>> The Last Pickle - Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> 2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>:
>>
>>> Hi Robert,
>>> Thanks for your response.
>>>
>>> Replication factor is 3.
>>>
>>> We are in the process of upgrading to 2.2.4. We have had too many
>>> performance issues with later versions of Cassandra (I have asked asked for
>>> help related to that in the forum). We are close to getting to similar
>>> performance now and hopefully upgrade in next few weeks. Lot of testing to
>>> do :(.
>>>
>>> We are not removing multiple nodes at the same time. All dead nodes are
>>> from same AZ so there were no errors when the nodes were down as expected
>>> (because we use QUORUM). However, As soon as I started removing nodes one
>>> by one, every time time we see lot of timeout and unavailable exceptions
>>> which doesn’t make any sense because I am just removing a node that doesn’t
>>> even exist.
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: Robert Coli <rc...@eventbrite.com>
>>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>>> Date: Wednesday, March 2, 2016 at 2:52 PM
>>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>>> Subject: Re: Removing Node causes bunch of HostUnavailableException
>>>
>>> On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com> wrote:
>>>
>>>> We have few dead nodes in the cluster (Amazon ASG removed those
>>>> thinking there is an issue with health). Now we are trying to remove those
>>>> dead nodes from the cluster so that other nodes can take over. As soon as I
>>>> execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions
>>>> both on reads and writes. What I am not able to understand is, these are
>>>> deadnodes and don’t even physically exists. Why would removenode command
>>>> cause any outage of nodes in Cassandra when we had no errors whatsoever
>>>> before removing them. I could not really find a jira ticket for this.
>>>>
>>>
>>> What is your replication factor?
>>>
>>> Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
>>>
>>> Also, removing multiple nodes with removenode means your consistency is
>>> pretty hosed. Repair ASAP, but there are potential cases where repair won't
>>> help.
>>>
>>> =Rob
>>>
>>>
>>> =Rob
>>>
>>>
>>
>>
>
Re: Removing Node causes bunch of HostUnavailableException
Posted by "Peddi, Praveen" <pe...@amazon.com>.
Hi Jack,
Which node(s) were getting the HostNotAvailable errors - all nodes for every query, or just a small portion of the nodes on some queries?
Not all read/writes are failing with Unavalable or Timeout exception. Writes failures were around 10% of total calls. Reads were little worse (as worse as 35% of total calls).
It may take some time for the gossip state to propagate; maybe some of it is corrupted or needs a full refresh.
Were any of the seed nodes in the collection of nodes that were removed? How many seed nodes does each node typically have?
We currently use all hosts as seed hosts which I know is a very bad idea and we are going to fix that soon. The reason we use all hosts as seed hosts is because these hosts can get recycled for many reasons and we didn’t want to hard code the host names so we programmatically get host names (we wrote our own seed host provider). Could that be the reason for these failures? If a dead node is in the seed nodes list and we try to remove that node, could that lead to blip of failures. The failures don’t last for more than few minutes.
-- Jack Krupansky
On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen <pe...@amazon.com>> wrote:
Thanks Alain for quick and detailed response. My answers inline. One thing I want to clarify is, the nodes got recycled due to some automatic health check failure. This means old nodes are dead and new nodes got added w/o our intervention. So replacing nodes would not work for us since the new nodes were already added.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM)
Do you use at leat 3 distinct AZ ? If so, you should indeed be fine regarding data integrity. Also repair should then work for you. If you have less than 3 AZ, then you are in troubles...
Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8 nodes were recycled, there were absolutely no outage on Cassandra (other two nodes wtill satisfy quorum consistency)
About the unreachable errors, I believe it can be due to the overload due to the missing nodes. Pressure on the remaining node might be too strong.
It is certainly possible but we have beefed up cluster with <3% CPU, hardly any network I/o and disk usage. We have 162 nodes in the cluster and each node doesn’t have more than 80 to 100MB of data.
However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn’t make any sense because I am just removing a node that doesn’t even exist.
This probably added even more load, if you are using vnodes, all the remaining nodes probably started streaming data to each other node at the speed of "nodetool getstreamthroughput". AWS network isn't that good, and is probably saturated. Also have you the phi_convict_threshold configured to a high value at least 10 or 12 ? This would avoid nodes to be marked down that often.
We are using c3.2xlarge which has good network throughput (1GB/sec I think). We are using default value which is 200MB/sec in 2.0.9. We will play with it in future and see if this could make any difference but as I mentioned the data size on each node is not huge.
Regarding phi_convict_threshold, our Cassandra is not bringing itself down. There was a bug in health check from one of our internal tool and that tool is recycling the nodes. Nothing to do with Cassandra health. Again we will keep an eye on it in future.
What does "nodetool tpstats" outputs ?
Nodetool tpstats on which node? Any node?
Also you might try to monitor resources and see what happens (my guess is you should focus at iowait, disk usage and network, have an eye at cpu too).
We did monitor cpu, disk and network and they are all very low.
A quick fix would probably be to hardly throttle the network on all the nodes and see if it helps:
nodetool setstreamthroughput 2
We will play with this config. 2.0.9 defaults to 200MB/sec which I think is too high.
If this work, you could incrementally increase it and monitor, find the good tuning and put it the cassandra.yaml.
I opened a ticket a while ago about that issue: https://issues.apache.org/jira/browse/CASSANDRA-9509
I voted for this issue. Lets see if it gets picked up :).
I hope this will help you to go back to a healthy state allowing you a fast upgrade ;-).
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com<ma...@thelastpickle.com>
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>>:
Hi Robert,
Thanks for your response.
Replication factor is 3.
We are in the process of upgrading to 2.2.4. We have had too many performance issues with later versions of Cassandra (I have asked asked for help related to that in the forum). We are close to getting to similar performance now and hopefully upgrade in next few weeks. Lot of testing to do :(.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM). However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn’t make any sense because I am just removing a node that doesn’t even exist.
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, March 2, 2016 at 2:52 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Removing Node causes bunch of HostUnavailableException
On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com>> wrote:
We have few dead nodes in the cluster (Amazon ASG removed those thinking there is an issue with health). Now we are trying to remove those dead nodes from the cluster so that other nodes can take over. As soon as I execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions both on reads and writes. What I am not able to understand is, these are deadnodes and don’t even physically exists. Why would removenode command cause any outage of nodes in Cassandra when we had no errors whatsoever before removing them. I could not really find a jira ticket for this.
What is your replication factor?
Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
Also, removing multiple nodes with removenode means your consistency is pretty hosed. Repair ASAP, but there are potential cases where repair won't help.
=Rob
=Rob
Re: Removing Node causes bunch of HostUnavailableException
Posted by Jack Krupansky <ja...@gmail.com>.
Which node(s) were getting the HostNotAvailable errors - all nodes for
every query, or just a small portion of the nodes on some queries?
It may take some time for the gossip state to propagate; maybe some of it
is corrupted or needs a full refresh.
Were any of the seed nodes in the collection of nodes that were removed?
How many seed nodes does each node typically have?
-- Jack Krupansky
On Thu, Mar 3, 2016 at 4:16 PM, Peddi, Praveen <pe...@amazon.com> wrote:
> Thanks Alain for quick and detailed response. My answers inline. One thing
> I want to clarify is, the nodes got recycled due to some automatic health
> check failure. This means old nodes are dead and new nodes got added w/o
> our intervention. So replacing nodes would not work for us since the new
> nodes were already added.
>
>
>
>> We are not removing multiple nodes at the same time. All dead nodes are
>> from same AZ so there were no errors when the nodes were down as expected
>> (because we use QUORUM)
>
>
> Do you use at leat 3 distinct AZ ? If so, you should indeed be fine
> regarding data integrity. Also repair should then work for you. If you have
> less than 3 AZ, then you are in troubles...
>
> Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8
> nodes were recycled, there were absolutely no outage on Cassandra (other
> two nodes wtill satisfy quorum consistency)
>
>
> About the unreachable errors, I believe it can be due to the overload due
> to the missing nodes. Pressure on the remaining node might be too strong.
>
> It is certainly possible but we have beefed up cluster with <3% CPU,
> hardly any network I/o and disk usage. We have 162 nodes in the cluster and
> each node doesn’t have more than 80 to 100MB of data.
>
>
>
> However, As soon as I started removing nodes one by one, every time time
>> we see lot of timeout and unavailable exceptions which doesn’t make any
>> sense because I am just removing a node that doesn’t even exist.
>>
>
> This probably added even more load, if you are using vnodes, all the
> remaining nodes probably started streaming data to each other node at the
> speed of "nodetool getstreamthroughput". AWS network isn't that good, and
> is probably saturated. Also have you the phi_convict_threshold configured
> to a high value at least 10 or 12 ? This would avoid nodes to be marked
> down that often.
>
> We are using c3.2xlarge which has good network throughput (1GB/sec I
> think). We are using default value which is 200MB/sec in 2.0.9. We will
> play with it in future and see if this could make any difference but as I
> mentioned the data size on each node is not huge.
> Regarding phi_convict_threshold, our Cassandra is not bringing itself
> down. There was a bug in health check from one of our internal tool and
> that tool is recycling the nodes. Nothing to do with Cassandra health.
> Again we will keep an eye on it in future.
>
>
> What does "nodetool tpstats" outputs ?
>
> Nodetool tpstats on which node? Any node?
>
>
> Also you might try to monitor resources and see what happens (my guess is
> you should focus at iowait, disk usage and network, have an eye at cpu too).
>
> We did monitor cpu, disk and network and they are all very low.
>
>
> A quick fix would probably be to hardly throttle the network on all the
> nodes and see if it helps:
>
> nodetool setstreamthroughput 2
>
> We will play with this config. 2.0.9 defaults to 200MB/sec which I think
> is too high.
>
>
> If this work, you could incrementally increase it and monitor, find the
> good tuning and put it the cassandra.yaml.
>
> I opened a ticket a while ago about that issue:
> https://issues.apache.org/jira/browse/CASSANDRA-9509
>
> I voted for this issue. Lets see if it gets picked up :).
>
>
> I hope this will help you to go back to a healthy state allowing you a
> fast upgrade ;-).
>
> C*heers,
> -----------------------
> Alain Rodriguez - alain@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>:
>
>> Hi Robert,
>> Thanks for your response.
>>
>> Replication factor is 3.
>>
>> We are in the process of upgrading to 2.2.4. We have had too many
>> performance issues with later versions of Cassandra (I have asked asked for
>> help related to that in the forum). We are close to getting to similar
>> performance now and hopefully upgrade in next few weeks. Lot of testing to
>> do :(.
>>
>> We are not removing multiple nodes at the same time. All dead nodes are
>> from same AZ so there were no errors when the nodes were down as expected
>> (because we use QUORUM). However, As soon as I started removing nodes one
>> by one, every time time we see lot of timeout and unavailable exceptions
>> which doesn’t make any sense because I am just removing a node that doesn’t
>> even exist.
>>
>>
>>
>>
>>
>>
>> From: Robert Coli <rc...@eventbrite.com>
>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Date: Wednesday, March 2, 2016 at 2:52 PM
>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Subject: Re: Removing Node causes bunch of HostUnavailableException
>>
>> On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com> wrote:
>>
>>> We have few dead nodes in the cluster (Amazon ASG removed those thinking
>>> there is an issue with health). Now we are trying to remove those dead
>>> nodes from the cluster so that other nodes can take over. As soon as I
>>> execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions
>>> both on reads and writes. What I am not able to understand is, these are
>>> deadnodes and don’t even physically exists. Why would removenode command
>>> cause any outage of nodes in Cassandra when we had no errors whatsoever
>>> before removing them. I could not really find a jira ticket for this.
>>>
>>
>> What is your replication factor?
>>
>> Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
>>
>> Also, removing multiple nodes with removenode means your consistency is
>> pretty hosed. Repair ASAP, but there are potential cases where repair won't
>> help.
>>
>> =Rob
>>
>>
>> =Rob
>>
>>
>
>
Re: Removing Node causes bunch of HostUnavailableException
Posted by "Peddi, Praveen" <pe...@amazon.com>.
Thanks Alain for quick and detailed response. My answers inline. One thing I want to clarify is, the nodes got recycled due to some automatic health check failure. This means old nodes are dead and new nodes got added w/o our intervention. So replacing nodes would not work for us since the new nodes were already added.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM)
Do you use at leat 3 distinct AZ ? If so, you should indeed be fine regarding data integrity. Also repair should then work for you. If you have less than 3 AZ, then you are in troubles...
Yes we use 3 distinct AZs and replicate to all 3 Azs which is why when 8 nodes were recycled, there were absolutely no outage on Cassandra (other two nodes wtill satisfy quorum consistency)
About the unreachable errors, I believe it can be due to the overload due to the missing nodes. Pressure on the remaining node might be too strong.
It is certainly possible but we have beefed up cluster with <3% CPU, hardly any network I/o and disk usage. We have 162 nodes in the cluster and each node doesn’t have more than 80 to 100MB of data.
However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn’t make any sense because I am just removing a node that doesn’t even exist.
This probably added even more load, if you are using vnodes, all the remaining nodes probably started streaming data to each other node at the speed of "nodetool getstreamthroughput". AWS network isn't that good, and is probably saturated. Also have you the phi_convict_threshold configured to a high value at least 10 or 12 ? This would avoid nodes to be marked down that often.
We are using c3.2xlarge which has good network throughput (1GB/sec I think). We are using default value which is 200MB/sec in 2.0.9. We will play with it in future and see if this could make any difference but as I mentioned the data size on each node is not huge.
Regarding phi_convict_threshold, our Cassandra is not bringing itself down. There was a bug in health check from one of our internal tool and that tool is recycling the nodes. Nothing to do with Cassandra health. Again we will keep an eye on it in future.
What does "nodetool tpstats" outputs ?
Nodetool tpstats on which node? Any node?
Also you might try to monitor resources and see what happens (my guess is you should focus at iowait, disk usage and network, have an eye at cpu too).
We did monitor cpu, disk and network and they are all very low.
A quick fix would probably be to hardly throttle the network on all the nodes and see if it helps:
nodetool setstreamthroughput 2
We will play with this config. 2.0.9 defaults to 200MB/sec which I think is too high.
If this work, you could incrementally increase it and monitor, find the good tuning and put it the cassandra.yaml.
I opened a ticket a while ago about that issue: https://issues.apache.org/jira/browse/CASSANDRA-9509
I voted for this issue. Lets see if it gets picked up :).
I hope this will help you to go back to a healthy state allowing you a fast upgrade ;-).
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com<ma...@thelastpickle.com>
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>>:
Hi Robert,
Thanks for your response.
Replication factor is 3.
We are in the process of upgrading to 2.2.4. We have had too many performance issues with later versions of Cassandra (I have asked asked for help related to that in the forum). We are close to getting to similar performance now and hopefully upgrade in next few weeks. Lot of testing to do :(.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM). However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn’t make any sense because I am just removing a node that doesn’t even exist.
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, March 2, 2016 at 2:52 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Removing Node causes bunch of HostUnavailableException
On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com>> wrote:
We have few dead nodes in the cluster (Amazon ASG removed those thinking there is an issue with health). Now we are trying to remove those dead nodes from the cluster so that other nodes can take over. As soon as I execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions both on reads and writes. What I am not able to understand is, these are deadnodes and don’t even physically exists. Why would removenode command cause any outage of nodes in Cassandra when we had no errors whatsoever before removing them. I could not really find a jira ticket for this.
What is your replication factor?
Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
Also, removing multiple nodes with removenode means your consistency is pretty hosed. Repair ASAP, but there are potential cases where repair won't help.
=Rob
=Rob
Re: Removing Node causes bunch of HostUnavailableException
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Just thought of a solution that might actually work even better.
You could try replacing one node at the time (instead of removing them)
I believe thhis would decrease the amount of streams significantly.
https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
Good luck,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-03-02 23:57 GMT+01:00 Alain RODRIGUEZ <ar...@gmail.com>:
> Hi Praveen,
>
>
>> We are not removing multiple nodes at the same time. All dead nodes are
>> from same AZ so there were no errors when the nodes were down as expected
>> (because we use QUORUM)
>
>
> Do you use at leat 3 distinct AZ ? If so, you should indeed be fine
> regarding data integrity. Also repair should then work for you. If you have
> less than 3 AZ, then you are in troubles...
>
> About the unreachable errors, I believe it can be due to the overload due
> to the missing nodes. Pressure on the remaining node might be too strong.
>
> However, As soon as I started removing nodes one by one, every time time
>> we see lot of timeout and unavailable exceptions which doesn’t make any
>> sense because I am just removing a node that doesn’t even exist.
>>
>
> This probably added even more load, if you are using vnodes, all the
> remaining nodes probably started streaming data to each other node at the
> speed of "nodetool getstreamthroughput". AWS network isn't that good, and
> is probably saturated. Also have you the phi_convict_threshold configured
> to a high value at least 10 or 12 ? This would avoid nodes to be marked
> down that often.
>
> What does "nodetool tpstats" outputs ?
>
> Also you might try to monitor resources and see what happens (my guess is
> you should focus at iowait, disk usage and network, have an eye at cpu too).
>
> A quick fix would probably be to hardly throttle the network on all the
> nodes and see if it helps:
>
> nodetool setstreamthroughput 2
>
> If this work, you could incrementally increase it and monitor, find the
> good tuning and put it the cassandra.yaml.
>
> I opened a ticket a while ago about that issue:
> https://issues.apache.org/jira/browse/CASSANDRA-9509
>
> I hope this will help you to go back to a healthy state allowing you a
> fast upgrade ;-).
>
> C*heers,
> -----------------------
> Alain Rodriguez - alain@thelastpickle.com
> France
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>:
>
>> Hi Robert,
>> Thanks for your response.
>>
>> Replication factor is 3.
>>
>> We are in the process of upgrading to 2.2.4. We have had too many
>> performance issues with later versions of Cassandra (I have asked asked for
>> help related to that in the forum). We are close to getting to similar
>> performance now and hopefully upgrade in next few weeks. Lot of testing to
>> do :(.
>>
>> We are not removing multiple nodes at the same time. All dead nodes are
>> from same AZ so there were no errors when the nodes were down as expected
>> (because we use QUORUM). However, As soon as I started removing nodes one
>> by one, every time time we see lot of timeout and unavailable exceptions
>> which doesn’t make any sense because I am just removing a node that doesn’t
>> even exist.
>>
>>
>>
>>
>>
>>
>> From: Robert Coli <rc...@eventbrite.com>
>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Date: Wednesday, March 2, 2016 at 2:52 PM
>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Subject: Re: Removing Node causes bunch of HostUnavailableException
>>
>> On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com> wrote:
>>
>>> We have few dead nodes in the cluster (Amazon ASG removed those thinking
>>> there is an issue with health). Now we are trying to remove those dead
>>> nodes from the cluster so that other nodes can take over. As soon as I
>>> execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions
>>> both on reads and writes. What I am not able to understand is, these are
>>> deadnodes and don’t even physically exists. Why would removenode command
>>> cause any outage of nodes in Cassandra when we had no errors whatsoever
>>> before removing them. I could not really find a jira ticket for this.
>>>
>>
>> What is your replication factor?
>>
>> Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
>>
>> Also, removing multiple nodes with removenode means your consistency is
>> pretty hosed. Repair ASAP, but there are potential cases where repair won't
>> help.
>>
>> =Rob
>>
>>
>> =Rob
>>
>>
>
>
Re: Removing Node causes bunch of HostUnavailableException
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi Praveen,
> We are not removing multiple nodes at the same time. All dead nodes are
> from same AZ so there were no errors when the nodes were down as expected
> (because we use QUORUM)
Do you use at leat 3 distinct AZ ? If so, you should indeed be fine
regarding data integrity. Also repair should then work for you. If you have
less than 3 AZ, then you are in troubles...
About the unreachable errors, I believe it can be due to the overload due
to the missing nodes. Pressure on the remaining node might be too strong.
However, As soon as I started removing nodes one by one, every time time we
> see lot of timeout and unavailable exceptions which doesn’t make any sense
> because I am just removing a node that doesn’t even exist.
>
This probably added even more load, if you are using vnodes, all the
remaining nodes probably started streaming data to each other node at the
speed of "nodetool getstreamthroughput". AWS network isn't that good, and
is probably saturated. Also have you the phi_convict_threshold configured
to a high value at least 10 or 12 ? This would avoid nodes to be marked
down that often.
What does "nodetool tpstats" outputs ?
Also you might try to monitor resources and see what happens (my guess is
you should focus at iowait, disk usage and network, have an eye at cpu too).
A quick fix would probably be to hardly throttle the network on all the
nodes and see if it helps:
nodetool setstreamthroughput 2
If this work, you could incrementally increase it and monitor, find the
good tuning and put it the cassandra.yaml.
I opened a ticket a while ago about that issue:
https://issues.apache.org/jira/browse/CASSANDRA-9509
I hope this will help you to go back to a healthy state allowing you a fast
upgrade ;-).
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-03-02 22:17 GMT+01:00 Peddi, Praveen <pe...@amazon.com>:
> Hi Robert,
> Thanks for your response.
>
> Replication factor is 3.
>
> We are in the process of upgrading to 2.2.4. We have had too many
> performance issues with later versions of Cassandra (I have asked asked for
> help related to that in the forum). We are close to getting to similar
> performance now and hopefully upgrade in next few weeks. Lot of testing to
> do :(.
>
> We are not removing multiple nodes at the same time. All dead nodes are
> from same AZ so there were no errors when the nodes were down as expected
> (because we use QUORUM). However, As soon as I started removing nodes one
> by one, every time time we see lot of timeout and unavailable exceptions
> which doesn’t make any sense because I am just removing a node that doesn’t
> even exist.
>
>
>
>
>
>
> From: Robert Coli <rc...@eventbrite.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Wednesday, March 2, 2016 at 2:52 PM
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: Removing Node causes bunch of HostUnavailableException
>
> On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com> wrote:
>
>> We have few dead nodes in the cluster (Amazon ASG removed those thinking
>> there is an issue with health). Now we are trying to remove those dead
>> nodes from the cluster so that other nodes can take over. As soon as I
>> execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions
>> both on reads and writes. What I am not able to understand is, these are
>> deadnodes and don’t even physically exists. Why would removenode command
>> cause any outage of nodes in Cassandra when we had no errors whatsoever
>> before removing them. I could not really find a jira ticket for this.
>>
>
> What is your replication factor?
>
> Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
>
> Also, removing multiple nodes with removenode means your consistency is
> pretty hosed. Repair ASAP, but there are potential cases where repair won't
> help.
>
> =Rob
>
>
> =Rob
>
>
Re: Removing Node causes bunch of HostUnavailableException
Posted by "Peddi, Praveen" <pe...@amazon.com>.
Hi Robert,
Thanks for your response.
Replication factor is 3.
We are in the process of upgrading to 2.2.4. We have had too many performance issues with later versions of Cassandra (I have asked asked for help related to that in the forum). We are close to getting to similar performance now and hopefully upgrade in next few weeks. Lot of testing to do :(.
We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM). However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn't make any sense because I am just removing a node that doesn't even exist.
From: Robert Coli <rc...@eventbrite.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, March 2, 2016 at 2:52 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Removing Node causes bunch of HostUnavailableException
On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com>> wrote:
We have few dead nodes in the cluster (Amazon ASG removed those thinking there is an issue with health). Now we are trying to remove those dead nodes from the cluster so that other nodes can take over. As soon as I execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions both on reads and writes. What I am not able to understand is, these are deadnodes and don't even physically exists. Why would removenode command cause any outage of nodes in Cassandra when we had no errors whatsoever before removing them. I could not really find a jira ticket for this.
What is your replication factor?
Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
Also, removing multiple nodes with removenode means your consistency is pretty hosed. Repair ASAP, but there are potential cases where repair won't help.
=Rob
=Rob
Re: Removing Node causes bunch of HostUnavailableException
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com> wrote:
> We have few dead nodes in the cluster (Amazon ASG removed those thinking
> there is an issue with health). Now we are trying to remove those dead
> nodes from the cluster so that other nodes can take over. As soon as I
> execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions
> both on reads and writes. What I am not able to understand is, these are
> deadnodes and don’t even physically exists. Why would removenode command
> cause any outage of nodes in Cassandra when we had no errors whatsoever
> before removing them. I could not really find a jira ticket for this.
>
What is your replication factor?
Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP.
Also, removing multiple nodes with removenode means your consistency is
pretty hosed. Repair ASAP, but there are potential cases where repair won't
help.
=Rob
=Rob