You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Andrew Bialecki <an...@gmail.com> on 2012/10/28 05:13:29 UTC
Simulating a failed node
Hey everyone,
I'm trying to simulate what happens when a node goes down to make sure my
cluster can gracefully handle node failures. For my setup I have a 3 node
cluster running 1.1.5. I'm then using the stress tool included in 1.1.5
coming from an external server and running it with the following arguments:
tools/bin/cassandra-stress -d <server1>,<server2>,<server3> -n 1000000
I start up the stress test and then down one of the nodes. The stress test
instantly fails with the following errors (which of course are the same
error from different threads) looking like:
...
Operation [158320] retried 10 times - error inserting key 0158320
((UnavailableException))
Operation [158429] retried 10 times - error inserting key 0158429
((UnavailableException))
Operation [158439] retried 10 times - error inserting key 0158439
((UnavailableException))
Operation [158470] retried 10 times - error inserting key 0158470
((UnavailableException))
158534,0,0,NaN,43
FAILURE
I'm sure my naive setup is flawed in some way, but what I was hoping for
was when the node went down it would fail to write to the downed node and
instead write to one of the other nodes in the clusters. So question is why
are writes failing even after a retry? It might be the stress client
doesn't pool connections (I took a quick look, but might've not looked
deeply enough), however I also tried only specifying the first two server
nodes and then downing the third with the same failure.
Thanks in advance.
Andrew
Re: Simulating a failed node
Posted by Andrew Bialecki <an...@gmail.com>.
The default replication factor and consistency level for the stress tool is
one, so that's what I'm using. I've also experimented and seen the same
behavior with RF=2, but I haven't tried a different CL.
On Sun, Oct 28, 2012 at 12:36 AM, Watanabe Maki <wa...@gmail.com>wrote:
> What RF and CL are you using?
>
>
> On 2012/10/28, at 13:13, Andrew Bialecki <an...@gmail.com>
> wrote:
>
> Hey everyone,
>
> I'm trying to simulate what happens when a node goes down to make sure my
> cluster can gracefully handle node failures. For my setup I have a 3 node
> cluster running 1.1.5. I'm then using the stress tool included in 1.1.5
> coming from an external server and running it with the following arguments:
>
> tools/bin/cassandra-stress -d <server1>,<server2>,<server3> -n 1000000
>
>
> I start up the stress test and then down one of the nodes. The stress test
> instantly fails with the following errors (which of course are the same
> error from different threads) looking like:
>
> ...
>
> Operation [158320] retried 10 times - error inserting key 0158320
> ((UnavailableException))
> Operation [158429] retried 10 times - error inserting key 0158429
> ((UnavailableException))
> Operation [158439] retried 10 times - error inserting key 0158439
> ((UnavailableException))
> Operation [158470] retried 10 times - error inserting key 0158470
> ((UnavailableException))
> 158534,0,0,NaN,43
> FAILURE
>
>
> I'm sure my naive setup is flawed in some way, but what I was hoping for
> was when the node went down it would fail to write to the downed node and
> instead write to one of the other nodes in the clusters. So question is why
> are writes failing even after a retry? It might be the stress client
> doesn't pool connections (I took a quick look, but might've not looked
> deeply enough), however I also tried only specifying the first two server
> nodes and then downing the third with the same failure.
>
> Thanks in advance.
>
> Andrew
>
>
Re: Simulating a failed node
Posted by Watanabe Maki <wa...@gmail.com>.
What RF and CL are you using?
On 2012/10/28, at 13:13, Andrew Bialecki <an...@gmail.com> wrote:
> Hey everyone,
>
> I'm trying to simulate what happens when a node goes down to make sure my cluster can gracefully handle node failures. For my setup I have a 3 node cluster running 1.1.5. I'm then using the stress tool included in 1.1.5 coming from an external server and running it with the following arguments:
>
> tools/bin/cassandra-stress -d <server1>,<server2>,<server3> -n 1000000
>
> I start up the stress test and then down one of the nodes. The stress test instantly fails with the following errors (which of course are the same error from different threads) looking like:
>
> ...
> Operation [158320] retried 10 times - error inserting key 0158320 ((UnavailableException))
> Operation [158429] retried 10 times - error inserting key 0158429 ((UnavailableException))
> Operation [158439] retried 10 times - error inserting key 0158439 ((UnavailableException))
> Operation [158470] retried 10 times - error inserting key 0158470 ((UnavailableException))
> 158534,0,0,NaN,43
> FAILURE
>
> I'm sure my naive setup is flawed in some way, but what I was hoping for was when the node went down it would fail to write to the downed node and instead write to one of the other nodes in the clusters. So question is why are writes failing even after a retry? It might be the stress client doesn't pool connections (I took a quick look, but might've not looked deeply enough), however I also tried only specifying the first two server nodes and then downing the third with the same failure.
>
> Thanks in advance.
>
> Andrew
Re: Simulating a failed node
Posted by Andrew Bialecki <an...@gmail.com>.
Thanks, extremely helpful. The key bit was I wasn't flushing the old
Keyspace before re-running the stress test, so I was stuck at RF = 1 from a
previous run despite passing RF = 2 to the stress tool.
On Sun, Oct 28, 2012 at 2:49 AM, Peter Schuller <peter.schuller@infidyne.com
> wrote:
> > Operation [158320] retried 10 times - error inserting key 0158320
> ((UnavailableException))
>
> This means that at the point where the thrift request to write data
> was handled, the co-ordinator node (the one your client is connected
> to) believed that, among the replicas responsible for the key, too
> many were down to satisfy the consistency level. Most likely causes
> would be that you're in fact not using RF > 2 (e.g., is the RF really
> > 1 for the keyspace you're inserting into), or you're in fact not
> using ONE.
>
> > I'm sure my naive setup is flawed in some way, but what I was hoping for
> was when the node went down it would fail to write to the downed node and
> instead write to one of the other nodes in the clusters. So question is why
> are writes failing even after a retry? It might be the stress client
> doesn't pool connections (I took
>
> Write always go to all responsible replicas that are up, and when
> enough return (according to consistency level), the insert succeeds.
>
> If replicas fail to respond you may get a TimeoutException.
>
> UnavailableException means it didn't even try because it didn't have
> enough replicas to even try to write to.
>
> (Note though: Reads are a bit of a different story and if you want to
> test behavior when nodes go down I suggest including that. See
> CASSANDRA-2540 and CASSANDRA-3927.)
>
> --
> / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
>
Re: Simulating a failed node
Posted by Peter Schuller <pe...@infidyne.com>.
> Operation [158320] retried 10 times - error inserting key 0158320 ((UnavailableException))
This means that at the point where the thrift request to write data
was handled, the co-ordinator node (the one your client is connected
to) believed that, among the replicas responsible for the key, too
many were down to satisfy the consistency level. Most likely causes
would be that you're in fact not using RF > 2 (e.g., is the RF really
> 1 for the keyspace you're inserting into), or you're in fact not
using ONE.
> I'm sure my naive setup is flawed in some way, but what I was hoping for was when the node went down it would fail to write to the downed node and instead write to one of the other nodes in the clusters. So question is why are writes failing even after a retry? It might be the stress client doesn't pool connections (I took
Write always go to all responsible replicas that are up, and when
enough return (according to consistency level), the insert succeeds.
If replicas fail to respond you may get a TimeoutException.
UnavailableException means it didn't even try because it didn't have
enough replicas to even try to write to.
(Note though: Reads are a bit of a different story and if you want to
test behavior when nodes go down I suggest including that. See
CASSANDRA-2540 and CASSANDRA-3927.)
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)