You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Timo Nentwig <ti...@toptarif.de> on 2010/12/09 16:40:36 UTC
Quorum: killing 1 out of 3 server kills the cluster (?)
Hi!
I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
What may I be doing wrong?
thx
tcn
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Peter Schuller <pe...@infidyne.com>.
> If you switch your writes to CL ONE when a failure occurs, you might as well
> use ONE for all writes. ONE and QUORUM behave the same when all nodes are
> working correctly.
Consistency wise yes, but not durability wise. Writing to QUOROM but
reading at ONE is useful if you want higher durability guarantees, but
is fine with inconsistent reads. In other words, if you want to avoid
loosing data if a node completely blows up and its data becomes
irrevocably lost forever. (The data that would be lost would be data
written at ONE that only reached that node, and no others.)
--
/ Peter Schuller
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Peter Schuller <pe...@infidyne.com>.
> That's finally a precise statement! :) I was wondering what " to at least 1 replica's commit log" is supposed to actually mean: http://wiki.apache.org/cassandra/API
The main idea is that it has been "officially delivered" to one
replicate. If Cassandra only did batch-wise commit such that a write
was never ACK:ed until it was durable, it would mean that it had been
durably written to 1 replica set.
I suspect the phrasing is to get around the fact that it is not
actually durably written if nodes are configured to use periodic sync
mode.
> Does quorum mean that data is replicated to q nodes or to at least q nodes?
That it is replicated to at least a quorom of nodes before the write
is considered successful. This does not prevent further propagation to
all nodes; data always gets replicated according to replication
factor. Consistency levels only affect the consistency requirements of
the particular request.
> I just added another blank machine to my cluster. Nothing happened as expected (stopped writing to the cluster) but after I ran nodetool repair it held more data than all other nodes. So it copied data from the other nodes to this one? I assumed that data is replicated to q nodes not to all, is quorum 'only' about consistency and not about saving storage space?
The new node should have gotten its appropriate amount according to
the ring responsibility (i.e., tokens). I'm not sure why a new node
would get more than its fair share (according to tokens) of data
though.
There is one extreme case which would be if the cluster has seen lots
of writes in degraded states so that there is a lot of data around the
cluster that has not yet reached their full replica sets. A repair on
a new node might make the new node be the only one that has all the
data it should have... but you'd have to have written data at low
consistency level during pretty shaky periods for this to have a
significant effect (especially if hinted handoff is turned on).
--
/ Peter Schuller
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Timo Nentwig <ti...@toptarif.de>.
On Dec 9, 2010, at 18:50, Tyler Hobbs wrote:
> If you switch your writes to CL ONE when a failure occurs, you might as well use ONE for all writes. ONE and QUORUM behave the same when all nodes are working correctly.
That's finally a precise statement! :) I was wondering what " to at least 1 replica's commit log" is supposed to actually mean: http://wiki.apache.org/cassandra/API
Does quorum mean that data is replicated to q nodes or to at least q nodes? I just added another blank machine to my cluster. Nothing happened as expected (stopped writing to the cluster) but after I ran nodetool repair it held more data than all other nodes. So it copied data from the other nodes to this one? I assumed that data is replicated to q nodes not to all, is quorum 'only' about consistency and not about saving storage space?
> - Tyler
>
> On Thu, Dec 9, 2010 at 11:26 AM, Timo Nentwig <ti...@toptarif.de> wrote:
>
> On Dec 9, 2010, at 17:55, Sylvain Lebresne wrote:
>
> >> I naively assume that if I kill either node that holds N1 (i.e. node 1 or 3), N1 will still remain on another node. Only if both fail, I actually lose data. But apparently this is not how it works...
> >
> > Sure, the data that N1 holds is also on another node and you won't
> > lose it by only losing N1.
> > But when you do a quorum query, you are saying to Cassandra "Please
> > please would you fail my request
> > if you can't get a response from 2 nodes". So if only 1 node holding
> > the data is up at the moment of the
> > query then Cassandra, which is a very polite software, do what you
> > asked and fail.
>
> And my application would fall back to ONE. Quorum writes will also fail so I would also use ONE so that the app stays up. What would I have to do make the data to redistribute when the broken node is up again? Simply call nodetool repair on it?
>
> > If you want Cassandra to send you an answer with only one node up, use
> > CL=ONE (as said by David).
> >
> >>
> >>> On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
> >>> I'ts 2 out of the number of replicas, not the number of nodes. At RF=2, you have
> >>> 2 replicas. And since quorum is also 2 with that replication factor,
> >>> you cannot lose
> >>> a node, otherwise some query will end up as UnavailableException.
> >>>
> >>> Again, this is not related to the total number of nodes. Even with 200
> >>> nodes, if
> >>> you use RF=2, you will have some query that fail (altough much less that what
> >>> you are probably seeing).
> >>>
> >>> On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de> wrote:
> >>>>
> >>>> On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
> >>>>
> >>>>> Quorum is really only useful when RF > 2, since the for a quorum to
> >>>>> succeed RF/2+1 replicas must be available.
> >>>>
> >>>> 2/2+1==2 and I killed 1 of 3, so... don't get it.
> >>>>
> >>>>> This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
> >>>>>
> >>>>> /d
> >>>>>
> >>>>> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
> >>>>>> Hi!
> >>>>>>
> >>>>>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
> >>>>>>
> >>>>>> What may I be doing wrong?
> >>>>>>
> >>>>>> thx
> >>>>>> tcn
> >>>>
> >>>>
> >>>
> >>
> >>
>
>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Tyler Hobbs <ty...@riptano.com>.
If you switch your writes to CL ONE when a failure occurs, you might as well
use ONE for all writes. ONE and QUORUM behave the same when all nodes are
working correctly.
- Tyler
On Thu, Dec 9, 2010 at 11:26 AM, Timo Nentwig <ti...@toptarif.de>wrote:
>
> On Dec 9, 2010, at 17:55, Sylvain Lebresne wrote:
>
> >> I naively assume that if I kill either node that holds N1 (i.e. node 1
> or 3), N1 will still remain on another node. Only if both fail, I actually
> lose data. But apparently this is not how it works...
> >
> > Sure, the data that N1 holds is also on another node and you won't
> > lose it by only losing N1.
> > But when you do a quorum query, you are saying to Cassandra "Please
> > please would you fail my request
> > if you can't get a response from 2 nodes". So if only 1 node holding
> > the data is up at the moment of the
> > query then Cassandra, which is a very polite software, do what you
> > asked and fail.
>
> And my application would fall back to ONE. Quorum writes will also fail so
> I would also use ONE so that the app stays up. What would I have to do make
> the data to redistribute when the broken node is up again? Simply call
> nodetool repair on it?
>
> > If you want Cassandra to send you an answer with only one node up, use
> > CL=ONE (as said by David).
> >
> >>
> >>> On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com>
> wrote:
> >>> I'ts 2 out of the number of replicas, not the number of nodes. At RF=2,
> you have
> >>> 2 replicas. And since quorum is also 2 with that replication factor,
> >>> you cannot lose
> >>> a node, otherwise some query will end up as UnavailableException.
> >>>
> >>> Again, this is not related to the total number of nodes. Even with 200
> >>> nodes, if
> >>> you use RF=2, you will have some query that fail (altough much less
> that what
> >>> you are probably seeing).
> >>>
> >>> On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de>
> wrote:
> >>>>
> >>>> On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
> >>>>
> >>>>> Quorum is really only useful when RF > 2, since the for a quorum to
> >>>>> succeed RF/2+1 replicas must be available.
> >>>>
> >>>> 2/2+1==2 and I killed 1 of 3, so... don't get it.
> >>>>
> >>>>> This means for RF = 2, consistency levels QUORUM and ALL yield the
> same result.
> >>>>>
> >>>>> /d
> >>>>>
> >>>>> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <
> timo.nentwig@toptarif.de> wrote:
> >>>>>> Hi!
> >>>>>>
> >>>>>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and
> use quorum for writes. But when I shut down one of them
> UnavailableExceptions are thrown. Why is that? Isn't that the sense of
> quorum and a fault-tolerant DB that it continues with the remaining 2 nodes
> and redistributes the data to the broken one as soons as its up again?
> >>>>>>
> >>>>>> What may I be doing wrong?
> >>>>>>
> >>>>>> thx
> >>>>>> tcn
> >>>>
> >>>>
> >>>
> >>
> >>
>
>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Sylvain Lebresne <sy...@yakaz.com>.
> And my application would fall back to ONE. Quorum writes will also fail so I would also use ONE so that the app stays up. What would I have to do make the data to redistribute when the broken node is up again? Simply call nodetool repair on it?
There is 3 mechanisms for that:
- hinted handoff: basically, when the node is back up, the other
node will send him what he missed.
- read-repair: whenever you read a data and an inconsistency is
detected (because one node is not up to date), it gets repaired.
- calling nodetool repair
The two first are automatic, you have nothing to do.
Nodetool repair is usually run only periodically (say once a week) to
repair some cold data that wasn't dealt with by
the two first mechanisms.
--
Sylvain
>
>> If you want Cassandra to send you an answer with only one node up, use
>> CL=ONE (as said by David).
>>
>>>
>>>> On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
>>>> I'ts 2 out of the number of replicas, not the number of nodes. At RF=2, you have
>>>> 2 replicas. And since quorum is also 2 with that replication factor,
>>>> you cannot lose
>>>> a node, otherwise some query will end up as UnavailableException.
>>>>
>>>> Again, this is not related to the total number of nodes. Even with 200
>>>> nodes, if
>>>> you use RF=2, you will have some query that fail (altough much less that what
>>>> you are probably seeing).
>>>>
>>>> On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>>>>>
>>>>> On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
>>>>>
>>>>>> Quorum is really only useful when RF > 2, since the for a quorum to
>>>>>> succeed RF/2+1 replicas must be available.
>>>>>
>>>>> 2/2+1==2 and I killed 1 of 3, so... don't get it.
>>>>>
>>>>>> This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
>>>>>>
>>>>>> /d
>>>>>>
>>>>>> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>>>>>>> Hi!
>>>>>>>
>>>>>>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
>>>>>>>
>>>>>>> What may I be doing wrong?
>>>>>>>
>>>>>>> thx
>>>>>>> tcn
>>>>>
>>>>>
>>>>
>>>
>>>
>
>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Timo Nentwig <ti...@toptarif.de>.
On Dec 9, 2010, at 17:55, Sylvain Lebresne wrote:
>> I naively assume that if I kill either node that holds N1 (i.e. node 1 or 3), N1 will still remain on another node. Only if both fail, I actually lose data. But apparently this is not how it works...
>
> Sure, the data that N1 holds is also on another node and you won't
> lose it by only losing N1.
> But when you do a quorum query, you are saying to Cassandra "Please
> please would you fail my request
> if you can't get a response from 2 nodes". So if only 1 node holding
> the data is up at the moment of the
> query then Cassandra, which is a very polite software, do what you
> asked and fail.
And my application would fall back to ONE. Quorum writes will also fail so I would also use ONE so that the app stays up. What would I have to do make the data to redistribute when the broken node is up again? Simply call nodetool repair on it?
> If you want Cassandra to send you an answer with only one node up, use
> CL=ONE (as said by David).
>
>>
>>> On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
>>> I'ts 2 out of the number of replicas, not the number of nodes. At RF=2, you have
>>> 2 replicas. And since quorum is also 2 with that replication factor,
>>> you cannot lose
>>> a node, otherwise some query will end up as UnavailableException.
>>>
>>> Again, this is not related to the total number of nodes. Even with 200
>>> nodes, if
>>> you use RF=2, you will have some query that fail (altough much less that what
>>> you are probably seeing).
>>>
>>> On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>>>>
>>>> On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
>>>>
>>>>> Quorum is really only useful when RF > 2, since the for a quorum to
>>>>> succeed RF/2+1 replicas must be available.
>>>>
>>>> 2/2+1==2 and I killed 1 of 3, so... don't get it.
>>>>
>>>>> This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
>>>>>
>>>>> /d
>>>>>
>>>>> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>>>>>> Hi!
>>>>>>
>>>>>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
>>>>>>
>>>>>> What may I be doing wrong?
>>>>>>
>>>>>> thx
>>>>>> tcn
>>>>
>>>>
>>>
>>
>>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Sylvain Lebresne <sy...@yakaz.com>.
> I naively assume that if I kill either node that holds N1 (i.e. node 1 or 3), N1 will still remain on another node. Only if both fail, I actually lose data. But apparently this is not how it works...
Sure, the data that N1 holds is also on another node and you won't
lose it by only losing N1.
But when you do a quorum query, you are saying to Cassandra "Please
please would you fail my request
if you can't get a response from 2 nodes". So if only 1 node holding
the data is up at the moment of the
query then Cassandra, which is a very polite software, do what you
asked and fail.
If you want Cassandra to send you an answer with only one node up, use
CL=ONE (as said by David).
>
>> On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
>> I'ts 2 out of the number of replicas, not the number of nodes. At RF=2, you have
>> 2 replicas. And since quorum is also 2 with that replication factor,
>> you cannot lose
>> a node, otherwise some query will end up as UnavailableException.
>>
>> Again, this is not related to the total number of nodes. Even with 200
>> nodes, if
>> you use RF=2, you will have some query that fail (altough much less that what
>> you are probably seeing).
>>
>> On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>> >
>> > On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
>> >
>> >> Quorum is really only useful when RF > 2, since the for a quorum to
>> >> succeed RF/2+1 replicas must be available.
>> >
>> > 2/2+1==2 and I killed 1 of 3, so... don't get it.
>> >
>> >> This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
>> >>
>> >> /d
>> >>
>> >> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>> >>> Hi!
>> >>>
>> >>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
>> >>>
>> >>> What may I be doing wrong?
>> >>>
>> >>> thx
>> >>> tcn
>> >
>> >
>>
>
>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by David Boxenhorn <da...@lookin2.com>.
If that is what you want, use CL=ONE
On Thu, Dec 9, 2010 at 6:43 PM, Timo Nentwig <ti...@toptarif.de>wrote:
>
> On Dec 9, 2010, at 17:39, David Boxenhorn wrote:
>
> > In other words, if you want to use QUORUM, you need to set RF>=3.
> >
> > (I know because I had exactly the same problem.)
>
> I naively assume that if I kill either node that holds N1 (i.e. node 1 or
> 3), N1 will still remain on another node. Only if both fail, I actually lose
> data. But apparently this is not how it works...
>
> > On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com>
> wrote:
> > I'ts 2 out of the number of replicas, not the number of nodes. At RF=2,
> you have
> > 2 replicas. And since quorum is also 2 with that replication factor,
> > you cannot lose
> > a node, otherwise some query will end up as UnavailableException.
> >
> > Again, this is not related to the total number of nodes. Even with 200
> > nodes, if
> > you use RF=2, you will have some query that fail (altough much less that
> what
> > you are probably seeing).
> >
> > On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de>
> wrote:
> > >
> > > On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
> > >
> > >> Quorum is really only useful when RF > 2, since the for a quorum to
> > >> succeed RF/2+1 replicas must be available.
> > >
> > > 2/2+1==2 and I killed 1 of 3, so... don't get it.
> > >
> > >> This means for RF = 2, consistency levels QUORUM and ALL yield the
> same result.
> > >>
> > >> /d
> > >>
> > >> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <
> timo.nentwig@toptarif.de> wrote:
> > >>> Hi!
> > >>>
> > >>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and
> use quorum for writes. But when I shut down one of them
> UnavailableExceptions are thrown. Why is that? Isn't that the sense of
> quorum and a fault-tolerant DB that it continues with the remaining 2 nodes
> and redistributes the data to the broken one as soons as its up again?
> > >>>
> > >>> What may I be doing wrong?
> > >>>
> > >>> thx
> > >>> tcn
> > >
> > >
> >
>
>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Nick Bailey <ni...@riptano.com>.
On Thu, Dec 9, 2010 at 10:43 AM, Timo Nentwig <ti...@toptarif.de>wrote:
>
> On Dec 9, 2010, at 17:39, David Boxenhorn wrote:
>
> > In other words, if you want to use QUORUM, you need to set RF>=3.
> >
> > (I know because I had exactly the same problem.)
>
> I naively assume that if I kill either node that holds N1 (i.e. node 1 or
> 3), N1 will still remain on another node. Only if both fail, I actually lose
> data. But apparently this is not how it works...
>
No this is correct. Killing one node with a replication factor of 2 will
not cause you to lose data. You are requiring a consistency level higher
than what is available. Change your app to use CL.ONE and all data will be
available even with one machine unavailable.
>
> > On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com>
> wrote:
> > I'ts 2 out of the number of replicas, not the number of nodes. At RF=2,
> you have
> > 2 replicas. And since quorum is also 2 with that replication factor,
> > you cannot lose
> > a node, otherwise some query will end up as UnavailableException.
> >
> > Again, this is not related to the total number of nodes. Even with 200
> > nodes, if
> > you use RF=2, you will have some query that fail (altough much less that
> what
> > you are probably seeing).
> >
> > On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de>
> wrote:
> > >
> > > On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
> > >
> > >> Quorum is really only useful when RF > 2, since the for a quorum to
> > >> succeed RF/2+1 replicas must be available.
> > >
> > > 2/2+1==2 and I killed 1 of 3, so... don't get it.
> > >
> > >> This means for RF = 2, consistency levels QUORUM and ALL yield the
> same result.
> > >>
> > >> /d
> > >>
> > >> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <
> timo.nentwig@toptarif.de> wrote:
> > >>> Hi!
> > >>>
> > >>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and
> use quorum for writes. But when I shut down one of them
> UnavailableExceptions are thrown. Why is that? Isn't that the sense of
> quorum and a fault-tolerant DB that it continues with the remaining 2 nodes
> and redistributes the data to the broken one as soons as its up again?
> > >>>
> > >>> What may I be doing wrong?
> > >>>
> > >>> thx
> > >>> tcn
> > >
> > >
> >
>
>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Timo Nentwig <ti...@toptarif.de>.
On Dec 9, 2010, at 17:39, David Boxenhorn wrote:
> In other words, if you want to use QUORUM, you need to set RF>=3.
>
> (I know because I had exactly the same problem.)
I naively assume that if I kill either node that holds N1 (i.e. node 1 or 3), N1 will still remain on another node. Only if both fail, I actually lose data. But apparently this is not how it works...
> On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
> I'ts 2 out of the number of replicas, not the number of nodes. At RF=2, you have
> 2 replicas. And since quorum is also 2 with that replication factor,
> you cannot lose
> a node, otherwise some query will end up as UnavailableException.
>
> Again, this is not related to the total number of nodes. Even with 200
> nodes, if
> you use RF=2, you will have some query that fail (altough much less that what
> you are probably seeing).
>
> On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de> wrote:
> >
> > On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
> >
> >> Quorum is really only useful when RF > 2, since the for a quorum to
> >> succeed RF/2+1 replicas must be available.
> >
> > 2/2+1==2 and I killed 1 of 3, so... don't get it.
> >
> >> This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
> >>
> >> /d
> >>
> >> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
> >>> Hi!
> >>>
> >>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
> >>>
> >>> What may I be doing wrong?
> >>>
> >>> thx
> >>> tcn
> >
> >
>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by David Boxenhorn <da...@lookin2.com>.
In other words, if you want to use QUORUM, you need to set RF>=3.
(I know because I had exactly the same problem.)
On Thu, Dec 9, 2010 at 6:05 PM, Sylvain Lebresne <sy...@yakaz.com> wrote:
> I'ts 2 out of the number of replicas, not the number of nodes. At RF=2, you
> have
> 2 replicas. And since quorum is also 2 with that replication factor,
> you cannot lose
> a node, otherwise some query will end up as UnavailableException.
>
> Again, this is not related to the total number of nodes. Even with 200
> nodes, if
> you use RF=2, you will have some query that fail (altough much less that
> what
> you are probably seeing).
>
> On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de>
> wrote:
> >
> > On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
> >
> >> Quorum is really only useful when RF > 2, since the for a quorum to
> >> succeed RF/2+1 replicas must be available.
> >
> > 2/2+1==2 and I killed 1 of 3, so... don't get it.
> >
> >> This means for RF = 2, consistency levels QUORUM and ALL yield the same
> result.
> >>
> >> /d
> >>
> >> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de>
> wrote:
> >>> Hi!
> >>>
> >>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use
> quorum for writes. But when I shut down one of them UnavailableExceptions
> are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant
> DB that it continues with the remaining 2 nodes and redistributes the data
> to the broken one as soons as its up again?
> >>>
> >>> What may I be doing wrong?
> >>>
> >>> thx
> >>> tcn
> >
> >
>
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Sylvain Lebresne <sy...@yakaz.com>.
I'ts 2 out of the number of replicas, not the number of nodes. At RF=2, you have
2 replicas. And since quorum is also 2 with that replication factor,
you cannot lose
a node, otherwise some query will end up as UnavailableException.
Again, this is not related to the total number of nodes. Even with 200
nodes, if
you use RF=2, you will have some query that fail (altough much less that what
you are probably seeing).
On Thu, Dec 9, 2010 at 5:00 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>
> On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
>
>> Quorum is really only useful when RF > 2, since the for a quorum to
>> succeed RF/2+1 replicas must be available.
>
> 2/2+1==2 and I killed 1 of 3, so... don't get it.
>
>> This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
>>
>> /d
>>
>> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>>> Hi!
>>>
>>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
>>>
>>> What may I be doing wrong?
>>>
>>> thx
>>> tcn
>
>
RE: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Viktor Jevdokimov <Vi...@adform.com>.
With 3 nodes and RF=2 you have 3 key ranges: N1+N2, N2+N3 and N3+N1.
Killing N1 you've got only 1 alive range N2+N3 and 2/3 of the range is down for Quorum, which is actually all, so N1+N2 and N3+N1 fails.
-----Original Message-----
From: Timo Nentwig [mailto:timo.nentwig@toptarif.de]
Sent: Thursday, December 09, 2010 6:01 PM
To: user@cassandra.apache.org
Subject: Re: Quorum: killing 1 out of 3 server kills the cluster (?)
On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
> Quorum is really only useful when RF > 2, since the for a quorum to
> succeed RF/2+1 replicas must be available.
2/2+1==2 and I killed 1 of 3, so... don't get it.
> This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
>
> /d
>
> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>> Hi!
>>
>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
>>
>> What may I be doing wrong?
>>
>> thx
>> tcn
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Timo Nentwig <ti...@toptarif.de>.
On Dec 9, 2010, at 16:50, Daniel Lundin wrote:
> Quorum is really only useful when RF > 2, since the for a quorum to
> succeed RF/2+1 replicas must be available.
2/2+1==2 and I killed 1 of 3, so... don't get it.
> This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
>
> /d
>
> On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
>> Hi!
>>
>> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
>>
>> What may I be doing wrong?
>>
>> thx
>> tcn
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Daniel Lundin <dl...@eintr.org>.
Quorum is really only useful when RF > 2, since the for a quorum to
succeed RF/2+1 replicas must be available.
This means for RF = 2, consistency levels QUORUM and ALL yield the same result.
/d
On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
> Hi!
>
> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
>
> What may I be doing wrong?
>
> thx
> tcn
Re: Quorum: killing 1 out of 3 server kills the cluster (?)
Posted by Thibaut Britz <th...@trendiction.com>.
Hi,
The UnavailableExceptions will be thrown because quorum of size 2
needs at least 2 nodes to be alive (as for qurom of size 3 as well).
The data won't be automatically redistributed to other nodes.
Thibaut
On Thu, Dec 9, 2010 at 4:40 PM, Timo Nentwig <ti...@toptarif.de> wrote:
> Hi!
>
> I've 3 servers running (0.7rc1) with a replication_factor of 2 and use quorum for writes. But when I shut down one of them UnavailableExceptions are thrown. Why is that? Isn't that the sense of quorum and a fault-tolerant DB that it continues with the remaining 2 nodes and redistributes the data to the broken one as soons as its up again?
>
> What may I be doing wrong?
>
> thx
> tcn