You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David Boxenhorn <da...@lookin2.com> on 2010/11/28 15:32:20 UTC
Taking down a node in a 3-node cluster, RF=2
For the vast majority of my data usage eventual consistency is fine (i.e.
CL=ONE) but I have a small amount of critical data for which I read and
write using CL=QUORUM.
If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean that
a value can be read from or written to any 2 nodes, or does it have to be
the particular 2 nodes that store the data? If it is the particular 2 nodes
that store the data, that means that I can't even take down one node, since
it will be the mandatory 2nd node for 1/3 of my data...
Re: Taking down a node in a 3-node cluster, RF=2
Posted by Jonathan Ellis <jb...@gmail.com>.
Put another way, "for RF=2, QUORUM is equivalent to ALL."
On Sun, Nov 28, 2010 at 12:03 PM, David Boxenhorn <da...@lookin2.com> wrote:
> OK. To sum up: RF=2 and QUORUM are incompatible (if you want to be able to
> take a node down).
>
> Right?
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Taking down a node in a 3-node cluster, RF=2
Posted by David Boxenhorn <da...@lookin2.com>.
For that matter, RF=1 and QUORUM are incompatible (if you want to be able to
take a node down).
In other words, if you use QUORUM, you need RF>=3.
On Sun, Nov 28, 2010 at 8:04 PM, Jake Luciani <ja...@gmail.com> wrote:
> Right.
>
>
> On Sun, Nov 28, 2010 at 1:03 PM, David Boxenhorn <da...@lookin2.com>wrote:
>
>> OK. To sum up: RF=2 and QUORUM are incompatible (if you want to be able to
>> take a node down).
>>
>> Right?
>>
>> On Sun, Nov 28, 2010 at 7:59 PM, Jake Luciani <ja...@gmail.com> wrote:
>>
>>> I was wrong on this scenario and I'll explain where I was incorrect.
>>>
>>> Hints are stored for a downed node but they don't count towards meeting a
>>> consistency level.
>>> Let's take 2 scenarios:
>>>
>>> RF=6, Nodes=10
>>>
>>> If you READ/WRITE with CL.QUORUM you will need 4 alive nodes if one is
>>> down it will still have 4 active replicas to write to, one of these will
>>> store a hint and update the downed node when it comes back.
>>>
>>> RF=2, Nodes=3
>>>
>>> If you READ/WRITE with CL.QUORUM you need 2 live nodes. If one of these
>>> 2 are down you can't meet the QUORUM level so the write will fail.
>>>
>>> In your scenario your best bet is to update to RF=3, then any two nodes
>>> will accept QUORUM
>>>
>>> Sorry for the confusion,
>>>
>>> -Jake
>>>
>>> On Sun, Nov 28, 2010 at 12:26 PM, David Boxenhorn <da...@lookin2.com>wrote:
>>>
>>>> Thank you, Jake. It does... except that in another context you told me:
>>>>
>>>> Hints only happen when a node is unavailable and you are writing with
>>>> CL.ANY
>>>> If you never write with CL.ANY then you can turn off hinted handoff.
>>>>
>>>> How do I reconcile this?
>>>>
>>>>
>>>> On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani <ja...@gmail.com> wrote:
>>>>
>>>>> If you read/write data with quorum then you can safely take a node down
>>>>> in this scenario. Subsequent writes will use hinted handoff to be passed to
>>>>> the node when it comes back up.
>>>>>
>>>>> More info is here: http://wiki.apache.org/cassandra/HintedHandoff
>>>>>
>>>>> Does that answer your question?
>>>>>
>>>>> -Jake
>>>>>
>>>>>
>>>>> On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory <ra...@gmail.com> wrote:
>>>>>
>>>>>> to me it makes sense that if hinted handoff is off then cassandra
>>>>>> cannot satisfy 2 out of every 3rd writes writes when one of the nodes is
>>>>>> down since this node is the designated node of 2/3 writes.
>>>>>> But I don't remember reading this somewhere. Does hinted handoff
>>>>>> affect David's situation?
>>>>>> (David, did you disable HH in your storage-config?
>>>>>> <HintedHandoffEnabled>false</HintedHandoffEnabled>)
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com>wrote:
>>>>>>
>>>>>>> For the vast majority of my data usage eventual consistency is fine
>>>>>>> (i.e. CL=ONE) but I have a small amount of critical data for which I read
>>>>>>> and write using CL=QUORUM.
>>>>>>>
>>>>>>> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that
>>>>>>> mean that a value can be read from or written to any 2 nodes, or does it
>>>>>>> have to be the particular 2 nodes that store the data? If it is the
>>>>>>> particular 2 nodes that store the data, that means that I can't even take
>>>>>>> down one node, since it will be the mandatory 2nd node for 1/3 of my data...
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> /Ran
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Taking down a node in a 3-node cluster, RF=2
Posted by Jake Luciani <ja...@gmail.com>.
Right.
On Sun, Nov 28, 2010 at 1:03 PM, David Boxenhorn <da...@lookin2.com> wrote:
> OK. To sum up: RF=2 and QUORUM are incompatible (if you want to be able to
> take a node down).
>
> Right?
>
> On Sun, Nov 28, 2010 at 7:59 PM, Jake Luciani <ja...@gmail.com> wrote:
>
>> I was wrong on this scenario and I'll explain where I was incorrect.
>>
>> Hints are stored for a downed node but they don't count towards meeting a
>> consistency level.
>> Let's take 2 scenarios:
>>
>> RF=6, Nodes=10
>>
>> If you READ/WRITE with CL.QUORUM you will need 4 alive nodes if one is
>> down it will still have 4 active replicas to write to, one of these will
>> store a hint and update the downed node when it comes back.
>>
>> RF=2, Nodes=3
>>
>> If you READ/WRITE with CL.QUORUM you need 2 live nodes. If one of these 2
>> are down you can't meet the QUORUM level so the write will fail.
>>
>> In your scenario your best bet is to update to RF=3, then any two nodes
>> will accept QUORUM
>>
>> Sorry for the confusion,
>>
>> -Jake
>>
>> On Sun, Nov 28, 2010 at 12:26 PM, David Boxenhorn <da...@lookin2.com>wrote:
>>
>>> Thank you, Jake. It does... except that in another context you told me:
>>>
>>> Hints only happen when a node is unavailable and you are writing with
>>> CL.ANY
>>> If you never write with CL.ANY then you can turn off hinted handoff.
>>>
>>> How do I reconcile this?
>>>
>>>
>>> On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani <ja...@gmail.com> wrote:
>>>
>>>> If you read/write data with quorum then you can safely take a node down
>>>> in this scenario. Subsequent writes will use hinted handoff to be passed to
>>>> the node when it comes back up.
>>>>
>>>> More info is here: http://wiki.apache.org/cassandra/HintedHandoff
>>>>
>>>> Does that answer your question?
>>>>
>>>> -Jake
>>>>
>>>>
>>>> On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory <ra...@gmail.com> wrote:
>>>>
>>>>> to me it makes sense that if hinted handoff is off then cassandra
>>>>> cannot satisfy 2 out of every 3rd writes writes when one of the nodes is
>>>>> down since this node is the designated node of 2/3 writes.
>>>>> But I don't remember reading this somewhere. Does hinted handoff affect
>>>>> David's situation?
>>>>> (David, did you disable HH in your storage-config?
>>>>> <HintedHandoffEnabled>false</HintedHandoffEnabled>)
>>>>>
>>>>>
>>>>> On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com>wrote:
>>>>>
>>>>>> For the vast majority of my data usage eventual consistency is fine
>>>>>> (i.e. CL=ONE) but I have a small amount of critical data for which I read
>>>>>> and write using CL=QUORUM.
>>>>>>
>>>>>> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that
>>>>>> mean that a value can be read from or written to any 2 nodes, or does it
>>>>>> have to be the particular 2 nodes that store the data? If it is the
>>>>>> particular 2 nodes that store the data, that means that I can't even take
>>>>>> down one node, since it will be the mandatory 2nd node for 1/3 of my data...
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> /Ran
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Taking down a node in a 3-node cluster, RF=2
Posted by David Boxenhorn <da...@lookin2.com>.
OK. To sum up: RF=2 and QUORUM are incompatible (if you want to be able to
take a node down).
Right?
On Sun, Nov 28, 2010 at 7:59 PM, Jake Luciani <ja...@gmail.com> wrote:
> I was wrong on this scenario and I'll explain where I was incorrect.
>
> Hints are stored for a downed node but they don't count towards meeting a
> consistency level.
> Let's take 2 scenarios:
>
> RF=6, Nodes=10
>
> If you READ/WRITE with CL.QUORUM you will need 4 alive nodes if one is down
> it will still have 4 active replicas to write to, one of these will store a
> hint and update the downed node when it comes back.
>
> RF=2, Nodes=3
>
> If you READ/WRITE with CL.QUORUM you need 2 live nodes. If one of these 2
> are down you can't meet the QUORUM level so the write will fail.
>
> In your scenario your best bet is to update to RF=3, then any two nodes
> will accept QUORUM
>
> Sorry for the confusion,
>
> -Jake
>
> On Sun, Nov 28, 2010 at 12:26 PM, David Boxenhorn <da...@lookin2.com>wrote:
>
>> Thank you, Jake. It does... except that in another context you told me:
>>
>> Hints only happen when a node is unavailable and you are writing with
>> CL.ANY
>> If you never write with CL.ANY then you can turn off hinted handoff.
>>
>> How do I reconcile this?
>>
>>
>> On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani <ja...@gmail.com> wrote:
>>
>>> If you read/write data with quorum then you can safely take a node down
>>> in this scenario. Subsequent writes will use hinted handoff to be passed to
>>> the node when it comes back up.
>>>
>>> More info is here: http://wiki.apache.org/cassandra/HintedHandoff
>>>
>>> Does that answer your question?
>>>
>>> -Jake
>>>
>>>
>>> On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory <ra...@gmail.com> wrote:
>>>
>>>> to me it makes sense that if hinted handoff is off then cassandra cannot
>>>> satisfy 2 out of every 3rd writes writes when one of the nodes is down since
>>>> this node is the designated node of 2/3 writes.
>>>> But I don't remember reading this somewhere. Does hinted handoff affect
>>>> David's situation?
>>>> (David, did you disable HH in your storage-config?
>>>> <HintedHandoffEnabled>false</HintedHandoffEnabled>)
>>>>
>>>>
>>>> On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com>wrote:
>>>>
>>>>> For the vast majority of my data usage eventual consistency is fine
>>>>> (i.e. CL=ONE) but I have a small amount of critical data for which I read
>>>>> and write using CL=QUORUM.
>>>>>
>>>>> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean
>>>>> that a value can be read from or written to any 2 nodes, or does it have to
>>>>> be the particular 2 nodes that store the data? If it is the particular 2
>>>>> nodes that store the data, that means that I can't even take down one node,
>>>>> since it will be the mandatory 2nd node for 1/3 of my data...
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> /Ran
>>>>
>>>>
>>>
>>
>
Re: Taking down a node in a 3-node cluster, RF=2
Posted by Jake Luciani <ja...@gmail.com>.
I was wrong on this scenario and I'll explain where I was incorrect.
Hints are stored for a downed node but they don't count towards meeting a
consistency level.
Let's take 2 scenarios:
RF=6, Nodes=10
If you READ/WRITE with CL.QUORUM you will need 4 alive nodes if one is down
it will still have 4 active replicas to write to, one of these will store a
hint and update the downed node when it comes back.
RF=2, Nodes=3
If you READ/WRITE with CL.QUORUM you need 2 live nodes. If one of these 2
are down you can't meet the QUORUM level so the write will fail.
In your scenario your best bet is to update to RF=3, then any two nodes will
accept QUORUM
Sorry for the confusion,
-Jake
On Sun, Nov 28, 2010 at 12:26 PM, David Boxenhorn <da...@lookin2.com> wrote:
> Thank you, Jake. It does... except that in another context you told me:
>
> Hints only happen when a node is unavailable and you are writing with
> CL.ANY
> If you never write with CL.ANY then you can turn off hinted handoff.
>
> How do I reconcile this?
>
>
> On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani <ja...@gmail.com> wrote:
>
>> If you read/write data with quorum then you can safely take a node down in
>> this scenario. Subsequent writes will use hinted handoff to be passed to
>> the node when it comes back up.
>>
>> More info is here: http://wiki.apache.org/cassandra/HintedHandoff
>>
>> Does that answer your question?
>>
>> -Jake
>>
>>
>> On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory <ra...@gmail.com> wrote:
>>
>>> to me it makes sense that if hinted handoff is off then cassandra cannot
>>> satisfy 2 out of every 3rd writes writes when one of the nodes is down since
>>> this node is the designated node of 2/3 writes.
>>> But I don't remember reading this somewhere. Does hinted handoff affect
>>> David's situation?
>>> (David, did you disable HH in your storage-config?
>>> <HintedHandoffEnabled>false</HintedHandoffEnabled>)
>>>
>>>
>>> On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com>wrote:
>>>
>>>> For the vast majority of my data usage eventual consistency is fine
>>>> (i.e. CL=ONE) but I have a small amount of critical data for which I read
>>>> and write using CL=QUORUM.
>>>>
>>>> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean
>>>> that a value can be read from or written to any 2 nodes, or does it have to
>>>> be the particular 2 nodes that store the data? If it is the particular 2
>>>> nodes that store the data, that means that I can't even take down one node,
>>>> since it will be the mandatory 2nd node for 1/3 of my data...
>>>>
>>>
>>>
>>>
>>> --
>>> /Ran
>>>
>>>
>>
>
Re: Taking down a node in a 3-node cluster, RF=2
Posted by Sylvain Lebresne <sy...@yakaz.com>.
On Sun, Nov 28, 2010 at 6:26 PM, David Boxenhorn <da...@lookin2.com> wrote:
> Hints only happen when a node is unavailable and you are writing with CL.ANY
>
> If you never write with CL.ANY then you can turn off hinted handoff.
Hints happens at any consistency level. But they never helps to
reach the consistency level requirements, except at CL.ANY.
To say this otherwise, HH are (only) useful to make dead nodes 'catch up'
quicker when they are back up (that is, without waiting for a read-repair or
a nodetool repair to happen). But they doesn't help a write to work where it
wouldn't have without HH.
Except at CL.ANY. At CL.ANY, even if all the replicas for the data you are
writting are dead, the write is still accepted and a hint will be used.
--
Sylvain
>
> How do I reconcile this?
>
>
> On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani <ja...@gmail.com> wrote:
>>
>> If you read/write data with quorum then you can safely take a node down in
>> this scenario. Subsequent writes will use hinted handoff to be passed to
>> the node when it comes back up.
>> More info is here: http://wiki.apache.org/cassandra/HintedHandoff
>>
>> Does that answer your question?
>> -Jake
>>
>> On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory <ra...@gmail.com> wrote:
>>>
>>> to me it makes sense that if hinted handoff is off then cassandra cannot
>>> satisfy 2 out of every 3rd writes writes when one of the nodes is down since
>>> this node is the designated node of 2/3 writes.
>>> But I don't remember reading this somewhere. Does hinted handoff affect
>>> David's situation?
>>> (David, did you disable HH in your storage-config?
>>> <HintedHandoffEnabled>false</HintedHandoffEnabled>)
>>>
>>> On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com>
>>> wrote:
>>>>
>>>> For the vast majority of my data usage eventual consistency is fine
>>>> (i.e. CL=ONE) but I have a small amount of critical data for which I read
>>>> and write using CL=QUORUM.
>>>>
>>>> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean
>>>> that a value can be read from or written to any 2 nodes, or does it have to
>>>> be the particular 2 nodes that store the data? If it is the particular 2
>>>> nodes that store the data, that means that I can't even take down one node,
>>>> since it will be the mandatory 2nd node for 1/3 of my data...
>>>
>>>
>>>
>>> --
>>> /Ran
>>
>
>
Re: Taking down a node in a 3-node cluster, RF=2
Posted by David Boxenhorn <da...@lookin2.com>.
Thank you, Jake. It does... except that in another context you told me:
Hints only happen when a node is unavailable and you are writing with CL.ANY
If you never write with CL.ANY then you can turn off hinted handoff.
How do I reconcile this?
On Sun, Nov 28, 2010 at 7:11 PM, Jake Luciani <ja...@gmail.com> wrote:
> If you read/write data with quorum then you can safely take a node down in
> this scenario. Subsequent writes will use hinted handoff to be passed to
> the node when it comes back up.
>
> More info is here: http://wiki.apache.org/cassandra/HintedHandoff
>
> Does that answer your question?
>
> -Jake
>
>
> On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory <ra...@gmail.com> wrote:
>
>> to me it makes sense that if hinted handoff is off then cassandra cannot
>> satisfy 2 out of every 3rd writes writes when one of the nodes is down since
>> this node is the designated node of 2/3 writes.
>> But I don't remember reading this somewhere. Does hinted handoff affect
>> David's situation?
>> (David, did you disable HH in your storage-config?
>> <HintedHandoffEnabled>false</HintedHandoffEnabled>)
>>
>>
>> On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com>wrote:
>>
>>> For the vast majority of my data usage eventual consistency is fine (i.e.
>>> CL=ONE) but I have a small amount of critical data for which I read and
>>> write using CL=QUORUM.
>>>
>>> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean
>>> that a value can be read from or written to any 2 nodes, or does it have to
>>> be the particular 2 nodes that store the data? If it is the particular 2
>>> nodes that store the data, that means that I can't even take down one node,
>>> since it will be the mandatory 2nd node for 1/3 of my data...
>>>
>>
>>
>>
>> --
>> /Ran
>>
>>
>
Re: Taking down a node in a 3-node cluster, RF=2
Posted by Sylvain Lebresne <sy...@yakaz.com>.
Actually you can't. As explained in the wiki page linked:
"A hinted write does NOT count towards ConsistencyLevel requirements
of ONE, QUORUM, or ALL"
For CL.QUORUM, you do need QUORUM *replicas* to be alive to answer
the query. At RF=2, QUORUM=2 so no, you cannot take down any node
down or (some) quorum writes/reads will result in UnavailableException. And this
is not related to the number of node you have, only to the replication factor.
If you want to support having a node down, you need to have RF=3. For that
very reason, this is the minimum replication factor I would advise for a
production cluster.
--
Sylvain
On Sun, Nov 28, 2010 at 6:11 PM, Jake Luciani <ja...@gmail.com> wrote:
> If you read/write data with quorum then you can safely take a node down in
> this scenario. Subsequent writes will use hinted handoff to be passed to
> the node when it comes back up.
> More info is here: http://wiki.apache.org/cassandra/HintedHandoff
>
> Does that answer your question?
> -Jake
>
> On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory <ra...@gmail.com> wrote:
>>
>> to me it makes sense that if hinted handoff is off then cassandra cannot
>> satisfy 2 out of every 3rd writes writes when one of the nodes is down since
>> this node is the designated node of 2/3 writes.
>> But I don't remember reading this somewhere. Does hinted handoff affect
>> David's situation?
>> (David, did you disable HH in your storage-config?
>> <HintedHandoffEnabled>false</HintedHandoffEnabled>)
>>
>> On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com>
>> wrote:
>>>
>>> For the vast majority of my data usage eventual consistency is fine (i.e.
>>> CL=ONE) but I have a small amount of critical data for which I read and
>>> write using CL=QUORUM.
>>>
>>> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean
>>> that a value can be read from or written to any 2 nodes, or does it have to
>>> be the particular 2 nodes that store the data? If it is the particular 2
>>> nodes that store the data, that means that I can't even take down one node,
>>> since it will be the mandatory 2nd node for 1/3 of my data...
>>
>>
>>
>> --
>> /Ran
>
>
Re: Taking down a node in a 3-node cluster, RF=2
Posted by Jake Luciani <ja...@gmail.com>.
If you read/write data with quorum then you can safely take a node down in
this scenario. Subsequent writes will use hinted handoff to be passed to
the node when it comes back up.
More info is here: http://wiki.apache.org/cassandra/HintedHandoff
Does that answer your question?
-Jake
On Sun, Nov 28, 2010 at 9:42 AM, Ran Tavory <ra...@gmail.com> wrote:
> to me it makes sense that if hinted handoff is off then cassandra cannot
> satisfy 2 out of every 3rd writes writes when one of the nodes is down since
> this node is the designated node of 2/3 writes.
> But I don't remember reading this somewhere. Does hinted handoff affect
> David's situation?
> (David, did you disable HH in your storage-config?
> <HintedHandoffEnabled>false</HintedHandoffEnabled>)
>
>
> On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com>wrote:
>
>> For the vast majority of my data usage eventual consistency is fine (i.e.
>> CL=ONE) but I have a small amount of critical data for which I read and
>> write using CL=QUORUM.
>>
>> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean
>> that a value can be read from or written to any 2 nodes, or does it have to
>> be the particular 2 nodes that store the data? If it is the particular 2
>> nodes that store the data, that means that I can't even take down one node,
>> since it will be the mandatory 2nd node for 1/3 of my data...
>>
>
>
>
> --
> /Ran
>
>
Re: Taking down a node in a 3-node cluster, RF=2
Posted by Ran Tavory <ra...@gmail.com>.
to me it makes sense that if hinted handoff is off then cassandra cannot
satisfy 2 out of every 3rd writes writes when one of the nodes is down since
this node is the designated node of 2/3 writes.
But I don't remember reading this somewhere. Does hinted handoff affect
David's situation?
(David, did you disable HH in your storage-config?
<HintedHandoffEnabled>false</HintedHandoffEnabled>)
On Sun, Nov 28, 2010 at 4:32 PM, David Boxenhorn <da...@lookin2.com> wrote:
> For the vast majority of my data usage eventual consistency is fine (i.e.
> CL=ONE) but I have a small amount of critical data for which I read and
> write using CL=QUORUM.
>
> If I have a cluster with 3 nodes and RF=2, and CL=QUORUM does that mean
> that a value can be read from or written to any 2 nodes, or does it have to
> be the particular 2 nodes that store the data? If it is the particular 2
> nodes that store the data, that means that I can't even take down one node,
> since it will be the mandatory 2nd node for 1/3 of my data...
>
--
/Ran