You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nikolay Kоvshov <nk...@yandex.ru> on 2012/02/13 14:06:45 UTC
How to bring cluster to consistency
Hello everybody
I have a very simple cluster containing 2 servers. Replication_factor = 2, Consistency_level of reads and writes = 1
10.111.1.141 datacenter1 rack1 Up Normal 1.5 TB 100.00% vjpigMzv4KkX3x7z
10.111.1.142 datacenter1 rack1 Up Normal 1.41 TB 0.00% 聶jpigMzv4KkX3x7z
Note the size please.
Say, server1 cassandra dies and I restart it later. Hinted_handoff = enabled, auto_bootstrap = true
During that time server2 received reads and writes. I want changes to be copied to server1 when it joins the cluster. As I have replication_factor 2, I suppose each data piece should be stored on both servers. Auto_bootstrapping doesn't seem to work that way - changed data doesn't migrate.
I run nodetool repair and it is always killed by OOM. What else can I do to bring cluster to consistency?
Thank you in advance
Re: How to bring cluster to consistency
Posted by aaron morton <aa...@thelastpickle.com>.
> Sorry if this is a 4th copy of letter, but cassandra.apache.org constantly tells me that my message looks like spam…
Send as text.
What version are you using ?
It looks like you are using the ByteOrderedPartitioner , is that correct ?
I would try to get the repair done first, what was the error ?
> I believe it is affected by data size. At least some estimation about how much time and memory it could take would be of use.
You can try to reduce the memory requirements of repair by reducing the in_memory_compaction_limit. How much memory do you have allocated to cassandra ?
BUT… 1.5T of data is a lot for a single node. Most people work with around 200G to 400G per node. Otherwise things like repair and move take a very long time. There are also a number of structures (bloom filters, index samples) maintained in memory that vary with respect to the data load.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 14/02/2012, at 2:35 AM, Nikolay Kоvshov wrote:
> Sorry if this is a 4th copy of letter, but cassandra.apache.org constantly tells me that my message looks like spam...
>
>> 2/ both of your nodes seem to be using the same token? The output indicates that 100% of your key range is assigned to 10.111.1.141 (and therefore 10.111.1.142 holds replicas only)
>
> Well, I didn't assign anything. I just filled nodes with data, that's Cassandra itself who assigned that. I am trying to perform nodetool move now. Still I didn't understand from wiki what that means (keys assigned to servers). When both servers are up I can write to 1 and read from 2, or I can write to 2 and read from 1 and all works perfect.
>
>> 3/ maybe repair is being affected by above, but in my experience it can be sensitive
>
> I believe it is affected by data size. At least some estimation about how much time and memory it could take would be of use.
>
> 13.02.2012, 17:19, "Dominic Williams" <dw...@fightmymonster.com>:
>> Hi Nikolay,Some points that may be useful:
>> 1/ auto_bootstrap = true is used for telling a new node to join the ring (the cluster). It has nothing to do with hinted handoff
>> 2/ both of your nodes seem to be using the same token? The output indicates that 100% of your key range is assigned to 10.111.1.141 (and therefore 10.111.1.142 holds replicas only)
>> 3/ maybe repair is being affected by above, but in my experience it can be sensitive
>>
>> On 13 February 2012 13:06, Nikolay Kоvshov <nk...@yandex.ru> wrote:
>>> Hello everybody
>>>
>>> I have a very simple cluster containing 2 servers. Replication_factor = 2, Consistency_level of reads and writes = 1
>>>
>>> 10.111.1.141 datacenter1 rack1 Up Normal 1.5 TB 100.00% vjpigMzv4KkX3x7z
>>> 10.111.1.142 datacenter1 rack1 Up Normal 1.41 TB 0.00% 聶jpigMzv4KkX3x7z
>>>
>>> Note the size please.
>>>
>>> Say, server1 cassandra dies and I restart it later. Hinted_handoff = enabled, auto_bootstrap = true
>>>
>>> During that time server2 received reads and writes. I want changes to be copied to server1 when it joins the cluster. As I have replication_factor 2, I suppose each data piece should be stored on both servers. Auto_bootstrapping doesn't seem to work that way - changed data doesn't migrate.
>>>
>>> I run nodetool repair and it is always killed by OOM. What else can I do to bring cluster to consistency?
>>>
>>> Thank you in advance
Re: How to bring cluster to consistency
Posted by Nikolay Kоvshov <nk...@yandex.ru>.
Sorry if this is a 4th copy of letter, but cassandra.apache.org constantly tells me that my message looks like spam...
> 2/ both of your nodes seem to be using the same token? The output indicates that 100% of your key range is assigned to 10.111.1.141 (and therefore 10.111.1.142 holds replicas only)
Well, I didn't assign anything. I just filled nodes with data, that's Cassandra itself who assigned that. I am trying to perform nodetool move now. Still I didn't understand from wiki what that means (keys assigned to servers). When both servers are up I can write to 1 and read from 2, or I can write to 2 and read from 1 and all works perfect.
> 3/ maybe repair is being affected by above, but in my experience it can be sensitive
I believe it is affected by data size. At least some estimation about how much time and memory it could take would be of use.
13.02.2012, 17:19, "Dominic Williams" <dw...@fightmymonster.com>:
> Hi Nikolay,Some points that may be useful:
> 1/ auto_bootstrap = true is used for telling a new node to join the ring (the cluster). It has nothing to do with hinted handoff
> 2/ both of your nodes seem to be using the same token? The output indicates that 100% of your key range is assigned to 10.111.1.141 (and therefore 10.111.1.142 holds replicas only)
> 3/ maybe repair is being affected by above, but in my experience it can be sensitive
>
> On 13 February 2012 13:06, Nikolay Kоvshov <nk...@yandex.ru> wrote:
>> Hello everybody
>>
>> I have a very simple cluster containing 2 servers. Replication_factor = 2, Consistency_level of reads and writes = 1
>>
>> 10.111.1.141 datacenter1 rack1 Up Normal 1.5 TB 100.00% vjpigMzv4KkX3x7z
>> 10.111.1.142 datacenter1 rack1 Up Normal 1.41 TB 0.00% 聶jpigMzv4KkX3x7z
>>
>> Note the size please.
>>
>> Say, server1 cassandra dies and I restart it later. Hinted_handoff = enabled, auto_bootstrap = true
>>
>> During that time server2 received reads and writes. I want changes to be copied to server1 when it joins the cluster. As I have replication_factor 2, I suppose each data piece should be stored on both servers. Auto_bootstrapping doesn't seem to work that way - changed data doesn't migrate.
>>
>> I run nodetool repair and it is always killed by OOM. What else can I do to bring cluster to consistency?
>>
>> Thank you in advance
Re: How to bring cluster to consistency
Posted by Dominic Williams <dw...@fightmymonster.com>.
Hi Nikolay,
Some points that may be useful:
1/ auto_bootstrap = true is used for telling a new node to join the ring
(the cluster). It has nothing to do with hinted handoff
2/ both of your nodes seem to be using the same token? The output indicates
that 100% of your key range is assigned to 10.111.1.141 (and
therefore 10.111.1.142 holds replicas only)
3/ maybe repair is being affected by above, but in my experience it can be
sensitive
On 13 February 2012 13:06, Nikolay Kоvshov <nk...@yandex.ru> wrote:
> Hello everybody
>
> I have a very simple cluster containing 2 servers. Replication_factor = 2,
> Consistency_level of reads and writes = 1
>
> 10.111.1.141 datacenter1 rack1 Up Normal 1.5 TB
> 100.00% vjpigMzv4KkX3x7z
> 10.111.1.142 datacenter1 rack1 Up Normal 1.41 TB
> 0.00% 聶jpigMzv4KkX3x7z
>
> Note the size please.
>
> Say, server1 cassandra dies and I restart it later. Hinted_handoff =
> enabled, auto_bootstrap = true
>
> During that time server2 received reads and writes. I want changes to be
> copied to server1 when it joins the cluster. As I have replication_factor
> 2, I suppose each data piece should be stored on both servers.
> Auto_bootstrapping doesn't seem to work that way - changed data doesn't
> migrate.
>
> I run nodetool repair and it is always killed by OOM. What else can I do
> to bring cluster to consistency?
>
> Thank you in advance
>