You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Logendran, Dharsan (Dharsan)" <dh...@alcatel-lucent.com> on 2014/01/20 23:47:35 UTC
Question about node tool repair
Hi,
We have a two node cluster with the replication factor of 2. The db has more than 2500 column families(tables). The nodetool -pr repair on an empty database(one or table has a litter data) takes about 30 hours to complete. We are using Cassandra Version 2.0.4. Is there any way for us to speed up this?.
Thanks
Dharsan
Re: Question about node tool repair
Posted by sankalp kohli <ko...@gmail.com>.
Can you give the logs of both the machines. Logs will tell why it is taken
so long.
On a side note, you are using 2500 Cfs. I think you need to redesign this
schema.
Also 2 node cluster with RF=2, you might want to add a machine if it is
prod.
On Mon, Jan 20, 2014 at 2:47 PM, Logendran, Dharsan (Dharsan) <
dharsan.logendran@alcatel-lucent.com> wrote:
> Hi,
>
>
>
> We have a two node cluster with the replication factor of 2. The db has
> more than 2500 column families(tables). The nodetool -pr repair on an
> empty database(one or table has a litter data) takes about 30 hours to
> complete. We are using Cassandra Version 2.0.4. Is there any way for us
> to speed up this?.
>
>
>
> Thanks
>
> Dharsan
>
>
Re: Question about node tool repair
Posted by Artur Kronenberg <ar...@openmarket.com>.
About repairs,
we encountered a similar problem with our setup where repairs would take
ages to complete. Based on your setup you can try loading data into page
cache before running repairs. Depending on how much data you can hold in
cache, this will speed up your repairs massively.
-- artur
On 21/01/14 20:33, Logendran, Dharsan (Dharsan) wrote:
>
> Thanks Rob,
>
> Dharsan
>
> *From:*Robert Coli [mailto:rcoli@eventbrite.com]
> *Sent:* January-21-14 2:26 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Question about node tool repair
>
> On Mon, Jan 20, 2014 at 2:47 PM, Logendran, Dharsan (Dharsan)
> <dharsan.logendran@alcatel-lucent.com
> <ma...@alcatel-lucent.com>> wrote:
>
> We have a two node cluster with the replication factor of 2. The db
> has more than 2500 column families(tables). The nodetool -pr repair
> on an empty database(one or table has a litter data) takes about 30
> hours to complete. We are using Cassandra Version 2.0.4. Is there
> any way for us to speed up this?.
>
> Cassandra 2.0.2 made aspects of repair serial and therefore logically
> much slower as a function of replication factor. Yours is not the
> first report I have heard of >= 2.0.2 era repair being unreasonably slow.
>
> https://issues.apache.org/jira/browse/CASSANDRA-5950
>
> You can use -par (not at all confusingly named with -pr!) to get the
> old parallel behavior.
>
> Cassandra 2.1 has this ticket to improve repair with vnodes.
>
> https://issues.apache.org/jira/browse/CASSANDRA-5220
>
> But really you should strongly consider how much you need to run
> repair, and at very least probably increase gc_grace_seconds from the
> unreasonably low default of 10 days to 32 days, and then run your
> repair on the first of each month.
>
> https://issues.apache.org/jira/browse/CASSANDRA-5850
>
> IMO it is just a complete and total error if repair of an actually
> empty database is anything but a NO-OP. I would file a JIRA ticket,
> were I you.
>
> =Rob
>
RE: Question about node tool repair
Posted by "Logendran, Dharsan (Dharsan)" <dh...@alcatel-lucent.com>.
Thanks Rob,
Dharsan
From: Robert Coli [mailto:rcoli@eventbrite.com]
Sent: January-21-14 2:26 PM
To: user@cassandra.apache.org
Subject: Re: Question about node tool repair
On Mon, Jan 20, 2014 at 2:47 PM, Logendran, Dharsan (Dharsan) <dh...@alcatel-lucent.com>> wrote:
We have a two node cluster with the replication factor of 2. The db has more than 2500 column families(tables). The nodetool -pr repair on an empty database(one or table has a litter data) takes about 30 hours to complete. We are using Cassandra Version 2.0.4. Is there any way for us to speed up this?.
Cassandra 2.0.2 made aspects of repair serial and therefore logically much slower as a function of replication factor. Yours is not the first report I have heard of >= 2.0.2 era repair being unreasonably slow.
https://issues.apache.org/jira/browse/CASSANDRA-5950
You can use -par (not at all confusingly named with -pr!) to get the old parallel behavior.
Cassandra 2.1 has this ticket to improve repair with vnodes.
https://issues.apache.org/jira/browse/CASSANDRA-5220
But really you should strongly consider how much you need to run repair, and at very least probably increase gc_grace_seconds from the unreasonably low default of 10 days to 32 days, and then run your repair on the first of each month.
https://issues.apache.org/jira/browse/CASSANDRA-5850
IMO it is just a complete and total error if repair of an actually empty database is anything but a NO-OP. I would file a JIRA ticket, were I you.
=Rob
Re: Question about node tool repair
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Jan 20, 2014 at 2:47 PM, Logendran, Dharsan (Dharsan) <
dharsan.logendran@alcatel-lucent.com> wrote:
> We have a two node cluster with the replication factor of 2. The db
> has more than 2500 column families(tables). The nodetool -pr repair on an
> empty database(one or table has a litter data) takes about 30 hours to
> complete. We are using Cassandra Version 2.0.4. Is there any way for us
> to speed up this?.
>
Cassandra 2.0.2 made aspects of repair serial and therefore logically much
slower as a function of replication factor. Yours is not the first report I
have heard of >= 2.0.2 era repair being unreasonably slow.
https://issues.apache.org/jira/browse/CASSANDRA-5950
You can use -par (not at all confusingly named with -pr!) to get the old
parallel behavior.
Cassandra 2.1 has this ticket to improve repair with vnodes.
https://issues.apache.org/jira/browse/CASSANDRA-5220
But really you should strongly consider how much you need to run repair,
and at very least probably increase gc_grace_seconds from the unreasonably
low default of 10 days to 32 days, and then run your repair on the first of
each month.
https://issues.apache.org/jira/browse/CASSANDRA-5850
IMO it is just a complete and total error if repair of an actually empty
database is anything but a NO-OP. I would file a JIRA ticket, were I you.
=Rob