tuning repairs and compaction options

[Reik Schatz <re...@gmail.com>]

Hi, we are running a 9 node cluster under load. The nodes are running in
EC2 on i2.2xlarge instances. Cassandra version is 2.2.4. One node was down
yesterday for more than 3 hours. So we manually started an incremental
repair this morning via nodetool (anti-entropy repair?)

What we can see is that user CPU on that node goes up to over 95% and also
goes up on all other nodes. Also the number of SSTables is exploding, I
guess due to anticompaction.

What are my tuning options to have a more gentle repair behaviour? Which
settings should I look at if I want CPU to stay below 50% for instance. My
worry is always to impact the read/write performance during times when we
do anti-entropy repairs.

Cheers,
Reik

Re: tuning repairs and compaction options

[Jens Rantil <je...@tink.se>]

Hi Reik,

You could always throttle your repair by running smaller chunks of the
repair. See https://github.com/BrianGallew/cassandra_range_repair.

Regarding the compaction, you can always change the compactionthroughput
using `nodetool setcompactionthroughput`.

Hope this helps,
Jens

On Fri, May 6, 2016 at 9:47 AM Reik Schatz <re...@gmail.com> wrote:

> Hi, we are running a 9 node cluster under load. The nodes are running in
> EC2 on i2.2xlarge instances. Cassandra version is 2.2.4. One node was down
> yesterday for more than 3 hours. So we manually started an incremental
> repair this morning via nodetool (anti-entropy repair?)
>
> What we can see is that user CPU on that node goes up to over 95% and also
> goes up on all other nodes. Also the number of SSTables is exploding, I
> guess due to anticompaction.
>
> What are my tuning options to have a more gentle repair behaviour? Which
> settings should I look at if I want CPU to stay below 50% for instance. My
> worry is always to impact the read/write performance during times when we
> do anti-entropy repairs.
>
> Cheers,
> Reik
>
-- 

Jens Rantil
Backend Developer @ Tink

Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.

Re: tuning repairs and compaction options

[Nate McCall <na...@thelastpickle.com>]

>
> Hi, we are running a 9 node cluster under load. The nodes are running in
> EC2 on i2.2xlarge instances. Cassandra version is 2.2.4. One node was down
> yesterday for more than 3 hours. So we manually started an incremental
> repair this morning via nodetool (anti-entropy repair?)
>
> What we can see is that user CPU on that node goes up to over 95% and also
> goes up on all other nodes. Also the number of SSTables is exploding, I
> guess due to anticompaction.
>

You might be seeing https://issues.apache.org/jira/browse/CASSANDRA-10342
(fixed in 2.2.6).


>
> What are my tuning options to have a more gentle repair behaviour? Which
> settings should I look at if I want CPU to stay below 50% for instance. My
> worry is always to impact the read/write performance during times when we
> do anti-entropy repairs.
>

+1 on cassandra_range_repair script.




-- 
-----------------
Nate McCall
Austin, TX
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com