You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Hefeng Yuan <hf...@rhapsody.com> on 2011/08/22 07:22:25 UTC
Avoid Simultaneous Minor Compactions?
We just noticed that at one time, 4 nodes were doing minor compaction together, each of them took 20~60 minutes.
We're on 0.8.1, 6 nodes, RF5.
This simultaneous compactions slowed down the whole cluster, we have local_quorum consistency level, therefore, dynamic_snitch is not helping us.
Aside from lower down the compaction thread priority, is there any other way to tell the cluster hold on doing this if other nodes are already compacting?
Thanks,
Hefeng
Re: Avoid Simultaneous Minor Compactions?
Posted by aaron morton <aa...@thelastpickle.com>.
Change one thing at a time and work out what metric it is you want to improve.
I would start with reducing compaction_throughput_mb_per_sec. Have a look in your logs for the "Enqueuing flush of Memtable…" messages, count up how many serialised bytes you are flushing and then check it against the advice in the yaml file.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 23/08/2011, at 11:36 AM, Hefeng Yuan wrote:
> Shall I lower this or increase it? Or probably ask in this way, do we suggest to let it run longer while using less CPU, or should we let it finish faster with more CPU usage?
> The problem we're facing is, with the default setting, they run slow and also eat a lot of CPU in the meanwhile.
>
> I'm thinking about the following changes, does this make sense?
> 1. lower the compaction thread priority
> 2. shorten the compaction threshold to 2~20
> 3. lower compaction_throughput_mb_per_sec to 10
>
> Thanks,
> Hefeng
>
> On Aug 22, 2011, at 8:09 AM, Jonathan Ellis wrote:
>
>> Specifically, look at compaction_throughput_mb_per_sec in cassandra.yaml
>>
>> On Mon, Aug 22, 2011 at 12:39 AM, Ryan King <ry...@twitter.com> wrote:
>>> You should throttle your compactions to a sustainable level.
>>>
>>> -ryan
>>>
>>> On Sun, Aug 21, 2011 at 10:22 PM, Hefeng Yuan <hf...@rhapsody.com> wrote:
>>>> We just noticed that at one time, 4 nodes were doing minor compaction together, each of them took 20~60 minutes.
>>>> We're on 0.8.1, 6 nodes, RF5.
>>>> This simultaneous compactions slowed down the whole cluster, we have local_quorum consistency level, therefore, dynamic_snitch is not helping us.
>>>>
>>>> Aside from lower down the compaction thread priority, is there any other way to tell the cluster hold on doing this if other nodes are already compacting?
>>>>
>>>> Thanks,
>>>> Hefeng
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
Re: Avoid Simultaneous Minor Compactions?
Posted by Hefeng Yuan <hf...@rhapsody.com>.
Shall I lower this or increase it? Or probably ask in this way, do we suggest to let it run longer while using less CPU, or should we let it finish faster with more CPU usage?
The problem we're facing is, with the default setting, they run slow and also eat a lot of CPU in the meanwhile.
I'm thinking about the following changes, does this make sense?
1. lower the compaction thread priority
2. shorten the compaction threshold to 2~20
3. lower compaction_throughput_mb_per_sec to 10
Thanks,
Hefeng
On Aug 22, 2011, at 8:09 AM, Jonathan Ellis wrote:
> Specifically, look at compaction_throughput_mb_per_sec in cassandra.yaml
>
> On Mon, Aug 22, 2011 at 12:39 AM, Ryan King <ry...@twitter.com> wrote:
>> You should throttle your compactions to a sustainable level.
>>
>> -ryan
>>
>> On Sun, Aug 21, 2011 at 10:22 PM, Hefeng Yuan <hf...@rhapsody.com> wrote:
>>> We just noticed that at one time, 4 nodes were doing minor compaction together, each of them took 20~60 minutes.
>>> We're on 0.8.1, 6 nodes, RF5.
>>> This simultaneous compactions slowed down the whole cluster, we have local_quorum consistency level, therefore, dynamic_snitch is not helping us.
>>>
>>> Aside from lower down the compaction thread priority, is there any other way to tell the cluster hold on doing this if other nodes are already compacting?
>>>
>>> Thanks,
>>> Hefeng
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: Avoid Simultaneous Minor Compactions?
Posted by Jonathan Ellis <jb...@gmail.com>.
Specifically, look at compaction_throughput_mb_per_sec in cassandra.yaml
On Mon, Aug 22, 2011 at 12:39 AM, Ryan King <ry...@twitter.com> wrote:
> You should throttle your compactions to a sustainable level.
>
> -ryan
>
> On Sun, Aug 21, 2011 at 10:22 PM, Hefeng Yuan <hf...@rhapsody.com> wrote:
>> We just noticed that at one time, 4 nodes were doing minor compaction together, each of them took 20~60 minutes.
>> We're on 0.8.1, 6 nodes, RF5.
>> This simultaneous compactions slowed down the whole cluster, we have local_quorum consistency level, therefore, dynamic_snitch is not helping us.
>>
>> Aside from lower down the compaction thread priority, is there any other way to tell the cluster hold on doing this if other nodes are already compacting?
>>
>> Thanks,
>> Hefeng
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Avoid Simultaneous Minor Compactions?
Posted by Ryan King <ry...@twitter.com>.
You should throttle your compactions to a sustainable level.
-ryan
On Sun, Aug 21, 2011 at 10:22 PM, Hefeng Yuan <hf...@rhapsody.com> wrote:
> We just noticed that at one time, 4 nodes were doing minor compaction together, each of them took 20~60 minutes.
> We're on 0.8.1, 6 nodes, RF5.
> This simultaneous compactions slowed down the whole cluster, we have local_quorum consistency level, therefore, dynamic_snitch is not helping us.
>
> Aside from lower down the compaction thread priority, is there any other way to tell the cluster hold on doing this if other nodes are already compacting?
>
> Thanks,
> Hefeng