You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com> on 2014/08/26 20:38:10 UTC

Too many SSTables after rebalancing cluster (LCS)

Hey folks,

After adding more nodes and moving tokens of "old" nodes to rebalance the
ring, I noticed that the "old" nodes had significant more data then the
newly bootstrapped nodes, even after cleanup.

I noticed that the old nodes had a much larger number of SSTables on LCS
CFs, and most of them located on the last level:

Node N-1 (old node): [1, 10, 102/100, 173, 2403, 0, 0, 0, 0] (total:2695)

*Node N (new node): [1, 10, 108/100, 214, 0, 0, 0, 0, 0] (total: 339)*Node
N+1 (old node): [1, 10, 87, 113, 1076, 0, 0, 0, 0] (total: 1287)

Since these sstables have a lot of tombstones, and they're not updated
frequently, they remain in the last level forever, and are never cleaned.

What is the solution here? The good old "change to STCS and then back to
LCS", or is there something less brute force?

Environment: Cassandra 1.2.16 - non-vnondes

Any help would be very much appreciated.

Cheers,

-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200

Re: Too many SSTables after rebalancing cluster (LCS)

Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
Deleting the json manifest worked like a charm. After 2 days of compactions
I've got 50GB extra space! :)

Just a quick addendum, after deleting the json metadata file, I needed to
restart the node, otherwise it just reloads the file from memory.

Version: 1.2.16

On Wed, Aug 27, 2014 at 8:13 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Wed, Aug 27, 2014 at 3:27 PM, Nate McCall <na...@thelastpickle.com>
> wrote:
>
>> Another option to force things - deleting the json metadata file for that
>> table will cause LCS to put all SSTables in level 0 and begin recompacting
>> them.
>>
>
> That's possible in versions where the level is in a JSON file, which is
> versions before 2.0. In 2.0+ you can use nodetool for the same purpose.
>
> https://issues.apache.org/jira/browse/CASSANDRA-5271 (Fixed; 2.0 beta 1):
> "Create tool to drop sstables to level 0"
>
> =Rob
>
>


-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200

Re: Too many SSTables after rebalancing cluster (LCS)

Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, Aug 27, 2014 at 3:27 PM, Nate McCall <na...@thelastpickle.com> wrote:

> Another option to force things - deleting the json metadata file for that
> table will cause LCS to put all SSTables in level 0 and begin recompacting
> them.
>

That's possible in versions where the level is in a JSON file, which is
versions before 2.0. In 2.0+ you can use nodetool for the same purpose.

https://issues.apache.org/jira/browse/CASSANDRA-5271 (Fixed; 2.0 beta 1):
"Create tool to drop sstables to level 0"

=Rob

Re: Too many SSTables after rebalancing cluster (LCS)

Posted by Nate McCall <na...@thelastpickle.com>.
Another option to force things - deleting the json metadata file for that
table will cause LCS to put all SSTables in level 0 and begin recompacting
them.


On Wed, Aug 27, 2014 at 5:15 PM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:

> Great idea, will try that (right now is 10%, but being more aggressive
> should hopefully work).
>
> Cheers!
>
>
> On Wed, Aug 27, 2014 at 7:02 PM, Nate McCall <na...@thelastpickle.com>
> wrote:
>
>> Try turning down 'tombstone_threshold' to something like '0.05' from it's
>> default of '0.2.' This will cause the SSTable to be considered for
>> tombstone only compactions more frequently (if %5 of the columns are
>> tombstones instead of 20%).
>>
>> For a bit more info, see:
>>
>> http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/compactSubprop.html
>>
>>
>> On Tue, Aug 26, 2014 at 1:38 PM, Paulo Ricardo Motta Gomes <
>> paulo.motta@chaordicsystems.com> wrote:
>>
>>> Hey folks,
>>>
>>> After adding more nodes and moving tokens of "old" nodes to rebalance
>>> the ring, I noticed that the "old" nodes had significant more data then the
>>> newly bootstrapped nodes, even after cleanup.
>>>
>>> I noticed that the old nodes had a much larger number of SSTables on LCS
>>> CFs, and most of them located on the last level:
>>>
>>> Node N-1 (old node): [1, 10, 102/100, 173, 2403, 0, 0, 0, 0] (total:2695)
>>>
>>> *Node N (new node): [1, 10, 108/100, 214, 0, 0, 0, 0, 0] (total: 339)*Node
>>> N+1 (old node): [1, 10, 87, 113, 1076, 0, 0, 0, 0] (total: 1287)
>>>
>>> Since these sstables have a lot of tombstones, and they're not updated
>>> frequently, they remain in the last level forever, and are never cleaned.
>>>
>>> What is the solution here? The good old "change to STCS and then back to
>>> LCS", or is there something less brute force?
>>>
>>> Environment: Cassandra 1.2.16 - non-vnondes
>>>
>>> Any help would be very much appreciated.
>>>
>>> Cheers,
>>>
>>> --
>>> *Paulo Motta*
>>>
>>> Chaordic | *Platform*
>>> *www.chaordic.com.br <http://www.chaordic.com.br/>*
>>> +55 48 3232.3200
>>>
>>
>>
>>
>> --
>> -----------------
>> Nate McCall
>> Austin, TX
>> @zznate
>>
>> Co-Founder & Sr. Technical Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>
>
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
>



-- 
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Too many SSTables after rebalancing cluster (LCS)

Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
Great idea, will try that (right now is 10%, but being more aggressive
should hopefully work).

Cheers!


On Wed, Aug 27, 2014 at 7:02 PM, Nate McCall <na...@thelastpickle.com> wrote:

> Try turning down 'tombstone_threshold' to something like '0.05' from it's
> default of '0.2.' This will cause the SSTable to be considered for
> tombstone only compactions more frequently (if %5 of the columns are
> tombstones instead of 20%).
>
> For a bit more info, see:
>
> http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/compactSubprop.html
>
>
> On Tue, Aug 26, 2014 at 1:38 PM, Paulo Ricardo Motta Gomes <
> paulo.motta@chaordicsystems.com> wrote:
>
>> Hey folks,
>>
>> After adding more nodes and moving tokens of "old" nodes to rebalance the
>> ring, I noticed that the "old" nodes had significant more data then the
>> newly bootstrapped nodes, even after cleanup.
>>
>> I noticed that the old nodes had a much larger number of SSTables on LCS
>> CFs, and most of them located on the last level:
>>
>> Node N-1 (old node): [1, 10, 102/100, 173, 2403, 0, 0, 0, 0] (total:2695)
>>
>> *Node N (new node): [1, 10, 108/100, 214, 0, 0, 0, 0, 0] (total: 339)*Node
>> N+1 (old node): [1, 10, 87, 113, 1076, 0, 0, 0, 0] (total: 1287)
>>
>> Since these sstables have a lot of tombstones, and they're not updated
>> frequently, they remain in the last level forever, and are never cleaned.
>>
>> What is the solution here? The good old "change to STCS and then back to
>> LCS", or is there something less brute force?
>>
>> Environment: Cassandra 1.2.16 - non-vnondes
>>
>> Any help would be very much appreciated.
>>
>> Cheers,
>>
>> --
>> *Paulo Motta*
>>
>> Chaordic | *Platform*
>> *www.chaordic.com.br <http://www.chaordic.com.br/>*
>> +55 48 3232.3200
>>
>
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>



-- 
*Paulo Motta*

Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200

Re: Too many SSTables after rebalancing cluster (LCS)

Posted by Nate McCall <na...@thelastpickle.com>.
Try turning down 'tombstone_threshold' to something like '0.05' from it's
default of '0.2.' This will cause the SSTable to be considered for
tombstone only compactions more frequently (if %5 of the columns are
tombstones instead of 20%).

For a bit more info, see:
http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/compactSubprop.html


On Tue, Aug 26, 2014 at 1:38 PM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:

> Hey folks,
>
> After adding more nodes and moving tokens of "old" nodes to rebalance the
> ring, I noticed that the "old" nodes had significant more data then the
> newly bootstrapped nodes, even after cleanup.
>
> I noticed that the old nodes had a much larger number of SSTables on LCS
> CFs, and most of them located on the last level:
>
> Node N-1 (old node): [1, 10, 102/100, 173, 2403, 0, 0, 0, 0] (total:2695)
>
> *Node N (new node): [1, 10, 108/100, 214, 0, 0, 0, 0, 0] (total: 339)*Node
> N+1 (old node): [1, 10, 87, 113, 1076, 0, 0, 0, 0] (total: 1287)
>
> Since these sstables have a lot of tombstones, and they're not updated
> frequently, they remain in the last level forever, and are never cleaned.
>
> What is the solution here? The good old "change to STCS and then back to
> LCS", or is there something less brute force?
>
> Environment: Cassandra 1.2.16 - non-vnondes
>
> Any help would be very much appreciated.
>
> Cheers,
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
>



-- 
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Too many SSTables after rebalancing cluster (LCS)

Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Aug 26, 2014 at 11:38 AM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:

> What is the solution here? The good old "change to STCS and then back to
> LCS", or is there something less brute force?
>

In theory you could use user defined compaction via JMX, but I'd probably
just switch to STCS, major compact  and switch back to LCS because I don't
fully comprehend if and when user defined compaction and/or even works with
LCS.

=Rob