You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Oleg Dulin <ol...@gmail.com> on 2014/03/09 14:55:55 UTC
need help with Cassandra 1.2 Full GCing -- output of jmap histogram
I am trying to understand why one of my nodes keeps full GC.
I have Xmx set to 8gigs, memtable total size is 2 gigs.
Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Posted by Takenori Sato <ts...@cloudian.com>.
In addition to the suggestions by Jonathan, you can run a user defined
compaction against a particular set of SSTable files, where you want to
remove tombstones.
But to do that, you need to find such an optimal set. Here you can find a
couple of helpful tools.
https://github.com/cloudian/support-tools
On Mon, Mar 10, 2014 at 7:41 PM, Oleg Dulin <ol...@gmail.com> wrote:
> I get that :)
>
> What I'd like to know is how to fix that :)
>
>
> On 2014-03-09 20:24:54 +0000, Takenori Sato said:
>
> You have millions of org.apache.cassandra.db.DeletedColumn instances on
>> the snapshot.
>>
>> This means you have lots of column tombstones, and I guess, which are
>> read into memory by slice query.
>>
>>
>> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <ol...@gmail.com> wrote:
>> I am trying to understand why one of my nodes keeps full GC.
>>
>> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>>
>> Consider the top entries from jmap -histo:live @
>> http://pastebin.com/UaatHfpJ
>>
>> --
>> Regards,
>> Oleg Dulin
>> http://www.olegdulin.com
>>
>
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
>
Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Posted by Jonathan Lacefield <jl...@datastax.com>.
Sorry to hear about the frustration. How often are you deleting data/what
are you setting for ttl on cols?
Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487
<http://www.linkedin.com/in/jlacefield>
<http://www.datastax.com/what-we-offer/products-services/training/virtual-training>
On Tue, Mar 25, 2014 at 4:22 PM, Oleg Dulin <ol...@gmail.com> wrote:
> Sigh, so I am back to where I started from...
>
> I did lower gc_grace...
>
> jmap -histo:live shows heap is stuffed with DeletedColumn and
> ExpiringColumn....
>
> This is extremely frustrating.
>
>
> On 2014-03-11 19:24:50 +0000, Oleg Dulin said:
>
> Good news is that since I lowered gc_grace period it collected over
>> 100Gigs of tombstones and seems much happier now.
>>
>> Oleg
>>
>> On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said:
>>
>> Hello,
>>>
>>> You have several options:
>>>
>>> 1) going forward lower gc_grace_seconds http://www.datastax.com/
>>> documentation/cassandra/1.2/cassandra/configuration/
>>> configStorage_r.html?pagename=docs&version=1.2&file=
>>> configuration/storage_configuration#gc-grace-seconds
>>> - this is very use case specific. Default is 10 days. Some
>>> users will put this at 0 for specific use cases.
>>> 2) you could also lower tombstone compaction threshold and interval to
>>> get tombstone compaction to fire more often on your tables/cfs:
>>> https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436
>>> 3) to clean out old tombstones you could always run a manual
>>> compaction, those these aren't typically recommended though:
>>> http://www.datastax.com/documentation/cassandra/1.2/
>>> cassandra/tools/toolsNodetool_r.html
>>>
>>> For 1 and 2, be sure your disks can keep up with compaction to ensure
>>> tombstone, or other, compaction fires regularly enough to clean out old
>>> tombstones. Also, you probably want to ensure you are using Level
>>> Compaction: http://www.datastax.com/dev/blog/when-to-use-leveled-
>>> compaction. Again, this assumes your disk system can handle the
>>> increased io from Leveled Compaction.
>>>
>>> Also, you may be running into this with the older version of
>>> Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541
>>>
>>> Hope this helps.
>>>
>>> Jonathan
>>>
>>>
>>> Jonathan Lacefield
>>> Solutions Architect, DataStax
>>> (404) 822 3487
>>> <image>
>>>
>>>
>>> <image>
>>>
>>>
>>> On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <ol...@gmail.com>
>>> wrote:
>>> I get that :)
>>>
>>> What I'd like to know is how to fix that :)
>>>
>>> On 2014-03-09 20:24:54 +0000, Takenori Sato said:
>>>
>>> You have millions of org.apache.cassandra.db.DeletedColumn instances on
>>> the snapshot.
>>>
>>> This means you have lots of column tombstones, and I guess, which are
>>> read into memory by slice query.
>>>
>>>
>>> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <ol...@gmail.com>
>>> wrote:
>>> I am trying to understand why one of my nodes keeps full GC.
>>>
>>> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>>>
>>> Consider the top entries from jmap -histo:live @
>>> http://pastebin.com/UaatHfpJ
>>>
>>> --
>>> Regards,
>>> Oleg Dulin
>>> http://www.olegdulin.com
>>>
>>>
>>> --
>>> Regards,
>>> Oleg Dulin
>>> http://www.olegdulin.com
>>>
>>
>> S
>>
>
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
>
Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Posted by Oleg Dulin <ol...@gmail.com>.
Sigh, so I am back to where I started from...
I did lower gc_grace...
jmap -histo:live shows heap is stuffed with DeletedColumn and
ExpiringColumn....
This is extremely frustrating.
On 2014-03-11 19:24:50 +0000, Oleg Dulin said:
> Good news is that since I lowered gc_grace period it collected over
> 100Gigs of tombstones and seems much happier now.
>
> Oleg
>
> On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said:
>
>> Hello,
>>
>> You have several options:
>>
>> 1) going forward lower gc_grace_seconds
>> http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configStorage_r.html?pagename=docs&version=1.2&file=configuration/storage_configuration#gc-grace-seconds
>>
>> - this is very use case specific. Default is 10 days. Some
>> users will put this at 0 for specific use cases.
>> 2) you could also lower tombstone compaction threshold and interval
>> to get tombstone compaction to fire more often on your tables/cfs:
>> https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436
>> 3) to clean out old tombstones you could always run a manual
>> compaction, those these aren't typically recommended though:
>> http://www.datastax.com/documentation/cassandra/1.2/cassandra/tools/toolsNodetool_r.html
>>
>>
>> For 1 and 2, be sure your disks can keep up with compaction to ensure
>> tombstone, or other, compaction fires regularly enough to clean out old
>> tombstones. Also, you probably want to ensure you are using Level
>> Compaction:
>> http://www.datastax.com/dev/blog/when-to-use-leveled-compaction.
>> Again, this assumes your disk system can handle the increased io from
>> Leveled Compaction.
>>
>> Also, you may be running into this with the older version of
>> Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541
>>
>> Hope this helps.
>>
>> Jonathan
>>
>>
>> Jonathan Lacefield
>> Solutions Architect, DataStax
>> (404) 822 3487
>> <image>
>>
>>
>> <image>
>>
>>
>> On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <ol...@gmail.com> wrote:
>> I get that :)
>>
>> What I'd like to know is how to fix that :)
>>
>> On 2014-03-09 20:24:54 +0000, Takenori Sato said:
>>
>> You have millions of org.apache.cassandra.db.DeletedColumn instances on
>> the snapshot.
>>
>> This means you have lots of column tombstones, and I guess, which are
>> read into memory by slice query.
>>
>>
>> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <ol...@gmail.com> wrote:
>> I am trying to understand why one of my nodes keeps full GC.
>>
>> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>>
>> Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ
>>
>> --
>> Regards,
>> Oleg Dulin
>> http://www.olegdulin.com
>>
>>
>> --
>> Regards,
>> Oleg Dulin
>> http://www.olegdulin.com
>
> S
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Posted by Oleg Dulin <ol...@gmail.com>.
Good news is that since I lowered gc_grace period it collected over
100Gigs of tombstones and seems much happier now.
Oleg
On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said:
> Hello,
>
> You have several options:
>
> 1) going forward lower gc_grace_seconds
> http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configStorage_r.html?pagename=docs&version=1.2&file=configuration/storage_configuration#gc-grace-seconds
>
> - this is very use case specific. Default is 10 days. Some
> users will put this at 0 for specific use cases.
> 2) you could also lower tombstone compaction threshold and interval
> to get tombstone compaction to fire more often on your tables/cfs:
> https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436
> 3) to clean out old tombstones you could always run a manual
> compaction, those these aren't typically recommended though:
> http://www.datastax.com/documentation/cassandra/1.2/cassandra/tools/toolsNodetool_r.html
>
>
> For 1 and 2, be sure your disks can keep up with compaction to ensure
> tombstone, or other, compaction fires regularly enough to clean out old
> tombstones. Also, you probably want to ensure you are using Level
> Compaction:
> http://www.datastax.com/dev/blog/when-to-use-leveled-compaction.
> Again, this assumes your disk system can handle the increased io from
> Leveled Compaction.
>
> Also, you may be running into this with the older version of
> Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541
>
> Hope this helps.
>
> Jonathan
>
>
> Jonathan Lacefield
> Solutions Architect, DataStax
> (404) 822 3487
> <image>
>
>
> <image>
>
>
> On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <ol...@gmail.com> wrote:
> I get that :)
>
> What I'd like to know is how to fix that :)
>
> On 2014-03-09 20:24:54 +0000, Takenori Sato said:
>
> You have millions of org.apache.cassandra.db.DeletedColumn instances on
> the snapshot.
>
> This means you have lots of column tombstones, and I guess, which are
> read into memory by slice query.
>
>
> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <ol...@gmail.com> wrote:
> I am trying to understand why one of my nodes keeps full GC.
>
> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>
> Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
S
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: need help with Cassandra 1.2 Full GCing -- output of jmap
histogram
Posted by Keith Wright <kw...@nanigans.com>.
I also want to point out an issue I filed that was closed as not an issue: CASSANDRA-6654 Basically if you’re using mixed TTLs on columns in a row, the “oldest” TTL is used to determine if tombstones of other columns can be removed. In other words, if you have a column with a 1 day TTL and a column with a 1 year TTL, the 1 day TTLed data (now tombstoned) will not be removed on compaction.
From: Jonathan Lacefield <jl...@datastax.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday, March 10, 2014 at 8:33 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Hello,
You have several options:
1) going forward lower gc_grace_seconds http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configStorage_r.html?pagename=docs&version=1.2&file=configuration/storage_configuration#gc-grace-seconds
- this is very use case specific. Default is 10 days. Some users will put this at 0 for specific use cases.
2) you could also lower tombstone compaction threshold and interval to get tombstone compaction to fire more often on your tables/cfs: https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436
3) to clean out old tombstones you could always run a manual compaction, those these aren't typically recommended though: http://www.datastax.com/documentation/cassandra/1.2/cassandra/tools/toolsNodetool_r.html
For 1 and 2, be sure your disks can keep up with compaction to ensure tombstone, or other, compaction fires regularly enough to clean out old tombstones. Also, you probably want to ensure you are using Level Compaction: http://www.datastax.com/dev/blog/when-to-use-leveled-compaction. Again, this assumes your disk system can handle the increased io from Leveled Compaction.
Also, you may be running into this with the older version of Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541
Hope this helps.
Jonathan
Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487
[http://s.c.lnkd.licdn.com/scds/common/u/img/logos/logo_linkedin_92x22.png]<http://www.linkedin.com/in/jlacefield>
[http://www.datastax.com/wp-content/themes/datastax-2013/images/email-sig/email-sig-virtual-training.png]<http://www.datastax.com/what-we-offer/products-services/training/virtual-training>
On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <ol...@gmail.com>> wrote:
I get that :)
What I'd like to know is how to fix that :)
On 2014-03-09 20:24:54 +0000, Takenori Sato said:
You have millions of org.apache.cassandra.db.DeletedColumn instances on the snapshot.
This means you have lots of column tombstones, and I guess, which are read into memory by slice query.
On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <ol...@gmail.com>> wrote:
I am trying to understand why one of my nodes keeps full GC.
I have Xmx set to 8gigs, memtable total size is 2 gigs.
Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ
--
Regards,
Oleg Dulin
http://www.olegdulin.com
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Posted by Jonathan Lacefield <jl...@datastax.com>.
Hello,
You have several options:
1) going forward lower gc_grace_seconds
http://www.datastax.com/documentation/cassandra/1.2/cassandra/configuration/configStorage_r.html?pagename=docs&version=1.2&file=configuration/storage_configuration#gc-grace-seconds
- this is very use case specific. Default is 10 days. Some users
will put this at 0 for specific use cases.
2) you could also lower tombstone compaction threshold and interval to
get tombstone compaction to fire more often on your tables/cfs:
https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436
3) to clean out old tombstones you could always run a manual compaction,
those these aren't typically recommended though:
http://www.datastax.com/documentation/cassandra/1.2/cassandra/tools/toolsNodetool_r.html
For 1 and 2, be sure your disks can keep up with compaction to ensure
tombstone, or other, compaction fires regularly enough to clean out old
tombstones. Also, you probably want to ensure you are using Level
Compaction: http://www.datastax.com/dev/blog/when-to-use-leveled-compaction.
Again, this assumes your disk system can handle the increased io from
Leveled Compaction.
Also, you may be running into this with the older version of Cassandra:
https://issues.apache.org/jira/browse/CASSANDRA-6541
Hope this helps.
Jonathan
Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487
<http://www.linkedin.com/in/jlacefield>
<http://www.datastax.com/what-we-offer/products-services/training/virtual-training>
On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <ol...@gmail.com> wrote:
> I get that :)
>
> What I'd like to know is how to fix that :)
>
> On 2014-03-09 20:24:54 +0000, Takenori Sato said:
>
> You have millions of org.apache.cassandra.db.DeletedColumn instances on
>> the snapshot.
>>
>> This means you have lots of column tombstones, and I guess, which are
>> read into memory by slice query.
>>
>>
>> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <ol...@gmail.com> wrote:
>> I am trying to understand why one of my nodes keeps full GC.
>>
>> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>>
>> Consider the top entries from jmap -histo:live @
>> http://pastebin.com/UaatHfpJ
>>
>> --
>> Regards,
>> Oleg Dulin
>> http://www.olegdulin.com
>>
>
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
>
Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Posted by Oleg Dulin <ol...@gmail.com>.
I get that :)
What I'd like to know is how to fix that :)
On 2014-03-09 20:24:54 +0000, Takenori Sato said:
> You have millions of org.apache.cassandra.db.DeletedColumn instances on
> the snapshot.
>
> This means you have lots of column tombstones, and I guess, which are
> read into memory by slice query.
>
>
> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <ol...@gmail.com> wrote:
> I am trying to understand why one of my nodes keeps full GC.
>
> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>
> Consider the top entries from jmap -histo:live @ http://pastebin.com/UaatHfpJ
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: need help with Cassandra 1.2 Full GCing -- output of jmap histogram
Posted by Takenori Sato <ts...@cloudian.com>.
You have millions of org.apache.cassandra.db.DeletedColumn instances on the
snapshot.
This means you have lots of column tombstones, and I guess, which are read
into memory by slice query.
On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <ol...@gmail.com> wrote:
> I am trying to understand why one of my nodes keeps full GC.
>
> I have Xmx set to 8gigs, memtable total size is 2 gigs.
>
> Consider the top entries from jmap -histo:live @
> http://pastebin.com/UaatHfpJ
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
>