You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Daning <da...@netseer.com> on 2011/09/28 19:35:25 UTC
Weird problem with empty CF
I have an app polling a few CFs (select first N * from CF), there were
data in CFs but later were deleted so CFs were empty for a long time. I
found Cassandra CPU usage was getting high to 80%, normally it uses less
than 30%. I issued the select query manually and feel the response is
slow. I have tried nodetool compact/repair for those CFs but that does
not work. later, I issue 'truncate' for all the CFs and CPU usage gets
down to 1%.
Can somebody explain to me why I need to truncate an empty CF? and what
else I could do to bring the CPU usage down?
I am running 0.8.6.
Thanks,
Daning
Re: Weird problem with empty CF
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like you have non-expired tombstones.
http://wiki.apache.org/cassandra/DistributedDeletes
On Wed, Sep 28, 2011 at 12:35 PM, Daning <da...@netseer.com> wrote:
> I have an app polling a few CFs (select first N * from CF), there were data
> in CFs but later were deleted so CFs were empty for a long time. I found
> Cassandra CPU usage was getting high to 80%, normally it uses less than 30%.
> I issued the select query manually and feel the response is slow. I have
> tried nodetool compact/repair for those CFs but that does not work. later, I
> issue 'truncate' for all the CFs and CPU usage gets down to 1%.
>
> Can somebody explain to me why I need to truncate an empty CF? and what else
> I could do to bring the CPU usage down?
>
> I am running 0.8.6.
>
> Thanks,
>
> Daning
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Weird problem with empty CF
Posted by aaron morton <aa...@thelastpickle.com>.
No.
It's generally only an issue with heavy delete workloads, and it's sometimes possible to design around it.
cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 5/10/2011, at 1:18 PM, Daning wrote:
> Thanks. Do you have plan to improve this? I think tombstone should be separated with live data since it serves different purpose, built in separate SSTable or indexed differently. It is pretty costly to do filtering while reading.
>
> Daning
>
> On 10/04/2011 01:34 PM, aaron morton wrote:
>>
>> I would not get gc_grace seconds to 0, set to to something small.
>>
>> gc_grace_seconds or ttl is only the minimum amount of time the column will stay in the data files. The columns are only purged when compaction runs some time after that timespan has ended.
>>
>> If you are seeing issues where a heavy delete workload is having an noticeably adverse effect on read performance then you should look at the data model. Consider ways to spread the write / read / delete workload over multiple rows.
>>
>> If you cannot get away from it then experiment with reducing the min_compactioon_threshold of the CF's so that compaction kicks in quicker, and (potentially) tombstones are purged faster.
>>
>> Chees
>>
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 5/10/2011, at 6:03 AM, Daning wrote:
>>
>>> Thanks Aaron. How about I set the gc_grace_seconds to 0 or like 2 hours? I like to clean up tomebstone sooner, I don't care losing some data and all my columns have ttl.
>>>
>>> If one node is down longer than gc_grace_seconds, and I got tombstone removed, once the node is up, from my understanding deleted data will be synced back. In this case my data will be processed twice and it will not be a big deal to me.
>>>
>>> Thanks,
>>>
>>> Daning
>>>
>>>
>>> On 10/04/2011 01:27 AM, aaron morton wrote:
>>>>
>>>> Yes that's the slice query skipping past the tombstone columns.
>>>>
>>>> Cheers
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Cassandra Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 4/10/2011, at 4:24 PM, Daning Wang wrote:
>>>>
>>>>> Lots of SliceQueryFilter in the log, is that handling tombstone?
>>>>>
>>>>> DEBUG [ReadStage:49] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582939743663:true:4@1317582939933000
>>>>> DEBUG [ReadStage:50] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317573253148778:true:4@1317573253354000
>>>>> DEBUG [ReadStage:43] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317669552951428:true:4@1317669553018000
>>>>> DEBUG [ReadStage:33] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317581886709261:true:4@1317581886957000
>>>>> DEBUG [ReadStage:52] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317568165152246:true:4@1317568165482000
>>>>> DEBUG [ReadStage:36] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317567265089211:true:4@1317567265405000
>>>>> DEBUG [ReadStage:53] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317674324843122:true:4@1317674324946000
>>>>> DEBUG [ReadStage:38] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317571990078721:true:4@1317571990141000
>>>>> DEBUG [ReadStage:57] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317671855234221:true:4@1317671855239000
>>>>> DEBUG [ReadStage:54] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317558305262954:true:4@1317558305337000
>>>>> DEBUG [RequestResponseStage:11] 2011-10-03 20:15:07,941 ResponseVerbHandler.java (line 48) Processing response on a callback from 12347@/10.210.101.104
>>>>> DEBUG [RequestResponseStage:9] 2011-10-03 20:15:07,941 AbstractRowResolver.java (line 66) Preprocessed data response
>>>>> DEBUG [RequestResponseStage:13] 2011-10-03 20:15:07,941 AbstractRowResolver.java (line 66) Preprocessed digest response
>>>>> DEBUG [ReadStage:58] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317581337972739:true:4@1317581338044000
>>>>> DEBUG [ReadStage:64] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582656796332:true:4@1317582656970000
>>>>> DEBUG [ReadStage:55] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317569432886284:true:4@1317569432984000
>>>>> DEBUG [ReadStage:45] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317572658687019:true:4@1317572658718000
>>>>> DEBUG [ReadStage:47] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582281617755:true:4@1317582281717000
>>>>> DEBUG [ReadStage:48] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317549607869226:true:4@1317549608118000
>>>>> DEBUG [ReadStage:34] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1:
>>>>> On Thu, Sep 29, 2011 at 2:17 PM, aaron morton <aa...@thelastpickle.com> wrote:
>>>>> As with any situation involving the un-dead, it really is the number of Zombies, Mummies or Vampires that is the concern.
>>>>>
>>>>> If you delete data there will always be tombstones. If you have a delete heavy workload there will be more tombstones. This is why implementing a queue with cassandra is a bad idea.
>>>>>
>>>>> gc_grace_seconds (and column TTL) are the *minimum* about of time the tombstones will stay in the data files, there is no maximum.
>>>>>
>>>>> Your read performance also depends on the number of SSTables the row is spread over, see http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/
>>>>>
>>>>> If you really wanted to purge them then yes a repair and then major compaction would be the way to go. Also consider if it's possible to design the data model around the problem, e.g. partitioning rows by date. IMHO I would look to make data model changes before implementing a compaction policy, or consider if cassandra is the right store if you have a delete heavy workload.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> -----------------
>>>>> Aaron Morton
>>>>> Freelance Cassandra Developer
>>>>> @aaronmorton
>>>>> http://www.thelastpickle.com
>>>>>
>>>>> On 30/09/2011, at 3:27 AM, Daning Wang wrote:
>>>>>
>>>>>> Jonathan/Aaron,
>>>>>>
>>>>>> Thank you guy's reply, I will change GCGracePeriod to 1 day to see what will happen.
>>>>>>
>>>>>> Is there a way to purge tombstones at anytime? because if tombstones affect performance, we want them to be purged right away, not after GCGracePeriod. We know all the nodes are up, and we can do repair first to make sure the consistency before purging.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Daning
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 28, 2011 at 5:22 PM, aaron morton <aa...@thelastpickle.com> wrote:
>>>>>> if I had to guess I would say it was spending time handling tombstones. If you see it happen again, and are interested, turn the logging up to DEBUG and look for messages from something starting with "Slice"
>>>>>>
>>>>>> Minor (automatic) compaction will, over time, purge the tombstones. Until then reads must read discard the data deleted by the tombstones. If you perform a big (i.e. 100k's ) delete this can reduce performance until compaction does it's thing.
>>>>>>
>>>>>> My second guess would be read repair (or the simple consistency checks on read) kicking in. That would show up in the "ReadRepairStage" in TPSTATS
>>>>>>
>>>>>> it may have been neither of those two things, just guesses. If you have more issues let us know and provide some more info.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> -----------------
>>>>>> Aaron Morton
>>>>>> Freelance Cassandra Developer
>>>>>> @aaronmorton
>>>>>> http://www.thelastpickle.com
>>>>>>
>>>>>> On 29/09/2011, at 6:35 AM, Daning wrote:
>>>>>>
>>>>>> > I have an app polling a few CFs (select first N * from CF), there were data in CFs but later were deleted so CFs were empty for a long time. I found Cassandra CPU usage was getting high to 80%, normally it uses less than 30%. I issued the select query manually and feel the response is slow. I have tried nodetool compact/repair for those CFs but that does not work. later, I issue 'truncate' for all the CFs and CPU usage gets down to 1%.
>>>>>> >
>>>>>> > Can somebody explain to me why I need to truncate an empty CF? and what else I could do to bring the CPU usage down?
>>>>>> >
>>>>>> > I am running 0.8.6.
>>>>>> >
>>>>>> > Thanks,
>>>>>> >
>>>>>> > Daning
>>>>>> >
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Weird problem with empty CF
Posted by Daning <da...@netseer.com>.
Thanks. Do you have plan to improve this? I think tombstone should be
separated with live data since it serves different purpose, built in
separate SSTable or indexed differently. It is pretty costly to do
filtering while reading.
Daning
On 10/04/2011 01:34 PM, aaron morton wrote:
> I would not get gc_grace seconds to 0, set to to something small.
>
> gc_grace_seconds or ttl is only the minimum amount of time the column
> will stay in the data files. The columns are only purged when
> compaction runs some time after that timespan has ended.
>
> If you are seeing issues where a heavy delete workload is having an
> noticeably adverse effect on read performance then you should look at
> the data model. Consider ways to spread the write / read / delete
> workload over multiple rows.
>
> If you cannot get away from it then experiment with reducing the
> min_compactioon_threshold of the CF's so that compaction kicks in
> quicker, and (potentially) tombstones are purged faster.
>
> Chees
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 5/10/2011, at 6:03 AM, Daning wrote:
>
>> Thanks Aaron. How about I set the gc_grace_seconds to 0 or like 2
>> hours? I like to clean up tomebstone sooner, I don't care losing some
>> data and all my columns have ttl.
>>
>> If one node is down longer than gc_grace_seconds, and I got tombstone
>> removed, once the node is up, from my understanding deleted data will
>> be synced back. In this case my data will be processed twice and it
>> will not be a big deal to me.
>>
>> Thanks,
>>
>> Daning
>>
>>
>> On 10/04/2011 01:27 AM, aaron morton wrote:
>>> Yes that's the slice query skipping past the tombstone columns.
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com <http://www.thelastpickle.com/>
>>>
>>> On 4/10/2011, at 4:24 PM, Daning Wang wrote:
>>>
>>>> Lots of SliceQueryFilter in the log, is that handling tombstone?
>>>>
>>>> DEBUG [ReadStage:49] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317582939743663:true:4@1317582939933000
>>>> DEBUG [ReadStage:50] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317573253148778:true:4@1317573253354000
>>>> DEBUG [ReadStage:43] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317669552951428:true:4@1317669553018000
>>>> DEBUG [ReadStage:33] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317581886709261:true:4@1317581886957000
>>>> DEBUG [ReadStage:52] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317568165152246:true:4@1317568165482000
>>>> DEBUG [ReadStage:36] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317567265089211:true:4@1317567265405000
>>>> DEBUG [ReadStage:53] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317674324843122:true:4@1317674324946000
>>>> DEBUG [ReadStage:38] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317571990078721:true:4@1317571990141000
>>>> DEBUG [ReadStage:57] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317671855234221:true:4@1317671855239000
>>>> DEBUG [ReadStage:54] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317558305262954:true:4@1317558305337000
>>>> DEBUG [RequestResponseStage:11] 2011-10-03 20:15:07,941
>>>> ResponseVerbHandler.java (line 48) Processing response on a
>>>> callback from 12347@/10.210.101.104 <http://10.210.101.104/>
>>>> DEBUG [RequestResponseStage:9] 2011-10-03 20:15:07,941
>>>> AbstractRowResolver.java (line 66) Preprocessed data response
>>>> DEBUG [RequestResponseStage:13] 2011-10-03 20:15:07,941
>>>> AbstractRowResolver.java (line 66) Preprocessed digest response
>>>> DEBUG [ReadStage:58] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317581337972739:true:4@1317581338044000
>>>> DEBUG [ReadStage:64] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317582656796332:true:4@1317582656970000
>>>> DEBUG [ReadStage:55] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317569432886284:true:4@1317569432984000
>>>> DEBUG [ReadStage:45] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317572658687019:true:4@1317572658718000
>>>> DEBUG [ReadStage:47] 2011-10-03 20:15:07,940 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317582281617755:true:4@1317582281717000
>>>> DEBUG [ReadStage:48] 2011-10-03 20:15:07,940 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1: 1317549607869226:true:4@1317549608118000
>>>> DEBUG [ReadStage:34] 2011-10-03 20:15:07,940 SliceQueryFilter.java
>>>> (line 123) collecting 0 of 1:
>>>> On Thu, Sep 29, 2011 at 2:17 PM, aaron morton
>>>> <aaron@thelastpickle.com <ma...@thelastpickle.com>> wrote:
>>>>
>>>> As with any situation involving the un-dead, it really is the
>>>> number of Zombies, Mummies or Vampires that is the concern.
>>>>
>>>> If you delete data there will always be tombstones. If you have
>>>> a delete heavy workload there will be more tombstones. This is
>>>> why implementing a queue with cassandra is a bad idea.
>>>>
>>>> gc_grace_seconds (and column TTL) are the *minimum* about of
>>>> time the tombstones will stay in the data files, there is no
>>>> maximum.
>>>>
>>>> Your read performance also depends on the number of SSTables
>>>> the row is spread over, see
>>>> http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/
>>>>
>>>> If you really wanted to purge them then yes a repair and then
>>>> major compaction would be the way to go. Also consider if it's
>>>> possible to design the data model around the problem, e.g.
>>>> partitioning rows by date. IMHO I would look to make data model
>>>> changes before implementing a compaction policy, or consider if
>>>> cassandra is the right store if you have a delete heavy workload.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Cassandra Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com <http://www.thelastpickle.com/>
>>>>
>>>> On 30/09/2011, at 3:27 AM, Daning Wang wrote:
>>>>
>>>>> Jonathan/Aaron,
>>>>>
>>>>> Thank you guy's reply, I will change GCGracePeriod to 1 day to
>>>>> see what will happen.
>>>>>
>>>>> Is there a way to purge tombstones at anytime? because if
>>>>> tombstones affect performance, we want them to be purged right
>>>>> away, not after GCGracePeriod. We know all the nodes are up,
>>>>> and we can do repair first to make sure the consistency before
>>>>> purging.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Daning
>>>>>
>>>>>
>>>>> On Wed, Sep 28, 2011 at 5:22 PM, aaron morton
>>>>> <aaron@thelastpickle.com <ma...@thelastpickle.com>> wrote:
>>>>>
>>>>> if I had to guess I would say it was spending time
>>>>> handling tombstones. If you see it happen again, and are
>>>>> interested, turn the logging up to DEBUG and look for
>>>>> messages from something starting with "Slice"
>>>>>
>>>>> Minor (automatic) compaction will, over time, purge the
>>>>> tombstones. Until then reads must read discard the data
>>>>> deleted by the tombstones. If you perform a big (i.e.
>>>>> 100k's ) delete this can reduce performance until
>>>>> compaction does it's thing.
>>>>>
>>>>> My second guess would be read repair (or the simple
>>>>> consistency checks on read) kicking in. That would show up
>>>>> in the "ReadRepairStage" in TPSTATS
>>>>>
>>>>> it may have been neither of those two things, just
>>>>> guesses. If you have more issues let us know and provide
>>>>> some more info.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> -----------------
>>>>> Aaron Morton
>>>>> Freelance Cassandra Developer
>>>>> @aaronmorton
>>>>> http://www.thelastpickle.com <http://www.thelastpickle.com/>
>>>>>
>>>>> On 29/09/2011, at 6:35 AM, Daning wrote:
>>>>>
>>>>> > I have an app polling a few CFs (select first N * from
>>>>> CF), there were data in CFs but later were deleted so CFs
>>>>> were empty for a long time. I found Cassandra CPU usage
>>>>> was getting high to 80%, normally it uses less than 30%. I
>>>>> issued the select query manually and feel the response is
>>>>> slow. I have tried nodetool compact/repair for those CFs
>>>>> but that does not work. later, I issue 'truncate' for all
>>>>> the CFs and CPU usage gets down to 1%.
>>>>> >
>>>>> > Can somebody explain to me why I need to truncate an
>>>>> empty CF? and what else I could do to bring the CPU usage
>>>>> down?
>>>>> >
>>>>> > I am running 0.8.6.
>>>>> >
>>>>> > Thanks,
>>>>> >
>>>>> > Daning
>>>>> >
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Weird problem with empty CF
Posted by aaron morton <aa...@thelastpickle.com>.
I would not get gc_grace seconds to 0, set to to something small.
gc_grace_seconds or ttl is only the minimum amount of time the column will stay in the data files. The columns are only purged when compaction runs some time after that timespan has ended.
If you are seeing issues where a heavy delete workload is having an noticeably adverse effect on read performance then you should look at the data model. Consider ways to spread the write / read / delete workload over multiple rows.
If you cannot get away from it then experiment with reducing the min_compactioon_threshold of the CF's so that compaction kicks in quicker, and (potentially) tombstones are purged faster.
Chees
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 5/10/2011, at 6:03 AM, Daning wrote:
> Thanks Aaron. How about I set the gc_grace_seconds to 0 or like 2 hours? I like to clean up tomebstone sooner, I don't care losing some data and all my columns have ttl.
>
> If one node is down longer than gc_grace_seconds, and I got tombstone removed, once the node is up, from my understanding deleted data will be synced back. In this case my data will be processed twice and it will not be a big deal to me.
>
> Thanks,
>
> Daning
>
>
> On 10/04/2011 01:27 AM, aaron morton wrote:
>>
>> Yes that's the slice query skipping past the tombstone columns.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 4/10/2011, at 4:24 PM, Daning Wang wrote:
>>
>>> Lots of SliceQueryFilter in the log, is that handling tombstone?
>>>
>>> DEBUG [ReadStage:49] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582939743663:true:4@1317582939933000
>>> DEBUG [ReadStage:50] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317573253148778:true:4@1317573253354000
>>> DEBUG [ReadStage:43] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317669552951428:true:4@1317669553018000
>>> DEBUG [ReadStage:33] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317581886709261:true:4@1317581886957000
>>> DEBUG [ReadStage:52] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317568165152246:true:4@1317568165482000
>>> DEBUG [ReadStage:36] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317567265089211:true:4@1317567265405000
>>> DEBUG [ReadStage:53] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317674324843122:true:4@1317674324946000
>>> DEBUG [ReadStage:38] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317571990078721:true:4@1317571990141000
>>> DEBUG [ReadStage:57] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317671855234221:true:4@1317671855239000
>>> DEBUG [ReadStage:54] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317558305262954:true:4@1317558305337000
>>> DEBUG [RequestResponseStage:11] 2011-10-03 20:15:07,941 ResponseVerbHandler.java (line 48) Processing response on a callback from 12347@/10.210.101.104
>>> DEBUG [RequestResponseStage:9] 2011-10-03 20:15:07,941 AbstractRowResolver.java (line 66) Preprocessed data response
>>> DEBUG [RequestResponseStage:13] 2011-10-03 20:15:07,941 AbstractRowResolver.java (line 66) Preprocessed digest response
>>> DEBUG [ReadStage:58] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317581337972739:true:4@1317581338044000
>>> DEBUG [ReadStage:64] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582656796332:true:4@1317582656970000
>>> DEBUG [ReadStage:55] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317569432886284:true:4@1317569432984000
>>> DEBUG [ReadStage:45] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317572658687019:true:4@1317572658718000
>>> DEBUG [ReadStage:47] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582281617755:true:4@1317582281717000
>>> DEBUG [ReadStage:48] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317549607869226:true:4@1317549608118000
>>> DEBUG [ReadStage:34] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1:
>>> On Thu, Sep 29, 2011 at 2:17 PM, aaron morton <aa...@thelastpickle.com> wrote:
>>> As with any situation involving the un-dead, it really is the number of Zombies, Mummies or Vampires that is the concern.
>>>
>>> If you delete data there will always be tombstones. If you have a delete heavy workload there will be more tombstones. This is why implementing a queue with cassandra is a bad idea.
>>>
>>> gc_grace_seconds (and column TTL) are the *minimum* about of time the tombstones will stay in the data files, there is no maximum.
>>>
>>> Your read performance also depends on the number of SSTables the row is spread over, see http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/
>>>
>>> If you really wanted to purge them then yes a repair and then major compaction would be the way to go. Also consider if it's possible to design the data model around the problem, e.g. partitioning rows by date. IMHO I would look to make data model changes before implementing a compaction policy, or consider if cassandra is the right store if you have a delete heavy workload.
>>>
>>> Cheers
>>>
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 30/09/2011, at 3:27 AM, Daning Wang wrote:
>>>
>>>> Jonathan/Aaron,
>>>>
>>>> Thank you guy's reply, I will change GCGracePeriod to 1 day to see what will happen.
>>>>
>>>> Is there a way to purge tombstones at anytime? because if tombstones affect performance, we want them to be purged right away, not after GCGracePeriod. We know all the nodes are up, and we can do repair first to make sure the consistency before purging.
>>>>
>>>> Thanks,
>>>>
>>>> Daning
>>>>
>>>>
>>>> On Wed, Sep 28, 2011 at 5:22 PM, aaron morton <aa...@thelastpickle.com> wrote:
>>>> if I had to guess I would say it was spending time handling tombstones. If you see it happen again, and are interested, turn the logging up to DEBUG and look for messages from something starting with "Slice"
>>>>
>>>> Minor (automatic) compaction will, over time, purge the tombstones. Until then reads must read discard the data deleted by the tombstones. If you perform a big (i.e. 100k's ) delete this can reduce performance until compaction does it's thing.
>>>>
>>>> My second guess would be read repair (or the simple consistency checks on read) kicking in. That would show up in the "ReadRepairStage" in TPSTATS
>>>>
>>>> it may have been neither of those two things, just guesses. If you have more issues let us know and provide some more info.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Cassandra Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 29/09/2011, at 6:35 AM, Daning wrote:
>>>>
>>>> > I have an app polling a few CFs (select first N * from CF), there were data in CFs but later were deleted so CFs were empty for a long time. I found Cassandra CPU usage was getting high to 80%, normally it uses less than 30%. I issued the select query manually and feel the response is slow. I have tried nodetool compact/repair for those CFs but that does not work. later, I issue 'truncate' for all the CFs and CPU usage gets down to 1%.
>>>> >
>>>> > Can somebody explain to me why I need to truncate an empty CF? and what else I could do to bring the CPU usage down?
>>>> >
>>>> > I am running 0.8.6.
>>>> >
>>>> > Thanks,
>>>> >
>>>> > Daning
>>>> >
>>>>
>>>>
>>>
>>>
>>
>
Re: Weird problem with empty CF
Posted by Daning <da...@netseer.com>.
Thanks Aaron. How about I set the gc_grace_seconds to 0 or like 2
hours? I like to clean up tomebstone sooner, I don't care losing some
data and all my columns have ttl.
If one node is down longer than gc_grace_seconds, and I got tombstone
removed, once the node is up, from my understanding deleted data will be
synced back. In this case my data will be processed twice and it will
not be a big deal to me.
Thanks,
Daning
On 10/04/2011 01:27 AM, aaron morton wrote:
> Yes that's the slice query skipping past the tombstone columns.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 4/10/2011, at 4:24 PM, Daning Wang wrote:
>
>> Lots of SliceQueryFilter in the log, is that handling tombstone?
>>
>> DEBUG [ReadStage:49] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317582939743663:true:4@1317582939933000
>> DEBUG [ReadStage:50] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317573253148778:true:4@1317573253354000
>> DEBUG [ReadStage:43] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317669552951428:true:4@1317669553018000
>> DEBUG [ReadStage:33] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317581886709261:true:4@1317581886957000
>> DEBUG [ReadStage:52] 2011-10-03 20:15:07,942 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317568165152246:true:4@1317568165482000
>> DEBUG [ReadStage:36] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317567265089211:true:4@1317567265405000
>> DEBUG [ReadStage:53] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317674324843122:true:4@1317674324946000
>> DEBUG [ReadStage:38] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317571990078721:true:4@1317571990141000
>> DEBUG [ReadStage:57] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317671855234221:true:4@1317671855239000
>> DEBUG [ReadStage:54] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317558305262954:true:4@1317558305337000
>> DEBUG [RequestResponseStage:11] 2011-10-03 20:15:07,941
>> ResponseVerbHandler.java (line 48) Processing response on a callback
>> from 12347@/10.210.101.104 <http://10.210.101.104/>
>> DEBUG [RequestResponseStage:9] 2011-10-03 20:15:07,941
>> AbstractRowResolver.java (line 66) Preprocessed data response
>> DEBUG [RequestResponseStage:13] 2011-10-03 20:15:07,941
>> AbstractRowResolver.java (line 66) Preprocessed digest response
>> DEBUG [ReadStage:58] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317581337972739:true:4@1317581338044000
>> DEBUG [ReadStage:64] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317582656796332:true:4@1317582656970000
>> DEBUG [ReadStage:55] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317569432886284:true:4@1317569432984000
>> DEBUG [ReadStage:45] 2011-10-03 20:15:07,941 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317572658687019:true:4@1317572658718000
>> DEBUG [ReadStage:47] 2011-10-03 20:15:07,940 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317582281617755:true:4@1317582281717000
>> DEBUG [ReadStage:48] 2011-10-03 20:15:07,940 SliceQueryFilter.java
>> (line 123) collecting 0 of 1: 1317549607869226:true:4@1317549608118000
>> DEBUG [ReadStage:34] 2011-10-03 20:15:07,940 SliceQueryFilter.java
>> (line 123) collecting 0 of 1:
>> On Thu, Sep 29, 2011 at 2:17 PM, aaron morton
>> <aaron@thelastpickle.com <ma...@thelastpickle.com>> wrote:
>>
>> As with any situation involving the un-dead, it really is the
>> number of Zombies, Mummies or Vampires that is the concern.
>>
>> If you delete data there will always be tombstones. If you have a
>> delete heavy workload there will be more tombstones. This is why
>> implementing a queue with cassandra is a bad idea.
>>
>> gc_grace_seconds (and column TTL) are the *minimum* about of time
>> the tombstones will stay in the data files, there is no maximum.
>>
>> Your read performance also depends on the number of SSTables the
>> row is spread over, see
>> http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/
>>
>> If you really wanted to purge them then yes a repair and then
>> major compaction would be the way to go. Also consider if it's
>> possible to design the data model around the problem, e.g.
>> partitioning rows by date. IMHO I would look to make data model
>> changes before implementing a compaction policy, or consider if
>> cassandra is the right store if you have a delete heavy workload.
>>
>> Cheers
>>
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com <http://www.thelastpickle.com/>
>>
>> On 30/09/2011, at 3:27 AM, Daning Wang wrote:
>>
>>> Jonathan/Aaron,
>>>
>>> Thank you guy's reply, I will change GCGracePeriod to 1 day to
>>> see what will happen.
>>>
>>> Is there a way to purge tombstones at anytime? because if
>>> tombstones affect performance, we want them to be purged right
>>> away, not after GCGracePeriod. We know all the nodes are up, and
>>> we can do repair first to make sure the consistency before purging.
>>>
>>> Thanks,
>>>
>>> Daning
>>>
>>>
>>> On Wed, Sep 28, 2011 at 5:22 PM, aaron morton
>>> <aaron@thelastpickle.com <ma...@thelastpickle.com>> wrote:
>>>
>>> if I had to guess I would say it was spending time handling
>>> tombstones. If you see it happen again, and are interested,
>>> turn the logging up to DEBUG and look for messages from
>>> something starting with "Slice"
>>>
>>> Minor (automatic) compaction will, over time, purge the
>>> tombstones. Until then reads must read discard the data
>>> deleted by the tombstones. If you perform a big (i.e. 100k's
>>> ) delete this can reduce performance until compaction does
>>> it's thing.
>>>
>>> My second guess would be read repair (or the simple
>>> consistency checks on read) kicking in. That would show up
>>> in the "ReadRepairStage" in TPSTATS
>>>
>>> it may have been neither of those two things, just guesses.
>>> If you have more issues let us know and provide some more info.
>>>
>>> Cheers
>>>
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com <http://www.thelastpickle.com/>
>>>
>>> On 29/09/2011, at 6:35 AM, Daning wrote:
>>>
>>> > I have an app polling a few CFs (select first N * from
>>> CF), there were data in CFs but later were deleted so CFs
>>> were empty for a long time. I found Cassandra CPU usage was
>>> getting high to 80%, normally it uses less than 30%. I
>>> issued the select query manually and feel the response is
>>> slow. I have tried nodetool compact/repair for those CFs but
>>> that does not work. later, I issue 'truncate' for all the
>>> CFs and CPU usage gets down to 1%.
>>> >
>>> > Can somebody explain to me why I need to truncate an empty
>>> CF? and what else I could do to bring the CPU usage down?
>>> >
>>> > I am running 0.8.6.
>>> >
>>> > Thanks,
>>> >
>>> > Daning
>>> >
>>>
>>>
>>
>>
>
Re: Weird problem with empty CF
Posted by aaron morton <aa...@thelastpickle.com>.
Yes that's the slice query skipping past the tombstone columns.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 4/10/2011, at 4:24 PM, Daning Wang wrote:
> Lots of SliceQueryFilter in the log, is that handling tombstone?
>
> DEBUG [ReadStage:49] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582939743663:true:4@1317582939933000
> DEBUG [ReadStage:50] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317573253148778:true:4@1317573253354000
> DEBUG [ReadStage:43] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317669552951428:true:4@1317669553018000
> DEBUG [ReadStage:33] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317581886709261:true:4@1317581886957000
> DEBUG [ReadStage:52] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317568165152246:true:4@1317568165482000
> DEBUG [ReadStage:36] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317567265089211:true:4@1317567265405000
> DEBUG [ReadStage:53] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317674324843122:true:4@1317674324946000
> DEBUG [ReadStage:38] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317571990078721:true:4@1317571990141000
> DEBUG [ReadStage:57] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317671855234221:true:4@1317671855239000
> DEBUG [ReadStage:54] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317558305262954:true:4@1317558305337000
> DEBUG [RequestResponseStage:11] 2011-10-03 20:15:07,941 ResponseVerbHandler.java (line 48) Processing response on a callback from 12347@/10.210.101.104
> DEBUG [RequestResponseStage:9] 2011-10-03 20:15:07,941 AbstractRowResolver.java (line 66) Preprocessed data response
> DEBUG [RequestResponseStage:13] 2011-10-03 20:15:07,941 AbstractRowResolver.java (line 66) Preprocessed digest response
> DEBUG [ReadStage:58] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317581337972739:true:4@1317581338044000
> DEBUG [ReadStage:64] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582656796332:true:4@1317582656970000
> DEBUG [ReadStage:55] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317569432886284:true:4@1317569432984000
> DEBUG [ReadStage:45] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317572658687019:true:4@1317572658718000
> DEBUG [ReadStage:47] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317582281617755:true:4@1317582281717000
> DEBUG [ReadStage:48] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1: 1317549607869226:true:4@1317549608118000
> DEBUG [ReadStage:34] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line 123) collecting 0 of 1:
> On Thu, Sep 29, 2011 at 2:17 PM, aaron morton <aa...@thelastpickle.com> wrote:
> As with any situation involving the un-dead, it really is the number of Zombies, Mummies or Vampires that is the concern.
>
> If you delete data there will always be tombstones. If you have a delete heavy workload there will be more tombstones. This is why implementing a queue with cassandra is a bad idea.
>
> gc_grace_seconds (and column TTL) are the *minimum* about of time the tombstones will stay in the data files, there is no maximum.
>
> Your read performance also depends on the number of SSTables the row is spread over, see http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/
>
> If you really wanted to purge them then yes a repair and then major compaction would be the way to go. Also consider if it's possible to design the data model around the problem, e.g. partitioning rows by date. IMHO I would look to make data model changes before implementing a compaction policy, or consider if cassandra is the right store if you have a delete heavy workload.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 30/09/2011, at 3:27 AM, Daning Wang wrote:
>
>> Jonathan/Aaron,
>>
>> Thank you guy's reply, I will change GCGracePeriod to 1 day to see what will happen.
>>
>> Is there a way to purge tombstones at anytime? because if tombstones affect performance, we want them to be purged right away, not after GCGracePeriod. We know all the nodes are up, and we can do repair first to make sure the consistency before purging.
>>
>> Thanks,
>>
>> Daning
>>
>>
>> On Wed, Sep 28, 2011 at 5:22 PM, aaron morton <aa...@thelastpickle.com> wrote:
>> if I had to guess I would say it was spending time handling tombstones. If you see it happen again, and are interested, turn the logging up to DEBUG and look for messages from something starting with "Slice"
>>
>> Minor (automatic) compaction will, over time, purge the tombstones. Until then reads must read discard the data deleted by the tombstones. If you perform a big (i.e. 100k's ) delete this can reduce performance until compaction does it's thing.
>>
>> My second guess would be read repair (or the simple consistency checks on read) kicking in. That would show up in the "ReadRepairStage" in TPSTATS
>>
>> it may have been neither of those two things, just guesses. If you have more issues let us know and provide some more info.
>>
>> Cheers
>>
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 29/09/2011, at 6:35 AM, Daning wrote:
>>
>> > I have an app polling a few CFs (select first N * from CF), there were data in CFs but later were deleted so CFs were empty for a long time. I found Cassandra CPU usage was getting high to 80%, normally it uses less than 30%. I issued the select query manually and feel the response is slow. I have tried nodetool compact/repair for those CFs but that does not work. later, I issue 'truncate' for all the CFs and CPU usage gets down to 1%.
>> >
>> > Can somebody explain to me why I need to truncate an empty CF? and what else I could do to bring the CPU usage down?
>> >
>> > I am running 0.8.6.
>> >
>> > Thanks,
>> >
>> > Daning
>> >
>>
>>
>
>
Re: Weird problem with empty CF
Posted by Daning Wang <da...@netseer.com>.
Lots of SliceQueryFilter in the log, is that handling tombstone?
DEBUG [ReadStage:49] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317582939743663:true:4@1317582939933000
DEBUG [ReadStage:50] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317573253148778:true:4@1317573253354000
DEBUG [ReadStage:43] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317669552951428:true:4@1317669553018000
DEBUG [ReadStage:33] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317581886709261:true:4@1317581886957000
DEBUG [ReadStage:52] 2011-10-03 20:15:07,942 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317568165152246:true:4@1317568165482000
DEBUG [ReadStage:36] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317567265089211:true:4@1317567265405000
DEBUG [ReadStage:53] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317674324843122:true:4@1317674324946000
DEBUG [ReadStage:38] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317571990078721:true:4@1317571990141000
DEBUG [ReadStage:57] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317671855234221:true:4@1317671855239000
DEBUG [ReadStage:54] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317558305262954:true:4@1317558305337000
DEBUG [RequestResponseStage:11] 2011-10-03 20:15:07,941
ResponseVerbHandler.java (line 48) Processing response on a callback from
12347@/10.210.101.104
DEBUG [RequestResponseStage:9] 2011-10-03 20:15:07,941
AbstractRowResolver.java (line 66) Preprocessed data response
DEBUG [RequestResponseStage:13] 2011-10-03 20:15:07,941
AbstractRowResolver.java (line 66) Preprocessed digest response
DEBUG [ReadStage:58] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317581337972739:true:4@1317581338044000
DEBUG [ReadStage:64] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317582656796332:true:4@1317582656970000
DEBUG [ReadStage:55] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317569432886284:true:4@1317569432984000
DEBUG [ReadStage:45] 2011-10-03 20:15:07,941 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317572658687019:true:4@1317572658718000
DEBUG [ReadStage:47] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317582281617755:true:4@1317582281717000
DEBUG [ReadStage:48] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line
123) collecting 0 of 1: 1317549607869226:true:4@1317549608118000
DEBUG [ReadStage:34] 2011-10-03 20:15:07,940 SliceQueryFilter.java (line
123) collecting 0 of 1:
On Thu, Sep 29, 2011 at 2:17 PM, aaron morton <aa...@thelastpickle.com>wrote:
> As with any situation involving the un-dead, it really is the number of
> Zombies, Mummies or Vampires that is the concern.
>
> If you delete data there will always be tombstones. If you have a delete
> heavy workload there will be more tombstones. This is why implementing a
> queue with cassandra is a bad idea.
>
> gc_grace_seconds (and column TTL) are the *minimum* about of time the
> tombstones will stay in the data files, there is no maximum.
>
> Your read performance also depends on the number of SSTables the row is
> spread over, see
> http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/
>
> If you really wanted to purge them then yes a repair and then major
> compaction would be the way to go. Also consider if it's possible to design
> the data model around the problem, e.g. partitioning rows by date. IMHO I
> would look to make data model changes before implementing a compaction
> policy, or consider if cassandra is the right store if you have a delete
> heavy workload.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 30/09/2011, at 3:27 AM, Daning Wang wrote:
>
> Jonathan/Aaron,
>
> Thank you guy's reply, I will change GCGracePeriod to 1 day to see what
> will happen.
>
> Is there a way to purge tombstones at anytime? because if tombstones affect
> performance, we want them to be purged right away, not after GCGracePeriod.
> We know all the nodes are up, and we can do repair first to make sure the
> consistency before purging.
>
> Thanks,
>
> Daning
>
>
> On Wed, Sep 28, 2011 at 5:22 PM, aaron morton <aa...@thelastpickle.com>wrote:
>
>> if I had to guess I would say it was spending time handling tombstones. If
>> you see it happen again, and are interested, turn the logging up to DEBUG
>> and look for messages from something starting with "Slice"
>>
>> Minor (automatic) compaction will, over time, purge the tombstones. Until
>> then reads must read discard the data deleted by the tombstones. If you
>> perform a big (i.e. 100k's ) delete this can reduce performance until
>> compaction does it's thing.
>>
>> My second guess would be read repair (or the simple consistency checks on
>> read) kicking in. That would show up in the "ReadRepairStage" in TPSTATS
>>
>> it may have been neither of those two things, just guesses. If you have
>> more issues let us know and provide some more info.
>>
>> Cheers
>>
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 29/09/2011, at 6:35 AM, Daning wrote:
>>
>> > I have an app polling a few CFs (select first N * from CF), there were
>> data in CFs but later were deleted so CFs were empty for a long time. I
>> found Cassandra CPU usage was getting high to 80%, normally it uses less
>> than 30%. I issued the select query manually and feel the response is slow.
>> I have tried nodetool compact/repair for those CFs but that does not work.
>> later, I issue 'truncate' for all the CFs and CPU usage gets down to 1%.
>> >
>> > Can somebody explain to me why I need to truncate an empty CF? and what
>> else I could do to bring the CPU usage down?
>> >
>> > I am running 0.8.6.
>> >
>> > Thanks,
>> >
>> > Daning
>> >
>>
>>
>
>
Re: Weird problem with empty CF
Posted by aaron morton <aa...@thelastpickle.com>.
As with any situation involving the un-dead, it really is the number of Zombies, Mummies or Vampires that is the concern.
If you delete data there will always be tombstones. If you have a delete heavy workload there will be more tombstones. This is why implementing a queue with cassandra is a bad idea.
gc_grace_seconds (and column TTL) are the *minimum* about of time the tombstones will stay in the data files, there is no maximum.
Your read performance also depends on the number of SSTables the row is spread over, see http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/
If you really wanted to purge them then yes a repair and then major compaction would be the way to go. Also consider if it's possible to design the data model around the problem, e.g. partitioning rows by date. IMHO I would look to make data model changes before implementing a compaction policy, or consider if cassandra is the right store if you have a delete heavy workload.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 30/09/2011, at 3:27 AM, Daning Wang wrote:
> Jonathan/Aaron,
>
> Thank you guy's reply, I will change GCGracePeriod to 1 day to see what will happen.
>
> Is there a way to purge tombstones at anytime? because if tombstones affect performance, we want them to be purged right away, not after GCGracePeriod. We know all the nodes are up, and we can do repair first to make sure the consistency before purging.
>
> Thanks,
>
> Daning
>
>
> On Wed, Sep 28, 2011 at 5:22 PM, aaron morton <aa...@thelastpickle.com> wrote:
> if I had to guess I would say it was spending time handling tombstones. If you see it happen again, and are interested, turn the logging up to DEBUG and look for messages from something starting with "Slice"
>
> Minor (automatic) compaction will, over time, purge the tombstones. Until then reads must read discard the data deleted by the tombstones. If you perform a big (i.e. 100k's ) delete this can reduce performance until compaction does it's thing.
>
> My second guess would be read repair (or the simple consistency checks on read) kicking in. That would show up in the "ReadRepairStage" in TPSTATS
>
> it may have been neither of those two things, just guesses. If you have more issues let us know and provide some more info.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/09/2011, at 6:35 AM, Daning wrote:
>
> > I have an app polling a few CFs (select first N * from CF), there were data in CFs but later were deleted so CFs were empty for a long time. I found Cassandra CPU usage was getting high to 80%, normally it uses less than 30%. I issued the select query manually and feel the response is slow. I have tried nodetool compact/repair for those CFs but that does not work. later, I issue 'truncate' for all the CFs and CPU usage gets down to 1%.
> >
> > Can somebody explain to me why I need to truncate an empty CF? and what else I could do to bring the CPU usage down?
> >
> > I am running 0.8.6.
> >
> > Thanks,
> >
> > Daning
> >
>
>
Re: Weird problem with empty CF
Posted by Daning Wang <da...@netseer.com>.
Jonathan/Aaron,
Thank you guy's reply, I will change GCGracePeriod to 1 day to see what will
happen.
Is there a way to purge tombstones at anytime? because if tombstones affect
performance, we want them to be purged right away, not after GCGracePeriod.
We know all the nodes are up, and we can do repair first to make sure the
consistency before purging.
Thanks,
Daning
On Wed, Sep 28, 2011 at 5:22 PM, aaron morton <aa...@thelastpickle.com>wrote:
> if I had to guess I would say it was spending time handling tombstones. If
> you see it happen again, and are interested, turn the logging up to DEBUG
> and look for messages from something starting with "Slice"
>
> Minor (automatic) compaction will, over time, purge the tombstones. Until
> then reads must read discard the data deleted by the tombstones. If you
> perform a big (i.e. 100k's ) delete this can reduce performance until
> compaction does it's thing.
>
> My second guess would be read repair (or the simple consistency checks on
> read) kicking in. That would show up in the "ReadRepairStage" in TPSTATS
>
> it may have been neither of those two things, just guesses. If you have
> more issues let us know and provide some more info.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/09/2011, at 6:35 AM, Daning wrote:
>
> > I have an app polling a few CFs (select first N * from CF), there were
> data in CFs but later were deleted so CFs were empty for a long time. I
> found Cassandra CPU usage was getting high to 80%, normally it uses less
> than 30%. I issued the select query manually and feel the response is slow.
> I have tried nodetool compact/repair for those CFs but that does not work.
> later, I issue 'truncate' for all the CFs and CPU usage gets down to 1%.
> >
> > Can somebody explain to me why I need to truncate an empty CF? and what
> else I could do to bring the CPU usage down?
> >
> > I am running 0.8.6.
> >
> > Thanks,
> >
> > Daning
> >
>
>
Re: Weird problem with empty CF
Posted by aaron morton <aa...@thelastpickle.com>.
if I had to guess I would say it was spending time handling tombstones. If you see it happen again, and are interested, turn the logging up to DEBUG and look for messages from something starting with "Slice"
Minor (automatic) compaction will, over time, purge the tombstones. Until then reads must read discard the data deleted by the tombstones. If you perform a big (i.e. 100k's ) delete this can reduce performance until compaction does it's thing.
My second guess would be read repair (or the simple consistency checks on read) kicking in. That would show up in the "ReadRepairStage" in TPSTATS
it may have been neither of those two things, just guesses. If you have more issues let us know and provide some more info.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 29/09/2011, at 6:35 AM, Daning wrote:
> I have an app polling a few CFs (select first N * from CF), there were data in CFs but later were deleted so CFs were empty for a long time. I found Cassandra CPU usage was getting high to 80%, normally it uses less than 30%. I issued the select query manually and feel the response is slow. I have tried nodetool compact/repair for those CFs but that does not work. later, I issue 'truncate' for all the CFs and CPU usage gets down to 1%.
>
> Can somebody explain to me why I need to truncate an empty CF? and what else I could do to bring the CPU usage down?
>
> I am running 0.8.6.
>
> Thanks,
>
> Daning
>