Recurring actions with 4 hour interval

[Joel Samuelsson <sa...@gmail.com>]

Hello,

We've been having a lot of problems with extremely long GC (and still do)
which I've asked about several times on this list (I can find links to
those discussions if anyone is interested).
We noticed a pattern that the GC pauses may be related to something
happening every 4 hours. Is there anything specific happening within
Cassandra with a 4 hour interval?

Any help is much appreciated,
Joel Samuelsson

Re: Recurring actions with 4 hour interval

[Nate McCall <na...@thelastpickle.com>]

Michael has a good point about the system tables - particularly hints and
batching (though neither should be a real tax unless you have bigger
issues).

If you have a monitoring system, add in the flush counters for these and
other higher traffic tables and see if there is a correlation at the four
hour mark with monitoring output of the GC statistics.


On Tue, Dec 10, 2013 at 9:58 AM, Laing, Michael
<mi...@nytimes.com>wrote:

> 2.0.3: system tables have a 1 hour memtable_flush_period which I have
> observed to trigger compaction on the 4 hour mark. Going by memory tho...
> -ml
>
>
> On Tue, Dec 10, 2013 at 10:31 AM, Andre Sprenger <
> andre.sprenger@getanet.de> wrote:
>
>> As far as I know there is nothing hard coded in Cassandra that kicks in
>> every 4 hours. Turn on GC logging, maybe dump the output of jstats to a
>> file and correlate this data with the Cassandra logs. Cassandra logs are
>> pretty good in telling you what is going on.
>>
>>
>> 2013/12/10 Joel Samuelsson <sa...@gmail.com>
>>
>>> Hello,
>>>
>>> We've been having a lot of problems with extremely long GC (and still
>>> do) which I've asked about several times on this list (I can find links to
>>> those discussions if anyone is interested).
>>> We noticed a pattern that the GC pauses may be related to something
>>> happening every 4 hours. Is there anything specific happening within
>>> Cassandra with a 4 hour interval?
>>>
>>> Any help is much appreciated,
>>> Joel Samuelsson
>>>
>>
>>
>


-- 
-----------------
Nate McCall
Austin, TX
@zznate

Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Recurring actions with 4 hour interval

["Laing, Michael" <mi...@nytimes.com>]

2.0.3: system tables have a 1 hour memtable_flush_period which I have
observed to trigger compaction on the 4 hour mark. Going by memory tho...
-ml


On Tue, Dec 10, 2013 at 10:31 AM, Andre Sprenger
<an...@getanet.de>wrote:

> As far as I know there is nothing hard coded in Cassandra that kicks in
> every 4 hours. Turn on GC logging, maybe dump the output of jstats to a
> file and correlate this data with the Cassandra logs. Cassandra logs are
> pretty good in telling you what is going on.
>
>
> 2013/12/10 Joel Samuelsson <sa...@gmail.com>
>
>> Hello,
>>
>> We've been having a lot of problems with extremely long GC (and still do)
>> which I've asked about several times on this list (I can find links to
>> those discussions if anyone is interested).
>> We noticed a pattern that the GC pauses may be related to something
>> happening every 4 hours. Is there anything specific happening within
>> Cassandra with a 4 hour interval?
>>
>> Any help is much appreciated,
>> Joel Samuelsson
>>
>
>

Re: Recurring actions with 4 hour interval

[Andre Sprenger <an...@getanet.de>]

As far as I know there is nothing hard coded in Cassandra that kicks in
every 4 hours. Turn on GC logging, maybe dump the output of jstats to a
file and correlate this data with the Cassandra logs. Cassandra logs are
pretty good in telling you what is going on.


2013/12/10 Joel Samuelsson <sa...@gmail.com>

> Hello,
>
> We've been having a lot of problems with extremely long GC (and still do)
> which I've asked about several times on this list (I can find links to
> those discussions if anyone is interested).
> We noticed a pattern that the GC pauses may be related to something
> happening every 4 hours. Is there anything specific happening within
> Cassandra with a 4 hour interval?
>
> Any help is much appreciated,
> Joel Samuelsson
>