You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by rajpal reddy <ra...@gmail.com> on 2018/06/13 13:42:38 UTC
G1GC CPU Spike
Hello,
we are using G1GC and noticing garbage collection taking a while and during that process we are seeing cpu spiking up to 70-80%. can you please let us know. if we have to tune any parameters for that. attaching the cassandra-env file with jam-options.
Re: CPU Spike with Jmx_exporter
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hello,
I did not work with the 'jmx_exporter' in a production cluster, but for
datadog agent and other collectors I could work with, the number of metrics
being collected was a key point.
Cassandra exposes a lot of metrics and I saw datadog agents taking too much
CPU, I even saw Graphite servers falling because of the load due to some
Cassandra nodes sending metrics. I would recommend you to make sure that
you are filtering-in only metrics that are used to display some charts or
used for alerting purposes. Restrict the pattern for the rules as much as
possible.
Also, for datadog agents, some work was done in the latest version so
metric collection requires less CPU. Maybe is there a similar update that
was released or that you could ask for. Also, CPU might be related to GC as
I see the agent is running inside a JVM, some tuning might help.
If you really cannot do much to improve it on your side, I would open an
issue or a discussion on prometheus side (
https://github.com/prometheus/jmx_exporter/issues maybe?).
C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2018-07-05 20:39 GMT+01:00 rajpal reddy <ra...@gmail.com>:
> We are seeing the CPU spike only when Jmx metrics are exposed using
> Jmx_exporter. tried setting up imx authentication still see cpu spike. if
> i stop using jmx exporter we don’t see any cpu spike. is there any thing
> we have to tune to make work with Jmx_exporter?
>
>
> On Jun 14, 2018, at 2:18 PM, rajpal reddy <ra...@gmail.com>
> wrote:
>
> Hey Chris,
>
> Sorry to bother you. Did you get a chance to look at the gclog file I sent
> last night.
>
> On Wed, Jun 13, 2018, 8:44 PM rajpal reddy <ra...@gmail.com>
> wrote:
>
>> Chris,
>>
>> sorry attached wrong log file. attaching gc collection seconds and cpu.
>> there were going high at the same time and also attached the gc.log.
>> grafana dashboard and gc.log timing are 4hours apart gc can be see 06/12th
>> around 22:50
>>
>> rate(jvm_gc_collection_seconds_sum{"}[5m])
>>
>> > On Jun 13, 2018, at 5:26 PM, Chris Lohfink <cl...@apple.com> wrote:
>> >
>> > There are not even a 100ms GC pause in that, are you certain theres a
>> problem?
>> >
>> >> On Jun 13, 2018, at 3:00 PM, rajpal reddy <ra...@gmail.com>
>> wrote:
>> >>
>> >> Thanks Chris I did attached the gc logs already. reattaching them
>> now<gc.log.1.current>.
>> >>
>> >> it started yesterday around 11:54PM
>> >>> On Jun 13, 2018, at 3:56 PM, Chris Lohfink <cl...@apple.com>
>> wrote:
>> >>>
>> >>>> What is the criteria for picking up the value for G1ReservePercent?
>> >>>
>> >>>
>> >>> it depends on the object allocation rate vs the size of the heap.
>> Cassandra ideally would be sub 500-600mb/s allocations but it can spike
>> pretty high with something like reading a wide partition or repair
>> streaming which might exceed what the g1 ygcs tenuring and timing is
>> prepared for from previous steady rate. Giving it a bigger buffer is a nice
>> safety net for allocation spikes.
>> >>>
>> >>>> is the HEAP_NEWSIZE is required only for CMS
>> >>>
>> >>>
>> >>> it should only set Xmn with that if using CMS, with G1 it should be
>> ignored or else yes it would be bad to set Xmn. Giving the gc logs will
>> give the results of all the bash scripts along with details of whats
>> happening so its your best option if you want help to share that.
>> >>>
>> >>> Chris
>> >>>
>> >>>> On Jun 13, 2018, at 12:17 PM, Subroto Barua <
>> sbarua116@yahoo.com.INVALID <sb...@yahoo.com.invalid>> wrote:
>> >>>>
>> >>>> Chris,
>> >>>> What is the criteria for picking up the value for G1ReservePercent?
>> >>>>
>> >>>> Subroto
>> >>>>
>> >>>>> On Jun 13, 2018, at 6:52 AM, Chris Lohfink <cl...@apple.com>
>> wrote:
>> >>>>>
>> >>>>> G1ReservePercent
>> >>>>
>> >>>> ------------------------------------------------------------
>> ---------
>> >>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> >>>> For additional commands, e-mail: user-help@cassandra.apache.org
>> >>>>
>> >>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> >>> For additional commands, e-mail: user-help@cassandra.apache.org
>> >>>
>> >>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> >> For additional commands, e-mail: user-help@cassandra.apache.org
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> > For additional commands, e-mail: user-help@cassandra.apache.org
>> >
>>
>>
>
CPU Spike with Jmx_exporter
Posted by rajpal reddy <ra...@gmail.com>.
We are seeing the CPU spike only when Jmx metrics are exposed using Jmx_exporter. tried setting up imx authentication still see cpu spike. if i stop using jmx exporter we don’t see any cpu spike. is there any thing we have to tune to make work with Jmx_exporter?
> On Jun 14, 2018, at 2:18 PM, rajpal reddy <ra...@gmail.com> wrote:
>
> Hey Chris,
>
> Sorry to bother you. Did you get a chance to look at the gclog file I sent last night.
>
> On Wed, Jun 13, 2018, 8:44 PM rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
> Chris,
>
> sorry attached wrong log file. attaching gc collection seconds and cpu. there were going high at the same time and also attached the gc.log. grafana dashboard and gc.log timing are 4hours apart gc can be see 06/12th around 22:50
>
> rate(jvm_gc_collection_seconds_sum{"}[5m])
>
> > On Jun 13, 2018, at 5:26 PM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
> >
> > There are not even a 100ms GC pause in that, are you certain theres a problem?
> >
> >> On Jun 13, 2018, at 3:00 PM, rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> Thanks Chris I did attached the gc logs already. reattaching them now<gc.log.1.current>.
> >>
> >> it started yesterday around 11:54PM
> >>> On Jun 13, 2018, at 3:56 PM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
> >>>
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>
> >>>
> >>> it depends on the object allocation rate vs the size of the heap. Cassandra ideally would be sub 500-600mb/s allocations but it can spike pretty high with something like reading a wide partition or repair streaming which might exceed what the g1 ygcs tenuring and timing is prepared for from previous steady rate. Giving it a bigger buffer is a nice safety net for allocation spikes.
> >>>
> >>>> is the HEAP_NEWSIZE is required only for CMS
> >>>
> >>>
> >>> it should only set Xmn with that if using CMS, with G1 it should be ignored or else yes it would be bad to set Xmn. Giving the gc logs will give the results of all the bash scripts along with details of whats happening so its your best option if you want help to share that.
> >>>
> >>> Chris
> >>>
> >>>> On Jun 13, 2018, at 12:17 PM, Subroto Barua <sb...@yahoo.com.INVALID> wrote:
> >>>>
> >>>> Chris,
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>>
> >>>> Subroto
> >>>>
> >>>>> On Jun 13, 2018, at 6:52 AM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
> >>>>>
> >>>>> G1ReservePercent
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> >>>> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
> >>>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> >>> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
> >>>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> >> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> > For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
> >
>
Re: G1GC CPU Spike
Posted by Chris Lohfink <cl...@apple.com>.
There are no bad GCs in the gclog (worse is like 100ms). Everything looks great actually from what I see. CPU utilization isn't inherently a bad thing for what its worth.
Chris
> On Jun 14, 2018, at 1:18 PM, rajpal reddy <ra...@gmail.com> wrote:
>
> Hey Chris,
>
> Sorry to bother you. Did you get a chance to look at the gclog file I sent last night.
>
> On Wed, Jun 13, 2018, 8:44 PM rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
> Chris,
>
> sorry attached wrong log file. attaching gc collection seconds and cpu. there were going high at the same time and also attached the gc.log. grafana dashboard and gc.log timing are 4hours apart gc can be see 06/12th around 22:50
>
> rate(jvm_gc_collection_seconds_sum{"}[5m])
>
> > On Jun 13, 2018, at 5:26 PM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
> >
> > There are not even a 100ms GC pause in that, are you certain theres a problem?
> >
> >> On Jun 13, 2018, at 3:00 PM, rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> Thanks Chris I did attached the gc logs already. reattaching them now<gc.log.1.current>.
> >>
> >> it started yesterday around 11:54PM
> >>> On Jun 13, 2018, at 3:56 PM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
> >>>
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>
> >>>
> >>> it depends on the object allocation rate vs the size of the heap. Cassandra ideally would be sub 500-600mb/s allocations but it can spike pretty high with something like reading a wide partition or repair streaming which might exceed what the g1 ygcs tenuring and timing is prepared for from previous steady rate. Giving it a bigger buffer is a nice safety net for allocation spikes.
> >>>
> >>>> is the HEAP_NEWSIZE is required only for CMS
> >>>
> >>>
> >>> it should only set Xmn with that if using CMS, with G1 it should be ignored or else yes it would be bad to set Xmn. Giving the gc logs will give the results of all the bash scripts along with details of whats happening so its your best option if you want help to share that.
> >>>
> >>> Chris
> >>>
> >>>> On Jun 13, 2018, at 12:17 PM, Subroto Barua <sb...@yahoo.com.INVALID> wrote:
> >>>>
> >>>> Chris,
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>>
> >>>> Subroto
> >>>>
> >>>>> On Jun 13, 2018, at 6:52 AM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
> >>>>>
> >>>>> G1ReservePercent
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> >>>> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
> >>>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> >>> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
> >>>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> >> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> > For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
> >
>
Re: G1GC CPU Spike
Posted by rajpal reddy <ra...@gmail.com>.
Hey Chris,
Sorry to bother you. Did you get a chance to look at the gclog file I sent
last night.
On Wed, Jun 13, 2018, 8:44 PM rajpal reddy <ra...@gmail.com> wrote:
> Chris,
>
> sorry attached wrong log file. attaching gc collection seconds and cpu.
> there were going high at the same time and also attached the gc.log.
> grafana dashboard and gc.log timing are 4hours apart gc can be see 06/12th
> around 22:50
>
> rate(jvm_gc_collection_seconds_sum{"}[5m])
>
> > On Jun 13, 2018, at 5:26 PM, Chris Lohfink <cl...@apple.com> wrote:
> >
> > There are not even a 100ms GC pause in that, are you certain theres a
> problem?
> >
> >> On Jun 13, 2018, at 3:00 PM, rajpal reddy <ra...@gmail.com>
> wrote:
> >>
> >> Thanks Chris I did attached the gc logs already. reattaching them
> now<gc.log.1.current>.
> >>
> >> it started yesterday around 11:54PM
> >>> On Jun 13, 2018, at 3:56 PM, Chris Lohfink <cl...@apple.com> wrote:
> >>>
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>
> >>>
> >>> it depends on the object allocation rate vs the size of the heap.
> Cassandra ideally would be sub 500-600mb/s allocations but it can spike
> pretty high with something like reading a wide partition or repair
> streaming which might exceed what the g1 ygcs tenuring and timing is
> prepared for from previous steady rate. Giving it a bigger buffer is a nice
> safety net for allocation spikes.
> >>>
> >>>> is the HEAP_NEWSIZE is required only for CMS
> >>>
> >>>
> >>> it should only set Xmn with that if using CMS, with G1 it should be
> ignored or else yes it would be bad to set Xmn. Giving the gc logs will
> give the results of all the bash scripts along with details of whats
> happening so its your best option if you want help to share that.
> >>>
> >>> Chris
> >>>
> >>>> On Jun 13, 2018, at 12:17 PM, Subroto Barua
> <sb...@yahoo.com.INVALID> wrote:
> >>>>
> >>>> Chris,
> >>>> What is the criteria for picking up the value for G1ReservePercent?
> >>>>
> >>>> Subroto
> >>>>
> >>>>> On Jun 13, 2018, at 6:52 AM, Chris Lohfink <cl...@apple.com>
> wrote:
> >>>>>
> >>>>> G1ReservePercent
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> >>>> For additional commands, e-mail: user-help@cassandra.apache.org
> >>>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> >>> For additional commands, e-mail: user-help@cassandra.apache.org
> >>>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> >> For additional commands, e-mail: user-help@cassandra.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: user-help@cassandra.apache.org
> >
>
>
Re: G1GC CPU Spike
Posted by rajpal reddy <ra...@gmail.com>.
Chris,
sorry attached wrong log file. attaching gc collection seconds and cpu. there were going high at the same time and also attached the gc.log. grafana dashboard and gc.log timing are 4hours apart gc can be see 06/12th around 22:50
Re: G1GC CPU Spike
Posted by Chris Lohfink <cl...@apple.com>.
There are not even a 100ms GC pause in that, are you certain theres a problem?
> On Jun 13, 2018, at 3:00 PM, rajpal reddy <ra...@gmail.com> wrote:
>
> Thanks Chris I did attached the gc logs already. reattaching them now<gc.log.1.current>.
>
> it started yesterday around 11:54PM
>> On Jun 13, 2018, at 3:56 PM, Chris Lohfink <cl...@apple.com> wrote:
>>
>>> What is the criteria for picking up the value for G1ReservePercent?
>>
>>
>> it depends on the object allocation rate vs the size of the heap. Cassandra ideally would be sub 500-600mb/s allocations but it can spike pretty high with something like reading a wide partition or repair streaming which might exceed what the g1 ygcs tenuring and timing is prepared for from previous steady rate. Giving it a bigger buffer is a nice safety net for allocation spikes.
>>
>>> is the HEAP_NEWSIZE is required only for CMS
>>
>>
>> it should only set Xmn with that if using CMS, with G1 it should be ignored or else yes it would be bad to set Xmn. Giving the gc logs will give the results of all the bash scripts along with details of whats happening so its your best option if you want help to share that.
>>
>> Chris
>>
>>> On Jun 13, 2018, at 12:17 PM, Subroto Barua <sb...@yahoo.com.INVALID> wrote:
>>>
>>> Chris,
>>> What is the criteria for picking up the value for G1ReservePercent?
>>>
>>> Subroto
>>>
>>>> On Jun 13, 2018, at 6:52 AM, Chris Lohfink <cl...@apple.com> wrote:
>>>>
>>>> G1ReservePercent
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: G1GC CPU Spike
Posted by rajpal reddy <ra...@gmail.com>.
Thanks Chris I did attached the gc logs already. reattaching them now
Re: G1GC CPU Spike
Posted by Chris Lohfink <cl...@apple.com>.
> What is the criteria for picking up the value for G1ReservePercent?
it depends on the object allocation rate vs the size of the heap. Cassandra ideally would be sub 500-600mb/s allocations but it can spike pretty high with something like reading a wide partition or repair streaming which might exceed what the g1 ygcs tenuring and timing is prepared for from previous steady rate. Giving it a bigger buffer is a nice safety net for allocation spikes.
> is the HEAP_NEWSIZE is required only for CMS
it should only set Xmn with that if using CMS, with G1 it should be ignored or else yes it would be bad to set Xmn. Giving the gc logs will give the results of all the bash scripts along with details of whats happening so its your best option if you want help to share that.
Chris
> On Jun 13, 2018, at 12:17 PM, Subroto Barua <sb...@yahoo.com.INVALID> wrote:
>
> Chris,
> What is the criteria for picking up the value for G1ReservePercent?
>
> Subroto
>
>> On Jun 13, 2018, at 6:52 AM, Chris Lohfink <cl...@apple.com> wrote:
>>
>> G1ReservePercent
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: G1GC CPU Spike
Posted by Subroto Barua <sb...@yahoo.com.INVALID>.
Chris,
What is the criteria for picking up the value for G1ReservePercent?
Subroto
> On Jun 13, 2018, at 6:52 AM, Chris Lohfink <cl...@apple.com> wrote:
>
> G1ReservePercent
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: G1GC CPU Spike
Posted by rajpal reddy <ra...@gmail.com>.
does setting HEAP_NEWSIZE=“800M” mean young generation can only use 800M?
> On Jun 13, 2018, at 12:51 PM, Steinmaurer, Thomas <th...@dynatrace.com> wrote:
>
> Explicitly setting Xmn with G1 basically results in <>overriding the target pause-time goal, thus should be avoided.
> http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html <http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html>
>
> Thomas
>
>
> From: rajpal reddy [mailto:rajpalreddy444@gmail.com <ma...@gmail.com>]
> Sent: Mittwoch, 13. Juni 2018 17:27
> To: user@cassandra.apache.org <ma...@cassandra.apache.org>
> Subject: Re: G1GC CPU Spike
>
> we have this has the Heap settings . is the HEAP_NEWSIZE is required only for CMS. can we get rid of that for G1GC so that it can be used?
> MAX_HEAP_SIZE="8192M"
> HEAP_NEWSIZE=“800M"
>
> On Jun 13, 2018, at 11:15 AM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
>
> That metric is the total number of seconds spent in GC, it will increase over time with every young gc which is expected. Whats interesting is the rate of growth not the fact that its increasing. If graphing tool has option to graph derivative you should use that instead.
>
> Chris
>
>
> On Jun 13, 2018, at 9:51 AM, rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
>
> jvm_gc_collection_seconds_count{gc="G1 Young Generation”} and also young generation seconds count keep increasing
>
> <PastedGraphic-1.png>
>
> On Jun 13, 2018, at 9:52 AM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
>
> The gc log file is best to share when asking for help with tuning. The top of file has all the computed args it ran with and it gives details on what part of the GC is taking time. I would guess the CPU spike is from full GCs which with that small heap of a heap is probably from evacuation failures. Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, along with increasing the amount of heap. 8GB is pretty small for G1, might be better off with CMS.
>
> Chris
>
>
> On Jun 13, 2018, at 8:42 AM, rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
>
> Hello,
>
> we are using G1GC and noticing garbage collection taking a while and during that process we are seeing cpu spiking up to 70-80%. can you please let us know. if we have to tune any parameters for that. attaching the cassandra-env file with jam-options.<cassandra-env.txt>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
>
>
>
> The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
RE: G1GC CPU Spike
Posted by "Steinmaurer, Thomas" <th...@dynatrace.com>.
Explicitly setting Xmn with G1 basically results in overriding the target pause-time goal, thus should be avoided.
http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html
Thomas
From: rajpal reddy [mailto:rajpalreddy444@gmail.com]
Sent: Mittwoch, 13. Juni 2018 17:27
To: user@cassandra.apache.org
Subject: Re: G1GC CPU Spike
we have this has the Heap settings . is the HEAP_NEWSIZE is required only for CMS. can we get rid of that for G1GC so that it can be used?
MAX_HEAP_SIZE="8192M"
HEAP_NEWSIZE=“800M"
On Jun 13, 2018, at 11:15 AM, Chris Lohfink <cl...@apple.com>> wrote:
That metric is the total number of seconds spent in GC, it will increase over time with every young gc which is expected. Whats interesting is the rate of growth not the fact that its increasing. If graphing tool has option to graph derivative you should use that instead.
Chris
On Jun 13, 2018, at 9:51 AM, rajpal reddy <ra...@gmail.com>> wrote:
jvm_gc_collection_seconds_count{gc="G1 Young Generation”} and also young generation seconds count keep increasing
<PastedGraphic-1.png>
On Jun 13, 2018, at 9:52 AM, Chris Lohfink <cl...@apple.com>> wrote:
The gc log file is best to share when asking for help with tuning. The top of file has all the computed args it ran with and it gives details on what part of the GC is taking time. I would guess the CPU spike is from full GCs which with that small heap of a heap is probably from evacuation failures. Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, along with increasing the amount of heap. 8GB is pretty small for G1, might be better off with CMS.
Chris
On Jun 13, 2018, at 8:42 AM, rajpal reddy <ra...@gmail.com>> wrote:
Hello,
we are using G1GC and noticing garbage collection taking a while and during that process we are seeing cpu spiking up to 70-80%. can you please let us know. if we have to tune any parameters for that. attaching the cassandra-env file with jam-options.<cassandra-env.txt>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<ma...@cassandra.apache.org>
For additional commands, e-mail: user-help@cassandra.apache.org<ma...@cassandra.apache.org>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<ma...@cassandra.apache.org>
For additional commands, e-mail: user-help@cassandra.apache.org<ma...@cassandra.apache.org>
The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
Re: G1GC CPU Spike
Posted by rajpal reddy <ra...@gmail.com>.
we have this has the Heap settings . is the HEAP_NEWSIZE is required only for CMS. can we get rid of that for G1GC so that it can be used?
MAX_HEAP_SIZE="8192M"
HEAP_NEWSIZE=“800M"
> On Jun 13, 2018, at 11:15 AM, Chris Lohfink <cl...@apple.com> wrote:
>
> That metric is the total number of seconds spent in GC, it will increase over time with every young gc which is expected. Whats interesting is the rate of growth not the fact that its increasing. If graphing tool has option to graph derivative you should use that instead.
>
> Chris
>
>> On Jun 13, 2018, at 9:51 AM, rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
>>
>> jvm_gc_collection_seconds_count{gc="G1 Young Generation”} and also young generation seconds count keep increasing
>>
>> <PastedGraphic-1.png>
>>
>>> On Jun 13, 2018, at 9:52 AM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
>>>
>>> The gc log file is best to share when asking for help with tuning. The top of file has all the computed args it ran with and it gives details on what part of the GC is taking time. I would guess the CPU spike is from full GCs which with that small heap of a heap is probably from evacuation failures. Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, along with increasing the amount of heap. 8GB is pretty small for G1, might be better off with CMS.
>>>
>>> Chris
>>>
>>>> On Jun 13, 2018, at 8:42 AM, rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> we are using G1GC and noticing garbage collection taking a while and during that process we are seeing cpu spiking up to 70-80%. can you please let us know. if we have to tune any parameters for that. attaching the cassandra-env file with jam-options.<cassandra-env.txt>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
>>>> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
>>> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
>>>
>>
>
Re: G1GC CPU Spike
Posted by Chris Lohfink <cl...@apple.com>.
That metric is the total number of seconds spent in GC, it will increase over time with every young gc which is expected. Whats interesting is the rate of growth not the fact that its increasing. If graphing tool has option to graph derivative you should use that instead.
Chris
> On Jun 13, 2018, at 9:51 AM, rajpal reddy <ra...@gmail.com> wrote:
>
> jvm_gc_collection_seconds_count{gc="G1 Young Generation”} and also young generation seconds count keep increasing
>
> <PastedGraphic-1.png>
>
>> On Jun 13, 2018, at 9:52 AM, Chris Lohfink <clohfink@apple.com <ma...@apple.com>> wrote:
>>
>> The gc log file is best to share when asking for help with tuning. The top of file has all the computed args it ran with and it gives details on what part of the GC is taking time. I would guess the CPU spike is from full GCs which with that small heap of a heap is probably from evacuation failures. Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, along with increasing the amount of heap. 8GB is pretty small for G1, might be better off with CMS.
>>
>> Chris
>>
>>> On Jun 13, 2018, at 8:42 AM, rajpal reddy <rajpalreddy444@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> we are using G1GC and noticing garbage collection taking a while and during that process we are seeing cpu spiking up to 70-80%. can you please let us know. if we have to tune any parameters for that. attaching the cassandra-env file with jam-options.<cassandra-env.txt>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
>>> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
>> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
>>
>
Re: G1GC CPU Spike
Posted by rajpal reddy <ra...@gmail.com>.
jvm_gc_collection_seconds_count{gc="G1 Young Generation”} and also young generation seconds count keep increasing
> On Jun 13, 2018, at 9:52 AM, Chris Lohfink <cl...@apple.com> wrote:
>
> The gc log file is best to share when asking for help with tuning. The top of file has all the computed args it ran with and it gives details on what part of the GC is taking time. I would guess the CPU spike is from full GCs which with that small heap of a heap is probably from evacuation failures. Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, along with increasing the amount of heap. 8GB is pretty small for G1, might be better off with CMS.
>
> Chris
>
>> On Jun 13, 2018, at 8:42 AM, rajpal reddy <ra...@gmail.com> wrote:
>>
>> Hello,
>>
>> we are using G1GC and noticing garbage collection taking a while and during that process we are seeing cpu spiking up to 70-80%. can you please let us know. if we have to tune any parameters for that. attaching the cassandra-env file with jam-options.<cassandra-env.txt>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
Re: G1GC CPU Spike
Posted by rajpal reddy <ra...@gmail.com>.
Chris,
We have total memory on the machine as 16G so we kept 50% to heap . initial we had CMS we had same issues so thought of changing to G1GC we have the same problem still
> On Jun 13, 2018, at 9:52 AM, Chris Lohfink <cl...@apple.com> wrote:
>
> The gc log file is best to share when asking for help with tuning. The top of file has all the computed args it ran with and it gives details on what part of the GC is taking time. I would guess the CPU spike is from full GCs which with that small heap of a heap is probably from evacuation failures. Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, along with increasing the amount of heap. 8GB is pretty small for G1, might be better off with CMS.
>
> Chris
>
>> On Jun 13, 2018, at 8:42 AM, rajpal reddy <ra...@gmail.com> wrote:
>>
>> Hello,
>>
>> we are using G1GC and noticing garbage collection taking a while and during that process we are seeing cpu spiking up to 70-80%. can you please let us know. if we have to tune any parameters for that. attaching the cassandra-env file with jam-options.<cassandra-env.txt>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: G1GC CPU Spike
Posted by rajpal reddy <ra...@gmail.com>.
Thanks Chris. please find the attached gc log. event occurred around 23:55 PM yesterday.
Re: G1GC CPU Spike
Posted by Chris Lohfink <cl...@apple.com>.
The gc log file is best to share when asking for help with tuning. The top of file has all the computed args it ran with and it gives details on what part of the GC is taking time. I would guess the CPU spike is from full GCs which with that small heap of a heap is probably from evacuation failures. Reserving more of the heap to be free (-XX:G1ReservePercent=25) can help, along with increasing the amount of heap. 8GB is pretty small for G1, might be better off with CMS.
Chris
> On Jun 13, 2018, at 8:42 AM, rajpal reddy <ra...@gmail.com> wrote:
>
> Hello,
>
> we are using G1GC and noticing garbage collection taking a while and during that process we are seeing cpu spiking up to 70-80%. can you please let us know. if we have to tune any parameters for that. attaching the cassandra-env file with jam-options.<cassandra-env.txt>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org