Minimizing stats.clj garbage collection effects

[Itai Frenkel <It...@forter.com>]

Hi,

We've enabled yourkit profiler garbage collection statistics in order to reduce GC latencies. It seems that stats.clj contribute a big part of the GCed objects. What settings should I tweak in order to reduce stats object creation ? How would it affect the ui reporting accuracy ?

topology.stats.sample.rate: 0.05
topology.builtin.metrics.bucket.size.secs: 60

Regards,
Itai

Re: Minimizing stats.clj garbage collection effects

[Itai Frenkel <It...@forter.com>]

We are trying to optimize very short GC latencies (hopefully around 10ms... every minute or hopefully more ... and the best way to do it - is not create so many objects). The use case is low throughput - low latency, which is not the usual use case I guess. We are still experimenting, most of the objects are storm related.



________________________________
From: Bobby Evans <ev...@yahoo-inc.com>
Sent: Wednesday, November 19, 2014 10:37 PM
To: user@storm.apache.org
Subject: Re: Minimizing stats.clj garbage collection effects

Only topology.stats.sample.rate might have an impact on it.  The other config has nothing to do with stats.clj.
>From reading through the code it looks like the RollingWindow record being used to keep track of stats is creating new objects each time it is updated.  This is a very functional way of doing it, but I am not sure it is best for GC.  Is GC causing you a lot of issues?

- Bobby


On Wednesday, November 19, 2014 8:40 AM, Itai Frenkel <It...@forter.com> wrote:


Hi,

We've enabled yourkit profiler garbage collection statistics in order to reduce GC latencies. It seems that stats.clj contribute a big part of the GCed objects. What settings should I tweak in order to reduce stats object creation ? How would it affect the ui reporting accuracy ?

topology.stats.sample.rate: 0.05
topology.builtin.metrics.bucket.size.secs: 60

Regards,
Itai

Re: Minimizing stats.clj garbage collection effects

[Bobby Evans <ev...@yahoo-inc.com>]

Only topology.stats.sample.rate might have an impact on it.  The other config has nothing to do with stats.clj.
>From reading through the code it looks like the RollingWindow record being used to keep track of stats is creating new objects each time it is updated.  This is a very functional way of doing it, but I am not sure it is best for GC.  Is GC causing you a lot of issues?

- Bobby
 

     On Wednesday, November 19, 2014 8:40 AM, Itai Frenkel <It...@forter.com> wrote:
   

  <!--#yiv6608971079 p {margin-top:0px;margin-bottom:0px;}-->
| Hi,

We've enabled yourkit profiler garbage collection statistics in order to reduce GC latencies. It seems that stats.clj contribute a big part of the GCed objects. What settings should I tweak in order to reduce stats object creation ? How would it affect the ui reporting accuracy ?

topology.stats.sample.rate: 0.05 |
| topology.builtin.metrics.bucket.size.secs: 60

Regards,
Itai
 |