Re: Producer garbage collection problem

[Florian Ollech <f....@yieldlab.de>]

Thanks, but those sadly don't help me. I tried your configuration but did
not change of fix the behaviour I'm seeing.

I investigated a bit further and figured out that after a full garbage
collection I seem to get a problem with the scala Stream API.
Using jmap -histo I can see a very quickly growing count of
scala.collection.immutable.Stream$Cons, Stream$$anonfun$takeWhile$1 and
Stream$$anonfun$continually$1. Cons grows with a rate of about 50,000/s and
the other two with half of that. At about 3,500,000 the GC kicks in again
and it begins anew.
During normal operation I get a growth rate of maybe 2,000/s and never go
above about 75,000 instances total.

I found a scala bug / pull request about Stream.continually / Stream.cons
that might point to the problem I'm experiencing:
https://github.com/scala/scala/pull/3450
I don't really now scala though, so maybe there is an alternative to using
Stream.continually that does not suddenly leak memory?
Has anyone ever experienced this and knows how to fix it?

Thanks,
Florian



On Fri, Jan 31, 2014 at 5:48 PM, Jun Rao <ju...@gmail.com> wrote:

> You can take a look at the GC setting described in
> https://cwiki.apache.org/confluence/display/KAFKA/Operations
>
> Thanks,
>
> Jun
>
>
> On Fri, Jan 31, 2014 at 2:26 AM, Florian Ollech <f.ollech@yieldlab.de
> >wrote:
>
> > Hi,
> >
> > I am currently trying to setup Kafka 0.8 and ran into a problem with the
> > producer. Every time the server triggers a garbage collection, because
> > permgen space is full, the server descends into a very unusual memory
> state
> > where a lot of garbage collection is suddenly happening (~5-10% gc cpu
> > time, when normally it is 0-0.5%). Additionally a lot of objects suddenly
> > get promoted from eden space to survivor space, old gen and perm gen.
> This
> > obviously slows down the whole server and turns up the system load by a
> > lot.
> > This behaviour only appears when the producer is active during a perm gen
> > gc and can be remedied by reloading the producer(s). Since reloading the
> > producer(s) every 5 seconds really is not how it is supposed to work, I
> am
> > wondering if any one has any pointers as to where the problem may be?
> >
> > My system has the following specs:
> > - oracle jdk 1.6.0_22-b04
> > - sending about 12000 messages per second, about 60 bytes each
> > - producer counts tried: 1, 10, 25, 50
> > - async producer
> > - ack makes no difference
> > - batch sizes tried: 100 - 5000
> >
> > I made screenshot of the behaviour: http://i.imgur.com/xtlVNaK.png
> > at ~10:24:30 the full gc happens, surivor space fills up and old gen
> > becomes a sawtooth.
> > at ~10:34 I reloaded the producers and everything was fine again.
> >
> > Cheers,
> > Flo
> >
>



-- 
Florian Ollech
Software Engineer - Yieldlab GmbH

Mail: f.ollech@yieldlab.de
Web: http://www.yieldlab.de

Geschäftsführer: Marco Klimkeit
Sitz: Hamburg HRB 115082

Re: Producer garbage collection problem

[pengfei li <lp...@gmail.com>]

Hi,

     I met the same problem. The scala bug
https://github.com/scala/scala/pull/3450 was fixed in version 2.11, and I
try the
kafka_2.11-0.8.2.1.tgz which compiled with scala 2.11, there is still the
same problem.
     Could you found the solution?

Thanks



2014-02-05 0:47 GMT+08:00 Florian Ollech <f....@yieldlab.de>:

> Thanks, but those sadly don't help me. I tried your configuration but did
> not change of fix the behaviour I'm seeing.
>
> I investigated a bit further and figured out that after a full garbage
> collection I seem to get a problem with the scala Stream API.
> Using jmap -histo I can see a very quickly growing count of
> scala.collection.immutable.Stream$Cons, Stream$$anonfun$takeWhile$1 and
> Stream$$anonfun$continually$1. Cons grows with a rate of about 50,000/s and
> the other two with half of that. At about 3,500,000 the GC kicks in again
> and it begins anew.
> During normal operation I get a growth rate of maybe 2,000/s and never go
> above about 75,000 instances total.
>
> I found a scala bug / pull request about Stream.continually / Stream.cons
> that might point to the problem I'm experiencing:
> https://github.com/scala/scala/pull/3450
> I don't really now scala though, so maybe there is an alternative to using
> Stream.continually that does not suddenly leak memory?
> Has anyone ever experienced this and knows how to fix it?
>
> Thanks,
> Florian
>
>
>
> On Fri, Jan 31, 2014 at 5:48 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > You can take a look at the GC setting described in
> > https://cwiki.apache.org/confluence/display/KAFKA/Operations
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Jan 31, 2014 at 2:26 AM, Florian Ollech <f.ollech@yieldlab.de
> > >wrote:
> >
> > > Hi,
> > >
> > > I am currently trying to setup Kafka 0.8 and ran into a problem with
> the
> > > producer. Every time the server triggers a garbage collection, because
> > > permgen space is full, the server descends into a very unusual memory
> > state
> > > where a lot of garbage collection is suddenly happening (~5-10% gc cpu
> > > time, when normally it is 0-0.5%). Additionally a lot of objects
> suddenly
> > > get promoted from eden space to survivor space, old gen and perm gen.
> > This
> > > obviously slows down the whole server and turns up the system load by a
> > > lot.
> > > This behaviour only appears when the producer is active during a perm
> gen
> > > gc and can be remedied by reloading the producer(s). Since reloading
> the
> > > producer(s) every 5 seconds really is not how it is supposed to work, I
> > am
> > > wondering if any one has any pointers as to where the problem may be?
> > >
> > > My system has the following specs:
> > > - oracle jdk 1.6.0_22-b04
> > > - sending about 12000 messages per second, about 60 bytes each
> > > - producer counts tried: 1, 10, 25, 50
> > > - async producer
> > > - ack makes no difference
> > > - batch sizes tried: 100 - 5000
> > >
> > > I made screenshot of the behaviour: http://i.imgur.com/xtlVNaK.png
> > > at ~10:24:30 the full gc happens, surivor space fills up and old gen
> > > becomes a sawtooth.
> > > at ~10:34 I reloaded the producers and everything was fine again.
> > >
> > > Cheers,
> > > Flo
> > >
> >
>
>
>
> --
> Florian Ollech
> Software Engineer - Yieldlab GmbH
>
> Mail: f.ollech@yieldlab.de
> Web: http://www.yieldlab.de
>
> Geschäftsführer: Marco Klimkeit
> Sitz: Hamburg HRB 115082
>