GC Woes

[Cory Kolbeck <ck...@gmail.com>]

Hi folks,

I've been chasing an issue for a bit now without much luck. We're seeing
occasional (1-2 times a day) pause times of 10+ seconds in a 0.8.2.0 broker
only handling ~3k messages/s. We're only seeing it on one node at a time in
a three node cluster, though which node is affected can change
occasionally. We've tried G1 and Parnew/CMS with various heap sizes and
configurations without fixing the issue.

In digging into things, I found a somewhat odd thing: YourKit's allocation
tracking shows that ~98% (by both count and size) of objects allocated are
closures around string formatting  created in the calls to trace() in the
ReplicaFetcherThread such as
https://github.com/apache/kafka/blob/0.8.2.0/core/src/main/scala/kafka/server/ReplicaFetcherThread.scala#L52

Can anyone replicate this? I see that trace() guards printing internally
with a call to isTraceEnabled, would folks be amenable to explicitly
wrapping the calls there in isTraceEnabled given that it's a decently tight
loop?

Also, if anyone is willing to pitch ideas for GC configs or experiments yo
try, I'm all ears.

Thanks,
Cory K