java 7 u67 + G1 & GC Pauses

[Neelesh <ne...@gmail.com>]

Hi,
   We are using java 7 , u61 with G1 collector on our brokers with the
following settings
-Xms8g -Xmx8g -XX:PermSize=48m -XX:MaxPermSize=48m -XX:+UseG1GC
-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35

along with a bunch of PrintGC flags.

After a week or so we started seeing that the tenured/old gen objects were
never collected and the occupied heap kept going all the way up to 8G and
the only cleanup we ever saw after that was from eden/survivors. It was as
if around 7G of heap is always occupied and even full GCs don't cleanup
anything.

We got an hprof dump and loaded it in MAT, and see that there are about 10
million sun.nio.ch.SocketChannelImpl instances (accounting for 16% of heap)
and about 75 million java.lang.Object instances (accounting for about 12%
of heap). Overall, if we add up all objects socket related, they account
for around 65% of heap space.

Is this something related to
https://issues.apache.org/jira/browse/KAFKA-2096 ?

Thanks!
-neelesh