garbage collection patterns on agent

[Joe Crobak <jo...@gmail.com>]

We are running flume (0.9.4-cdh3u2) agents with the following config:

source: thriftSource(12345)
sink:  batch(100) gzip autoBEChain

We have noticed that the Old Gen heapspace builds up slowly over the course
of hours and eventually causes a Mark and Sweep GC that stops the world for
up to 300ms. Has anyone done any tuning to avoid this kind of thing? We
were considering a low-pause GC like the CMS GC but are also wondering how
so much is being promoted to old-gen in the first place. If anyone has any
ideas on that, it'd be great to hear.

Here's a link to a screenshot of our old-gen jvm stats:
https://skitch.com/joecrow/g11us/results-resource-graphs-reports-opennms-web-console

Thanks,
Joe